QuantRocket 2.6.0 Release Notes

Announcements
#1

QuantRocket 2.6.0 is now available. This release includes an updated version of Zipline that works with modern packages and runs in QuantRocket's standard environment, plus other enhancements and bug fixes. All users are encouraged to update to the latest version. See how to update .

Highlights

  • Zipline modernization. The Zipline package, which was pinned to old versions of pandas, numpy, and other libraries and is no longer maintained by Quantopian, has been streamlined and updated to work with modern package versions. As a result, Zipline is now installed in the standard QuantRocket environment. Benefits include:
    1. In JupyterLab, you no longer need to use Zipline in a separate "Zipline environment" kernel (which has been removed) but can use the standard Python 3 kernel;
    2. You can execute Zipline pipelines in satellite scripts;
    3. You can more easily install custom packages in the zipline container (previously, doing so tended to break Zipline due to Zipline's dependency on old package versions).
  • Better futures calendar support in Zipline, including expanded calendar choices and support for calendars with trading breaks. See additional notes below under the quantrocket/zipline and trading_calendars sections.
  • Support for recording executions that happen outside of QuantRocket's knowledge. This can be used to keep the blotter aligned with your broker when one of your positions undergoes a tender offer, merger or acquisition, etc. See the usage guide.
  • Update to JupyterLab 3.0, the most notable feature of which is a table of contents in the left area when you have a notebook open, allowing you to more easily navigate through long notebooks:

763×345 18.5 KB

API Changes

  • The updated versions of pandas, numpy, etc. that Zipline now uses have the potential to cause breakages in Zipline algorithms if you interact with those libraries directly. For example, in newer versions of pandas you cannot mix timezone-aware and timezone-naive datetimes, while in earlier versions of pandas you could. Users should carefully test their algorithms after updating. The version changes of the major packages are shown below:
    • Python 3.6 -> Python 3.8
    • pandas 0.22 -> pandas 1.2
    • numpy 1.13 -> numpy 1.19
  • Several previously deprecated Zipline functions have been removed, including zipline.api.fetch_csv (use quantrocket.get_prices instead), zipline.api.set_do_not_order_list (use zipline.api.set_asset_restrictions instead), and zipline.api.symbol (use zipline.api.sid instead).
  • When requesting multiple fields for multiple assets using data.history(), Zipline previously returned the data as a pandas Panel, support for which was removed starting in pandas 1.0. For this type of request, Zipline will now return a DataFrame in which the columns are a multiindex of (field, asset). No code changes should be required, however, as fields and assets can be accessed in the same way as before, by first accessing the desired field then the desired asset(s):
# used to return a Panel, now returns a multiindex DataFrame
prices = data.history(assets, ["close", "open"], 150, '1d')
aapl = algo.sid("FIBBG000B9XRY4")
aapl_closes = prices["close"][aapl]
  • the quantrocket_trading_calendars package is now deprecated as all of its functionality has been merged into QuantRocket's fork of the trading_calendars package. Please update any imports of quantrocket_trading_calendars to import trading_calendars instead; no other code changes are required. Importing quantrocket_trading_calendars will still work but will print a deprecation warning.
  • Zipline users with a minute bundle associated with the CMES/CME/GLOBEX trading calendar should drop and re-ingest the bundle. Reason: the trading hours for the CMES/CME/GLOBEX calendar have been changed (corrected) from a 24-hour session to a 23-hour session (5pm - 4pm CT). To ensure proper querying of the minute bundle, this calendar change necessitates dropping and re-ingesting the bundle. Please also see the expanded set of choices for CME calendars as described below in the trading_calendars section, as there may be a more optimal calendar choice for your bundle than the standard CMES/CME/GLOBEX calendar.
  • With this release, QGrid, the Excel-like widget for filtering and sorting DataFrames in notebooks, has been removed from JupyterLab. QGrid is not compatible with the latest version of JupyterLab, and unlike other Quantopian-created packages that QuantRocket now maintains, we deem QGrid to be too minor in importance to warrant ongoing maintenance. Given the technical proficiency of most QuantRocket users, we believe few if any users are currently using or relying on QGrid. If you were using QGrid, please use standard pandas techniques for filtering and sorting your DataFrames instead.

Documentation

quantrocket-client:2.6.0

  • fix a performance issue in get_prices when querying a database with text fields (which is possible starting in version 2.5.0 due to custom database support). See related forum post.
  • when querying a real-time aggregate database with get_prices and passing a timezone parameter along with start_date and/or end_date, the timezone will now be passed to the realtime service to aid in the interpretation of start_date/end_date. See related forum post.

quantrocket/blotter:2.6.0

  • Add new endpoint for recording executions that happen outside of QuantRocket's knowledge. This can be used to keep the blotter aligned with your broker when one of your positions undergoes a tender offer, merger or acquisition, etc. See the usage guide.

quantrocket/houston:2.6.0

  • Update to OpenResty version 1.19.3.1.
  • Fix an issue where connecting to the IB Gateway GUI of a cloud deployment incorrectly counted against the concurrent install limit. See related forum post.

quantrocket/ibg:2.6.0

  • update to IB Gateway version 978, the current stable version

quantrocket/jupyter:2.6.0

  • Remove the separate "Zipline environment" kernel. Zipline is now installed in the standard Python 3 kernel.
  • update to JupyterLab 3.0. This release includes a table of contents feature to facilitate navigating through long notebooks.
  • update to quantrocket-client 2.6.0
  • remove QGrid. See the fuller note above in API Changes.

quantrocket/moonshot:2.6.0

  • add a new ACCOUNT_BALANCE_FIELD parameter to the Moonshot class. This parameter allows you to customize which account balance field should be used to calculate order quantities in live trading. See the full description of the parameter in the API Reference.
  • fix a bug that prevented specifying integer account numbers when using the --accounts options with quantrocket moonshot trade

quantrocket/postgres:2.6.0

  • update to TimescaleDB version 2.3.0. This is a maintenance upgrade with no significant new features.

quantrocket/realtime:2.6.0

  • fix an issue where including a large number of sids in the query parameters could result in "Argument too long" errors. See related forum post.
  • change the treatment of the end_date parameter in quantrocket.realtime.download_market_data_file to be inclusive when only a date with no time is specified. Previously such a query was exclusive because the end date was automatically normalized to 00:00:00 (midnight). See related forum post.
  • improve reliability when automatically restarting Alpaca real-time data collection after a network disconnection

quantrocket/zipline:2.6.0

  • modernize zipline to run on Python 3.8, Pandas 1.2, and other modern package versions. See the fuller note above.
  • support for using calendars that incorporate trading breaks (applicable to some futures exchanges and some Asian exchanges). The minutes during the break will now be omitted from Zipline, meaning handle_data and other functions will not be called for those minutes.
  • respect EODCancel (end-of-day cancel policy) in daily mode. Previously, Zipline applied EODCancel in minute mode but ignored it in daily mode. In daily mode, Zipline now cancels any orders which do not fill in the next session after they are placed. See related forum post.
  • improve memory profile of backtests by reducing Zipline's cache sizes. See related forum post.
  • when running a backtest or trading using a daily bundle, the --data-frequency/data_frequency parameter will default to "daily" if omitted. Previously, it was necessary to specify the data frequency as daily explicitly.
  • fix a bug that prevented specifying integer account numbers when using the --accounts options with quantrocket zipline trade
  • log a warning in live trading when a position has been delisted, instructing the user to manually record the execution (using the blotter's new API for recording executions). Previously, a NotImplemented error was raised when this situation arose. See related forum post.

trading_calendars:2.6.0

  • incorporate all functionality that was previously provided by quantrocket_trading_calendars. (The quantrocket_trading_calendars package is now deprecated and will print a deprecation warning if imported.)
  • add new calendars for different CME-operated exchanges and products to better support futures, including trading breaks where appropriate. See the usage guide for the associated hours and breaks for each new or updated exchange:
    • CMES/CME/GLOBEX (FX and interest rates)
    • CME_EQUITY (E-minis)
    • CME_EQUITY_LIQUID (E-mini liquid hours as defined by Interactive Brokers)
    • NYMEX (energy futures)
    • CBOT (agricultural futures)
  • include the 11:30am - 12:30pm lunch break in the calendar for the Tokyo stock exchange (XTKS/TSEJ)
Error in trading causes Zipline to crash
Open orders not closing
Get_prices fails for realtime aggregate database (argument list too long 'psql')
Get_prices handles timezones differently between history and realtime aggregate databases
IB Gateway Cloud Launch
Long Time taken for pipeline to get Custom Data with text field
Alpaca Realtime Service Intermittently Hangs and Doesn't Start/Resume Startup as Scheduled until Docker Restart
Zipline doesn't support trading breaks in custom calendars
Runaway memory usage by Zipline backtest during data pulls
#2

Awesome stuff as always. Excited to be able to manually add non-QR positions to keep blotter reconciled as well as the ability to pivot the ACCOUNT_BALANCE_FIELD in Moonshot.

#3

Exciting release for zipline users! Thanks for the detailed changelog, as always.

#4

Great additions. The updated libraries are much appreciated.

Thanks

#5

@Brian Thanks so much for this update which helps with my trading in the following ways:

  1. Support for Recording Non QR Blotter Executions -- I use a custom integrated broker, and this now enables me to log all my executions to the blotter so that I can take advantage of all the great performance tracking features. You probably already realize this, but this is one step closer to adding native support for custom brokers. Currently I'm using TradeZero, and basically overrode the Moonshot.trade method to source open order information from my broker and then place trades via my own custom blotter. With this enhancement I'll be able to send a copy of all my transactions from my custom blotter to the native blotter for performance tracking! (p.s. consider documenting this as a way to use a custom broker as it's much more valuable than just for catching missed executions with one of the supported brokers)

  2. The Alpaca realtime fix will now save me $200/mo :slight_smile:

Aside from the above, super excited about the upgrade of Zipline; makes the whole system feel so much cleaner. Also, I was literally wondering about a JupyterLab upgrade this past week, I even looked into doing it myself. In addition to the table of contents, the new version also has the extension manager (in non-experimental mode) which is nice.

#6

Wanted to comment here that one major (potential) benefit of the JupyterLab upgrade is dealing with a slowness issue when switching tabs between large notebooks in JupyterLab as referenced here: Switching to a big notebook in a tab is slow · Issue #4292 · jupyterlab/jupyterlab · GitHub

Haven't upgraded yet, but when I do, this will be one of the first things I test. Will report back with results.