When running zipline and using data.can_trade for a security, and it triggers as False, an error comes back as: "The requested TradingCalendar, XPHL, does not exist." My minute bundle uses trading calendar XNYS.
If I do not use data.can_trade and a security cannot trade, I also get an error - for example: "No minute data for sid 20990.:})
Has there been a change in trading calendars and how should assets that cannot trade be handled?
Can you be more specific? When you say "an error comes back" or "I get an error," that doesn't tell me what API call is returning an error. For example, I can't tell if you're saying that data.can_trade returns an error or you get an error on a subsequent call (and if so, which call). Please provide the specific code snippet producing the error and the full traceback of the error. To maximize the chance of a solution, it's always best to err on the side of giving too much information rather than too little.
Brian, absolutely. Here's the code that I believe is triggering the error:
def rebalance(context, data):
context.weight = compute_weights(context, data)
momentum_weight = 1.0 / len(context.longs) * context.weight
# Liquidate old positions
for security in context.portfolio.positions:
if security not in context.longs and data.can_trade(security):
order_target_percent(security, 0)
# Buy new positions
for security in context.longs:
if data.can_trade(security):
order_target_percent(security, momentum_weight)
Here's the code I run to run the algo:
backtest(
"momentum_long_100",
start_date="2008-01-01",
end_date="2010-09-21",
bundle='usstock-1min',
progress='M',
filepath_or_buffer="zipline/backtests/mom_long_100_results.csv")
And here is the error traceback:
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
<ipython-input-11-f8019dc1d980> in <module>
----> 1 backtest(
2 "momentum_long_100",
3 start_date="2008-01-01",
4 end_date="2010-09-21",
5 bundle='usstock-1min',
/opt/conda/lib/python3.8/site-packages/quantrocket/zipline.py in backtest(strategy, data_frequency, capital_base, bundle, start_date, end_date, progress, filepath_or_buffer)
551 response = houston.post("/zipline/backtests/{0}".format(strategy), params=params, timeout=60*60*96)
552
--> 553 houston.raise_for_status_with_json(response)
554
555 filepath_or_buffer = filepath_or_buffer or sys.stdout
/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
204 e.json_response = {}
205 e.args = e.args + ("please check the logs for more details",)
--> 206 raise e
207
208 # Instantiate houston so that all callers can share a TCP connection (for
/opt/conda/lib/python3.8/site-packages/quantrocket/houston.py in raise_for_status_with_json(response)
196 """
197 try:
--> 198 response.raise_for_status()
199 except requests.exceptions.HTTPError as e:
200 try:
/opt/conda/lib/python3.8/site-packages/requests/models.py in raise_for_status(self)
941
942 if http_error_msg:
--> 943 raise HTTPError(http_error_msg, response=self)
944
945 def close(self):
HTTPError: ('500 Server Error: INTERNAL SERVER ERROR for url: http://houston/zipline/backtests/momentum_long_100?bundle=usstock-1min&start_date=2008-01-01&end_date=2010-09-21&progress=M', {'status': 'error', 'msg': 'The requested TradingCalendar, XCIS, does not exist.'})
Interestingly the calendar it is trying to reference has changed from the last time I tried to run it. This calendar is not affiliated with my minute bundle, which makes me believe it's being pulled in some function attached to the source code.
I also tried replacing "data.can_trade()" with "not data.is_stale()", and while that avoided the calendar problem, it still triggered the "sid is missing minute data" error. Are these two issues unrelated, and if so, is there a workaround for the missing sid issue other than keeping the universe so liquid that it could never possibly trigger? This was run with USTradeableStocks() and volume.top(1000) as filters, so it should be very liquid already.
Thanks for your help!
Also, just in case I've misdiagnosed where the error is coming from, here is the full strategy code. Just a simple monthly rebalanced vol-timed long momentum strategy to get working as a proof of concept.
from zipline.pipeline import Pipeline, CustomFactor
from zipline.pipeline.data import USEquityPricing
from zipline.pipeline.data import sharadar
from zipline.pipeline.factors import AverageDollarVolume, SimpleMovingAverage
from zipline.pipeline.filters import AllPresent, All
from zipline.api import *
from zipline.api import record
import zipline.api as algo
from codeload.tradable_stocks import TradableStocksUS
from quantrocket.zipline import backtest
import logging as log
import numpy as np
import os
import pandas as pd
def initialize(context):
schedule_function(rebalance, date_rules.month_start(), time_rules.market_open(hours=0.5))
schedule_function(record_vars, date_rules.month_start(), time_rules.market_close())
attach_pipeline(make_pipeline(), 'pipeline')
set_commission(commission.PerShare(cost=0.0, min_trade_cost=0))
set_slippage(slippage.VolumeShareSlippage(volume_limit=0.50, price_impact=0.0))
class ComputeMomentum(CustomFactor):
inputs = [USEquityPricing.close]
window_length = 250
def compute(self, today, assets, out, close):
out[:] = close[225] / close[0]
def make_pipeline():
in_sp500 = sharadar.SP500.in_sp500.latest
stock_filter = TradableStocksUS()
universe_mask = in_sp500 & stock_filter
universe = USEquityPricing.volume.latest.top(1000, mask=universe_mask)
Momentum = ComputeMomentum(mask=universe)
sma20 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=20, mask=universe)
sma50 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=50, mask=universe)
sma100 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=100, mask=universe)
sma200 = SimpleMovingAverage(inputs=[USEquityPricing.close], window_length=200, mask=universe)
return Pipeline(
columns={
'Momentum': Momentum,
'sma20': sma20,
'sma50': sma50,
'sma100': sma100,
'sma200': sma200,
},
screen=(universe),
)
def before_trading_start(context, data):
context.SPY = algo.sid('FIBBG000BDTBL9')
pipe_results = pipeline_output('pipeline')
momentum = pipe_results['Momentum']
momentum15 = momentum.sort_values(ascending=False).nlargest(15)
context.longs = momentum15.index
def compute_weights(context, data):
prices = data.history(context.SPY, 'price', 20, '1d')
returns = prices.pct_change()
std_20 = returns.std() * np.sqrt(250)
if std_20 < 0.18:
weight = 1
else:
weight = 0
return weight
def rebalance(context, data):
context.weight = compute_weights(context, data)
momentum_weight = 1.0 / len(context.longs) * context.weight
# Liquidate old positions
for security in context.portfolio.positions:
if security not in context.longs and data.can_trade(security):
order_target_percent(security, 0)
# Buy new positions
for security in context.longs:
if data.can_trade(security):
order_target_percent(security, momentum_weight)
def record_vars(context, data):
record(
leverage = context.account.leverage,
cash = context.portfolio.cash/context.portfolio.portfolio_value,
weight = context.weight
)
This latest run came back looking for calendar XPHL again. Neither calendars are listed in the docs: trading calendars
And lastly as a sanity check:
from quantrocket.zipline import get_bundle_config
get_bundle_config('usstock-1min')
Returns:
{'ingest_type': 'usstock',
'sids': None,
'universes': None,
'free': False,
'data_frequency': 'minute',
'calendar': 'XNYS',
'start_date': '2007-01-03'}
Thanks again!
@Brian Ok, I did some more digging and here's what I found. When calling data.can_trade(security) on an asset, zipline checks for security.is_exchange_open, which sends a call to get_calendar(security) within the TradingCalendar package maintained by QuantRocket.
So far I have ingested the provided minute bundle, pulled NASDAQ STK listings and ES, NQ, and RUT futures. Within my bundle, there are several exchanges that are not recognized within TradingCalendar that are coming from the minute bundle.
from quantrocket.master import get_securities
securities = get_securities()
print('All Exchanges: {}'.format(securities.Exchange.unique()))
noexchange_list = ['FINR', 'XOTC', 'XCIS', 'XPHL', 'XPOR']
print('Unknown Exchanges: {}'.format(noexchange_list))
unknown_exchange = securities[securities.Exchange.isin(noexchange_list)]
print('Percentage of securities with unknown exchange: {}%'.format(round(len(unknown_exchange)/len(securities)*100,2)))
The non-recognized exchanges and % of unrecognized securities are listed below:
All Exchanges: ['XNYS' 'PINX' 'OTCM' 'PSGM' 'FINR' 'ARCX' 'XNAS' 'XASE' nan 'XOTC' 'XCIS'
'XPHL' 'BATS' 'XPOR' 'INDEX' 'XCME']
Unknown Exchanges: ['FINR', 'XOTC', 'XCIS', 'XPHL', 'XPOR']
Percentage of securities with unknown exchange: 4.86%
So the key question is: If I plan to use zipline, is there another way to filter out stocks that cannot trade, or do I need to remove these securities with unrecognized exchanges from my bundle?
You’re correct that some assets are associated with exchanges that don’t have calendars, so the data.can_trade -> asset.is_exchange_open check fails. This situation will be handled better in the next release, but in the meantime, you can address the problem by registering calendar aliases for any missing exchanges:
from trading_calendars import register_calendar_alias
register_calendar_alias("XPHL", "XNYS")
register_calendar_alias("XCIS", "XNYS")
# etcThis issue is fixed in release 2.7.0, without the need to register calendar aliases:
This topic was automatically closed after 6 days. New replies are no longer allowed.