Quantrocket history get doesn't work for minute dbs

Technical Help
#1

I'm not sure if/when this regression happened. quantrocket history get doesn't work for minute databases even though I can load the data from a Zipline bundle that's pointing to the same db. It does seem to work for daily data.

jupyter:/var/lib/quantrocket $ quantrocket history get 'fut-1min' --start-date 2021-03-02
error: Command '['sqlite3', '/var/lib/quantrocket/quantrocket.v2.history.fut-1min.sqlite/sid/quantrocket.v2.history.fut-1min.BBG000GTH1Z9.sqlite']'
  returned non-zero exit status 1.
msg: an unhandled exception occurred, please check flightlog for the traceback
status: error

Flightlog output

2021-10-13 21:50:18 quantrocket.history: ERROR Traceback (most recent call last):
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_wsgi_py", line 41, in app
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1994, in __call__
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return self.wsgi_app(environ, start_response)
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1985, in wsgi_app
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     response = self.handle_exception(e)
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return original_handler(e)
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1540, in handle_exception
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     reraise(exc_type, exc_value, tb)
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/_compat.py", line 32, in reraise
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     raise value.with_traceback(tb)
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1982, in wsgi_app
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     response = self.full_dispatch_request()
2021-10-13 21:50:18 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1614, in full_dispatch_request
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     rv = self.handle_user_exception(e)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return original_handler(e)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1517, in handle_user_exception
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     reraise(exc_type, exc_value, tb)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/_compat.py", line 32, in reraise
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG Resetting dropped connection: houston
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     raise value.with_traceback(tb)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1612, in full_dispatch_request
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     rv = self.dispatch_request()
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/app.py", line 1598, in dispatch_request
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return self.view_functions[rule.endpoint](**req.view_args)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask_restful/__init__.py", line 477, in wrapper
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     resp = resource(*args, **kwargs)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask/views.py", line 84, in view
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return self.dispatch_request(*args, **kwargs)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/flask_restful/__init__.py", line 587, in dispatch_request
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     resp = meth(*args, **kwargs)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/site-packages/webargs/core.py", line 441, in wrapper
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return func(*args, **kwargs)
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_app_py", line 246, in get
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_history_query_base_py", line 42, in get_history_to_file
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_history_query_base_py", line 53, in get_history_to_file_from_sql
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_history_query_shard_py", line 33, in get_history_to_csv_file_from_sql
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "sym://qrocket_history_query_base_py", line 85, in get_history_to_csv_file_from_sql
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/subprocess.py", line 415, in check_output
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR   File "/opt/conda/lib/python3.8/subprocess.py", line 516, in run
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR     raise CalledProcessError(retcode, process.args,
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR subprocess.CalledProcessError: Command '['sqlite3', '/var/lib/quantrocket/quantrocket.v2.history.fut-1min.sqlite/sid/quantrocket.v2.history.fut-1min.BBG000GTH1Z9.sqlite']' returned non-zero exit status 1.
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-10-13 21:50:18 quantrocket.history: ERROR 
2021-10-13 21:50:19 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20

The bundle was created with this command:

quantrocket zipline create-bundle-from-db 'fut-1min-bundle' --from-db 'fut-1min' --calendar 'GLOBEX' --start-date '2015-01-01'

This code gives me a valid dataframe:

from zipline.research import continuous_future, get_data

es = continuous_future("ES", roll="calendar")
data = get_data("2021-03-02T00:00:00-00:00")
asset = data.current_chain(es)[0]

num_bars = 1440
ohlc_dict = {'open': 'first', 'high': 'max',
             'low': 'min', 'close': 'last', 'volume': 'sum'}

df = data.history(asset, ["open", "high", "low", "close", "volume"],
                  num_bars, "1m").resample("1H", closed="right").agg(ohlc_dict).dropna()
df.index = df.index.tz_convert("America/Chicago")

df['hl2'] = (df['high'] + df['low']) / 2
df['hlc3'] = (df['high'] + df['low'] + df['close']) / 3
df['ohlc4'] = (df['open'] + df['high'] + df['low'] + df['close']) / 4

df
#2

Of course as soon as I post this, I find a workaround. If I specify --fields 'Open' 'High' 'Low' 'Close', it works fine :man_facepalming:t4:

jupyter:/var/lib/quantrocket $ quantrocket history get 'fut-1min' --start-date 2011-01-01 --end-date 2011-01-02 --fields 'Open' 'High' 'Low' 'Close' | csvlook -I --max-rows 10
| Sid          | Date                      | Open  | High  | Low   | Close |
| ------------ | ------------------------- | ----- | ----- | ----- | ----- |
| BBG000Q07ZN0 | 2011-01-02 23:00:00+00:00 | 91.31 | 91.36 | 91.21 | 91.36 |
| BBG000Q07ZN0 | 2011-01-02 23:01:00+00:00 | 91.34 | 91.42 | 91.31 | 91.41 |
| BBG000Q07ZN0 | 2011-01-02 23:02:00+00:00 | 91.4  | 91.43 | 91.38 | 91.42 |
| BBG000Q07ZN0 | 2011-01-02 23:03:00+00:00 | 91.43 | 91.46 | 91.39 | 91.44 |
| BBG000Q07ZN0 | 2011-01-02 23:04:00+00:00 | 91.41 | 91.41 | 91.4  | 91.4  |
| BBG000Q07ZN0 | 2011-01-02 23:05:00+00:00 | 91.41 | 91.44 | 91.41 | 91.43 |
| BBG000Q07ZN0 | 2011-01-02 23:06:00+00:00 | 91.44 | 91.44 | 91.41 | 91.42 |
| BBG000Q07ZN0 | 2011-01-02 23:07:00+00:00 | 91.42 | 91.42 | 91.39 | 91.42 |
| BBG000Q07ZN0 | 2011-01-02 23:08:00+00:00 | 91.43 | 91.43 | 91.4  | 91.4  |
| BBG000Q07ZN0 | 2011-01-02 23:09:00+00:00 | 91.43 | 91.46 | 91.43 | 91.46 |
| ...          | ...                       | ...   | ...   | ...   | ...   |

I think the command should still function without requiring fields to be specified, but for now I'm unblocked.

#3

The specific SQL error can be found in the detailed logs, immediately preceding the error traceback.