Usstock-1min get_prices notify support bug

Technical Help
#1

All usstock-1d & usstock-1min collects fine on the EC2.
(usstock-1min on local still not loading even though local is allocated 12GB & t3.large only has 8GB).

This other ticket might be related to this ticket.
usstock-1d pulls fine, usstock-1min doesnt. It's the identical pull just slicing on 15:58:00.

image
image1882×882 109 KB

image
image2140×180 42.1 KB

Trying inside moonshot dev I get the same error

DB = "usstock-1min"
DB_FIELDS = ["Close"]
DB_TIMES = "15:58:00"

image
image1870×184 42.9 KB

Full logs:

2021-04-14 12:15:53 quantrocket.zipline: ERROR Traceback (most recent call last):
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG Resetting dropped connection: houston
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "sym://qrocket_wsgi_py", line 41, in app
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1994, in __call__
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return self.wsgi_app(environ, start_response)
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     response = self.handle_exception(e)
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return original_handler(e)
2021-04-14 12:15:53 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     reraise(exc_type, exc_value, tb)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     raise value.with_traceback(tb)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     response = self.full_dispatch_request()
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     rv = self.handle_user_exception(e)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 271, in error_router
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return original_handler(e)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     reraise(exc_type, exc_value, tb)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/_compat.py", line 32, in reraise
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     raise value.with_traceback(tb)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     rv = self.dispatch_request()
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return self.view_functions[rule.endpoint](**req.view_args)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 477, in wrapper
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     resp = resource(*args, **kwargs)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask/views.py", line 84, in view
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return self.dispatch_request(*args, **kwargs)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/flask_restful/__init__.py", line 587, in dispatch_request
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     resp = meth(*args, **kwargs)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/webargs/core.py", line 441, in wrapper
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     return func(*args, **kwargs)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "sym://qrocket_app_py", line 619, in get
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_query_py", line 141, in get_history_to_file
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "sym://qrocket_qrzipline_bundles_query_py", line 207, in get_history
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/history_loader.py", line 549, in history
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     is_perspective_after)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/history_loader.py", line 431, in _ensure_sliding_windows
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     array = self._array(prefetch_dts, needed_assets, field)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/history_loader.py", line 595, in _array
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     assets,
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/zipline/data/minute_bars.py", line 1297, in load_raw_arrays
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     out = np.full(shape, np.nan)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR   File "/opt/conda/lib/python3.6/site-packages/numpy/core/numeric.py", line 302, in full
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR     a = empty(shape, dtype, order)
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR MemoryError
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
2021-04-14 12:15:53 quantrocket.zipline: ERROR 
2021-04-14 12:15:54 urllib3.connectionpool: DEBUG http://houston:80 "GET /version HTTP/1.1" 200 20
#2

MemoryError = you're trying to load too much data

#3

I made an extremely small data request.
How can a (1, 6310) shape cause 24% CPU utilization (~2GB)? Same issues in this ticket.
image
image
In order to get a memory error I would imagine get_prices it's trying to load the entire usstock-1min dataframe prior to slicing on fields & times. Or something is happening within Zipline's usstock-1min call.

I tested on both the t2 & t3 large to make sure there wasn't anything within the series upgrade I couldn't see.

Runaway memory usage by Zipline backtest during data pulls