"usstock-1min" data download issues

clehman7 · March 20, 2024, 12:57pm

I started the process of downloading the "usstock-1min" bundle last night around 6pm ET. I ran the following code:

from quantrocket.zipline import create_usstock_bundle
create_usstock_bundle("usstock-1min")
{'status': 'success', 'msg': 'successfully created usstock-1min bundle'}

from quantrocket.zipline import ingest_bundle

ingest a subset of securities first

ingest_bundle("usstock-1min")
{'status': 'the data will be ingested asynchronously'}

then ingest everything

ingest_bundle("usstock-1min")
{'status': 'the data will be ingested asynchronously'}

The download process started almost 15 hours ago. When it started it appeared to be downloading correctly. Below is a snapshot of the logs when it started:

   quantrocket-zipline-1|Ingesting minute bars for FIBBG000BLM0V1 (3298 of 24911)
    quantrocket-zipline-1|Ingesting minute bars for FIBBG000BLMDK5 (3299 of 24911)
    quantrocket-zipline-1|Ingesting minute bars for FIBBG000BLMDQ9 (3300 of 24911)

This morning when I checked on the progress I noticed a bunch of errors in the logs. Below are a couple of snapshots of the logs:

2024-03-19 23:56:54 quantrocket.zipline: INFO [usstock-1min] Ingesting minute bars for 24911 securities in usstock-1min bundle
2024-03-20 00:02:12 quantrocket.zipline: ERROR Traceback (most recent call last):
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 82, in mule_ingest_bundle
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_ingest_py", line 126, in _call_ingest_func
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 106, in ingest
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 537, in ingest
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 560, in _wait_for_minute_workers
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.9/threading.py", line 973, in _bootstrap_inner
2024-03-20 00:02:12 quantrocket.zipline: ERROR self.run()
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "/opt/conda/lib/python3.9/threading.py", line 910, in run
2024-03-20 00:02:12 quantrocket.zipline: ERROR self._target(*self._args, **self._kwargs)
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 503, in wrapper
2024-03-20 00:02:12 quantrocket.zipline: ERROR File "sym://qrocket_qrzipline_bundles_usstock_usstock_py", line 614, in _sync_minute_bars_for_sid
2024-03-20 00:02:12 quantrocket.zipline: ERROR RuntimeError: No active exception to reraise
2024-03-20 00:02:12 quantrocket.zipline: ERROR
2024-03-20 01:09:03 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
2024-03-20 03:29:54 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
2024-03-20 03:29:54 quantrocket.license-service: ERROR failure checking license:
2024-03-20 03:29:54 quantrocket.license-service: ERROR ClientError('An error occurred (InvalidSignatureException) when calling the Query operation: Signature expired: 20240320T032954Z is now earlier than 20240320T033129Z (20240320T034629Z - 15 min.)')
2024-03-20 04:20:52 quantrocket.license-service: ERROR failure checking license:
2024-03-20 04:20:52 quantrocket.license-service: ERROR ClientError('An error occurred (InvalidSignatureException) when calling the Query operation: Signature expired: 20240320T042052Z is now earlier than 20240320T042146Z (20240320T043646Z - 15 min.)')
2024-03-20 05:40:33 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
2024-03-20 05:40:33 quantrocket.license-service: ERROR failure checking license:
2024-03-20 05:40:33 quantrocket.license-service: ERROR ClientError('An error occurred (InvalidSignatureException) when calling the Query operation: Signature expired: 20240320T054033Z is now earlier than 20240320T054146Z (20240320T055646Z - 15 min.)')
2024-03-20 08:13:51 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
2024-03-20 09:31:49 quantrocket.license-service: ERROR failure checking license:
2024-03-20 09:31:49 quantrocket.license-service: ERROR ClientError('An error occurred (InvalidSignatureException) when calling the Query operation: Signature expired: 20240320T093149Z is now earlier than 20240320T093259Z (20240320T094759Z - 15 min.)')
2024-03-20 10:50:18 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
2024-03-20 12:51:25 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')

and in the detailed logs it's showing the following:

quantrocket-license-service-1|fetching license profile for d618........083a
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:51:23 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-blotter-1|recycling spooler after 90 tasks
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:51:23 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-blotter-1|OOOPS the spooler is no more...trying respawn...
quantrocket-blotter-1|spawned the uWSGI spooler on dir /var/tmp/uwsgi/spool with pid 81
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:51:24 +0000] "GET /master/securities.csv?vendors=ibkr&fields=ibkr_ConId HTTP/1.1" 400 137 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:51:24 +0000] "GET /master/securities.csv?vendors=alpaca&fields=alpaca_AssetId HTTP/1.1" 400 139 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:51:24 +0000] "GET /account/balances.csv?latest=True&fields=Account&fields=Broker HTTP/1.1" 400 77 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:51:24 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:51:24 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.9 - - [20/Mar/2024:12:51:25 +0000] "GET /account/balances.csv?latest=True&fields=NetLiquidation&fields=Paper HTTP/1.1" 400 77 "-" "python-urllib3/1.26.8"
quantrocket-flightlog-1|2024-03-20 12:51:25 quantrocket.license-service: WARNING AccountCheckError('account service unavailable, so live trading features will be disabled')
quantrocket-houston-1|172.18.0.9 - - [20/Mar/2024:12:51:25 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:52:25 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:52:25 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:53:25 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:53:25 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.7 - - [20/Mar/2024:12:53:28 +0000] "POST /ibgrouter/registration/ibg1 HTTP/1.1" 200 5 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:54:25 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:54:25 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:55:25 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:55:25 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-license-service-1|fetching license profile for d618........083a
quantrocket-houston-1|172.18.0.9 - - [20/Mar/2024:12:56:25 +0000] "GET /account/balances.csv?latest=True&fields=NetLiquidation&fields=Paper HTTP/1.1" 400 77 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.4 - - [20/Mar/2024:12:56:26 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.27.1"
quantrocket-houston-1|172.18.0.16 - - [20/Mar/2024:12:56:26 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.8"
quantrocket-houston-1|172.18.0.7 - - [20/Mar/2024:12:56:28 +0000] "POST /ibgrouter/registration/ibg1 HTTP/1.1" 200 5 "-" "python-requests/2.27.1"

Is the process ongoing and I just need to wait for it to complete (despite being close to 15 hours now) or is there an issue?

Thanks.

Brian · March 20, 2024, 1:24pm

It usually takes 12-18 hours to ingest the full minute bundle the first time, but the first set of logs you posted is not normal. The variety of errors I see there is most commonly associated with a system having resource constraints. To start with, check memory by running

docker stats

on the host machine to see if Docker has enough memory. More on memory problems.

From a terminal in JupyterLab you can then run:

df -h

to make sure there is enough disk space.

Start there.

clehman7 · March 20, 2024, 1:38pm

I believe I have plenty of memory. I allocated 16 GB to Docker. The docker stats came back with the following:

CONTAINER ID NAME ecb00498f231 quantrocket-blotter-1 b5339da0ecba quantrocket-fundamental-1 6083a2bbbbfd quantrocket-master-1 039ee2054a66 quantrocket-realtime-1 c9529a1d11ea quantrocket-account-1 49eabecf1726 quantrocket-history-1 36e9575191fa quantrocket-db-1 7af323c3ccd6 quantrocket-logspout-1 7fc857112f63 quantrocket-satellite-1 f2bc8ef89672 quantrocket-zipline-1 e8db32b65daa quantrocket-houston-1 f57f30a4fd01 quantrocket-ibgrouter-1 4f9b75cbd2ce quantrocket-moonshot-1 cb7517ffd51f quantrocket-postgres-1 a8ad9b21e345 quantrocket-ibg1-1 1267d9e0dc7f quantrocket-countdown-1 f951ccf52005 quantrocket-license-service-1 0e06bb1a173a quantrocket-flightlog-1 8334a3794c00 quantrocket-codeload-1 0442e3815251 quantrocket-jupyter-1 CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
0.11% 153.8MiB / 15.6GiB 0.96% 7.04MB / 3.7MB 12.3kB / 1.01MB 11
0.01% 93.15MiB / 15.6GiB 0.58% 8.88kB / 0B 0B / 905kB 12
0.04% 268.3MiB / 15.6GiB 1.68% 566kB / 273kB 4.1kB / 3.94MB 13
0.03% 112.7MiB / 15.6GiB 0.71% 24kB / 13.6kB 4.1kB / 12.3kB 25
0.00% 113.9MiB / 15.6GiB 0.71% 204kB / 118kB 561kB / 319kB 10
0.01% 87.89MiB / 15.6GiB 0.55% 8.51kB / 0B 0B / 53.2kB 15
0.01% 70.19MiB / 15.6GiB 0.44% 8.92kB / 0B 0B / 8.19kB 8
0.28% 20.47MiB / 15.6GiB 0.13% 556kB / 3.86MB 10.7MB / 24.6kB 11
0.01% 37.49MiB / 15.6GiB 0.23% 7.6kB / 0B 0B / 8.19kB 9
0.01% 277.7MiB / 15.6GiB 1.74% 20.9GB / 682MB 5.8MB / 21.3GB 22
0.00% 10.19MiB / 15.6GiB 0.06% 193MB / 194MB 10.6MB / 8.19kB 3
0.01% 50.51MiB / 15.6GiB 0.32% 1.8MB / 1.38MB 3.01MB / 0B 4
0.00% 159.9MiB / 15.6GiB 1.00% 7.68kB / 0B 57.3kB / 0B 17
0.05% 141.3MiB / 15.6GiB 0.88% 22.5kB / 16kB 38.8MB / 635MB 13
0.56% 375MiB / 15.6GiB 2.35% 1.31MB / 818kB 131MB / 1.07MB 42
0.03% 51.46MiB / 15.6GiB 0.32% 9.52kB / 1.97kB 24.5MB / 291kB 7
0.01% 130.8MiB / 15.6GiB 0.82% 6.55MB / 7.39MB 4.1kB / 16.4kB 9
0.01% 109.1MiB / 15.6GiB 0.68% 2.29MB / 125kB 4.1kB / 6.03MB 30
0.00% 48.43MiB / 15.6GiB 0.30% 27.1MB / 488kB 455kB / 29.2MB 3
0.32% 1.704GiB / 15.6GiB 10.92% 32.5MB / 157MB 102MB / 31MB 230

Regarding disk space I have the following:

df -h

Filesystem Size Used Avail Use% Mounted on
overlay 455G 34G 398G 8% /
tmpfs 64M 0 64M 0% /dev
shm 64M 0 64M 0% /dev/shm
/dev/vda1 455G 34G 398G 8% /codeload
tmpfs 7.9G 0 7.9G 0% /sys/firmware

Any suggestions about when or if I should kill the current download? If. you recommend stopping it what is the cleanest way to do that? Would I just restart the entire process of downloading again?

Thanks.

clehman7 · March 20, 2024, 1:42pm

Actually, forget about the the question of killing it. I added more disc and it looks like I restarted the Jupyter Notebook session. I've restated the download process. Hopefully it does smoothly. Thanks.

Brian · March 20, 2024, 2:51pm

The memory and disk space look fine. Yes, try it again. If there are similar errors again, try to find the point in the logs where the errors first start to see if that yields clues. The errors are indicative of timeouts which implies system slowness or possibly network problems but it doesn't seem to be a memory issue, which would be the usual culprit.

clehman7 · March 21, 2024, 8:51pm

I was able to successfully download the "usstock-1min" data and can now successfully query the data. However, when I run the following:

from quantrocket.history import list_databases
list_databases()

It does not return the name of this database. It simply shows empty brackets [].

Also, where is the database stored? I can't find it in Jupyter or Docker?

Thank you.

Brian · March 21, 2024, 9:06pm

You downloaded a Zipline bundle which is distinct from a history database. Try this:

from quantrocket.zipline import list_bundles
list_bundles()

More on history databases vs bundles in the usage guide.

Zipline bundles are stored in /var/lib/quantrocket/zipline/data. They are mostly binary files.

clehman7 · March 21, 2024, 9:08pm

Thank you Brian!