Get Prices 502 Server Error

I'm able to successfully run the following code:

prices = get_prices("usstock-1min", 
                    data_frequency="daily",
                    start_date="2017-01-01", 
                    end_date="2018-01-01",
                    timezone="America/New_York",
                    fields=["Open","High","Low","Close", "Volume"])

prices.head()```

However if I expand the query and extend the end date to "2020-01-01" I receive the following error:

```---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[15], line 2
      1 from quantrocket import get_prices
----> 2 prices = get_prices("usstock-1min", 
      3                     data_frequency="daily",
      4                     start_date="2017-01-01", 
      5                     end_date="2020-01-01",
      6                     timezone="America/New_York",
      7                     fields=["Open","High","Low","Close", "Volume"])
      9 prices.head()

File /opt/conda/lib/python3.11/site-packages/quantrocket/price.py:443, in get_prices(codes, start_date, end_date, universes, sids, exclude_universes, exclude_sids, times, fields, timezone, infer_timezone, cont_fut, data_frequency)
    439 tmp_filepath = "{dir}{sep}zipline.{db}.{pid}.{time}.csv".format(
    440     dir=TMP_DIR, sep=os.path.sep, db=db, pid=os.getpid(), time=time.time())
    442 try:
--> 443     download_bundle_file(db, tmp_filepath, **kwargs)
    444 except NoHistoricalData as e:
    445     # don't complain about NoHistoricalData if we're checking
    446     # multiple databases, unless none of them have data
    447     if len(dbs) == 1:

File /opt/conda/lib/python3.11/site-packages/quantrocket/zipline.py:739, in download_bundle_file(code, filepath_or_buffer, start_date, end_date, data_frequency, universes, sids, exclude_universes, exclude_sids, times, fields)
    735 response = houston.get("/zipline/bundles/data/{0}.csv".format(code), params=params,
    736                        timeout=60*30)
    738 try:
--> 739     houston.raise_for_status_with_json(response)
    740 except requests.HTTPError as e:
    741     # Raise a dedicated exception
    742     # This endpoint returns "no minute|daily data match the query parameters"
    743     no_data_messages = (
    744         "data match the query parameters",
    745         "no free securities match",
    746         "no data has been ingested",
    747     )

File /opt/conda/lib/python3.11/site-packages/quantrocket/houston.py:225, in Houston.raise_for_status_with_json(response)
    223     e.json_response = {}
    224     e.args = e.args + ("please check the logs for more details",)
--> 225 raise e

File /opt/conda/lib/python3.11/site-packages/quantrocket/houston.py:217, in Houston.raise_for_status_with_json(response)
    212 """
    213 Raises 400/500 error codes, attaching a json response to the
    214 exception, if possible.
    215 """
    216 try:
--> 217     response.raise_for_status()
    218 except requests.exceptions.HTTPError as e:
    219     try:

File /opt/conda/lib/python3.11/site-packages/requests/models.py:1021, in Response.raise_for_status(self)
   1016     http_error_msg = (
   1017         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1018     )
   1020 if http_error_msg:
-> 1021     raise HTTPError(http_error_msg, response=self)

HTTPError: ('502 Server Error: Bad Gateway for url: http://houston/zipline/bundles/data/usstock-1min.csv?start_date=2017-01-01&end_date=2020-01-01&data_frequency=daily&fields=Close&fields=High&fields=Low&fields=Open&fields=Volume', 'please check the logs for more details')```

Any guidance here would be appreciated.  Thanks.

Please open the detailed logs and re-run the query. The full traceback will be printed to the logs. Please paste the full traceback here.

Ah, looks like an out of memory issue per the flight log pasted below. Is there an efficient way to clear memory in Quantrocket without rebooting my entire system? Thank you.

quantrocket-houston-1|172.18.0.16 - - [06/May/2024:12:40:31 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.31.0"
quantrocket-houston-1|172.18.0.5 - - [06/May/2024:12:40:31 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.18"
quantrocket-houston-1|172.18.0.17 - - [06/May/2024:12:40:44 +0000] "GET /history/databases HTTP/1.1" 200 3 "-" "python-urllib3/1.26.18"
quantrocket-houston-1|172.18.0.17 - - [06/May/2024:12:40:44 +0000] "GET /realtime/databases HTTP/1.1" 200 3 "-" "python-urllib3/1.26.18"
quantrocket-houston-1|172.18.0.17 - - [06/May/2024:12:40:44 +0000] "GET /zipline/bundles HTTP/1.1" 200 50 "-" "python-urllib3/1.26.18"
quantrocket-account-1|can't yet collect ECB exchange rates because no base currencies found in quantrocket.v2.account.balance.sqlite
quantrocket-houston-1|2024/05/06 12:40:51 [error] 12#12: *178503 upstream prematurely closed connection while reading response header from upstream, client: 172.18.0.17, server: , request: "GET /zipline/bundles/data/usstock-1min.csv?start_date=2017-01-01&end_date=2020-01-01&data_frequency=daily&fields=Low&fields=Open&fields=Volume&fields=High&fields=Close HTTP/1.1", upstream: "uwsgi://172.18.0.21:80", host: "houston"
quantrocket-houston-1|172.18.0.17 - - [06/May/2024:12:40:51 +0000] "GET /zipline/bundles/data/usstock-1min.csv?start_date=2017-01-01&end_date=2020-01-01&data_frequency=daily&fields=Low&fields=Open&fields=Volume&fields=High&fields=Close HTTP/1.1" 502 0 "-" "python-urllib3/1.26.18"
quantrocket-zipline-1|DAMN ! worker 3 (pid: 42187) died, killed by signal 9 :( trying respawn ...
quantrocket-zipline-1|Respawned uWSGI worker 3 (new pid: 42318)
quantrocket-houston-1|172.18.0.5 - - [06/May/2024:12:40:51 +0000] "GET /master/securities.csv?vendors=ibkr&fields=ibkr_ConId HTTP/1.1" 400 137 "-" "python-urllib3/1.26.18"
quantrocket-houston-1|172.18.0.5 - - [06/May/2024:12:40:51 +0000] "GET /master/securities.csv?vendors=alpaca&fields=alpaca_AssetId HTTP/1.1" 400 139 "-" "python-urllib3/1.26.18"
quantrocket-flightlog-1|2024-05-06 12:40:52 quantrocket.zipline: ERROR the system killed the worker handling the request, likely an Out Of Memory error; please add more memory or try a smaller request
  quantrocket-houston-1|172.18.0.21 - - [06/May/2024:12:40:52 +0000] "POST /flightlog/handler HTTP/1.1" 200 5 "-" "-"
  quantrocket-zipline-1|subprocess 42319 exited with code 0
  quantrocket-houston-1|172.18.0.10 - - [06/May/2024:12:40:55 +0000] "POST /ibgrouter/registration/ibg1 HTTP/1.1" 200 5 "-" "python-requests/2.31.0"
  quantrocket-houston-1|172.18.0.16 - - [06/May/2024:12:41:31 +0000] "GET /ibg1/gateway HTTP/1.1" 200 22 "-" "python-requests/2.31.0"
  quantrocket-houston-1|172.18.0.5 - - [06/May/2024:12:41:31 +0000] "GET /ibgrouter/gateways?status=running HTTP/1.1" 200 3 "-" "python-urllib3/1.26.18"

There is a section of the usage guide about dealing with memory problems.

1 Like

Thank you