Custom Package Failing

Technical Help
#1

I have installed many packages via these docs prior but databento is giving pyarrow version issues.

import databento as db
> ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject`.

Tried adjusting pyarrow version==11.0.0 per these suggestions to no avail.

I went into the jupyter container

pip show databento
Error in sitecustomize; set PYTHONVERBOSE for traceback:
ImportError: /usr/lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /opt/conda/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-aarch64-linux-gnu.so)
Name: databento
Version: 0.52.0
Summary: Official Python client library for Databento
Home-page: 
Author: Databento
Author-email: [email protected]
License: Apache-2.0
Location: /opt/conda/lib/python3.11/site-packages
Requires: aiohttp, databento-dbn, numpy, pandas, pyarrow, requests, zstandard
Required-by:

I have confirmed pip install databento works on local laptop & can run import outside of QR.

GPT is suggested this.

apt update
apt install --only-upgrade libstdc++6

Running a grep to check I still only have 3.4.28 as last entry

strings /usr/lib/aarch64-linux-gnu/libstdc++.so.6 | grep GLIBCXX
> 
GLIBCXX_3.4.26
GLIBCXX_3.4.27
GLIBCXX_3.4.28

Eventually ended up here, seems promising, but wanted confirmation on the best path prior to more attempts.

#2

My suggestion would be to use requests with Databento's HTTP API and load the data into pandas on your own, rather than using their Python package. The root issue is that their package isn't happy with the version of pyarrow that's already installed, but updating pyarrow (and other packages with C dependencies, like pandas and numpy) can be quite challenging.

If you want to tinker with trying to get the databento Python package to install, you should try to update pyarrow to a satisfactory version before installing databento, so that databento doesn't try to install it with pip. Packages like pyarrow with C dependencies are installed in QuantRocket with conda, so that's what you should use to update them, not pip or apt.

#3

Hey Brian, yeah I think that's path of least resistance moving forward too. The last link I sent gave the conda recommendation but considering any update of QR would require worrying about this package management I'll build from requests. Thanks for your thoughts.