Retrieving historical data from Alpaca/Polygon.io

We actually implemented a Polygon historical data integration in a development branch earlier this year but abandoned it when we obtained the minute US stock dataset. The abandoned integration mirrors the Interactive Brokers integration, collecting data from the API and saving it to a SQLite database. Basically exactly what you're asking for.

I don’t plan to make this feature generally available because the only purpose would be for collecting historical second or tick data, and I don’t want to mislead customers by implying that they can save massive amounts of data to a SQLite database and have good performance. Many trading platforms overpromise and underdeliver, and I think that's a bad business strategy.

That said, if you are interested in using this integration, we could explore the possibility of dusting it off and making it available on a custom basis for an additional fee, with the understanding that you would be responsible for loading a small enough amount of data to maintain the level of performance you require. Reach out privately if you’re interested.

As for loading arbitrary custom data, that is frequently requested, and QuantRocket will surely support it in some capacity in the future, but we're not there yet. The challenge is that many users like QuantRocket because it makes working with data easy, but it is basically not possible to create an API that can deliver that same ease of use against "arbitrary" data. An API must be designed in tandem with the underlying data, not operating blindly. This doesn't mean a platform can't support custom data and still deliver good user experience, but it means there is no one-size-fits-all solution and we are still sorting out the best mix of solutions.

2 Likes

Thats fair, and I'd be happy to take any risk. we only need small windows at any one time randomized, so should work well. I would just need to know what the fee would be and i can bring that idea back to the team and see what they think, but sounds like it should work for me, thanks. Just let me know.

On a related note, I was hoping we could use Polygon to import their available fundamental data. It's not nearly as rich as Sharadar or Morningstar, but it's included with a subscription to Alpaca, and includes many of the key data points such as marketcap, etc...

1 Like

I second that as a nice-to-have as well. While not a deal breaker for me that would be a welcome addition.

@Brian Any idea when you can get back to me on a time frame and price on this?

I'm going to need the historical Polygon downloader as well if the usstock data set doesn't included extended hours in intraday data. This is a must for me.

1 Like

@the, it looks like this discussion is dead on arrival because Polygon does not offer second-level aggregation in their historical data endpoint:

$ curl 'https://api.polygon.io/v2/aggs/ticker/AAPL/range/1/second/2020-10-14/2020-10-14?apiKey=***'
{"status":"ERROR","error":"Invalid time span. The only supported resolutions are minute|hour|day|week|month|quarter|year"}

@Brian Do you have the advanced level subscription ($500 a month)? Or are you testing on free level?

second-level as far as I know isnt provided at the free tier or even the lowest level of paid tier. It is only available at the higher priced subscriptions.

Can you post the output you get from the above query? The error message matches their documentation, which indicates that aggregates are only available down to minute level. They provide historical ticks or historical aggregates (bars) of 1 minute or larger, but not historical second bars AFAIK. The historical ticks is a different API endpoint than the one we have the draft integration for.

ahh, it is possible it was historical ticks and not historical aggregates, i need to check. but yea i will run it in a bit just to make certain. Was fairly certain it was 1 second.

@Brian Ok just tested. Seems your right I was getting tick data not second data and even with my paid account that endpoint wont provide second data.

Is ingesting tick data as historical data an option?

In the meantime can you tell me what data set you support that has minute level access to TQQQ and SQQQ? Sharadar seems to only have daily, EDI I cant even get an account on (they dont have any sort of public pricing model or clear setup) and IBKR does provide it (which is where im getting it from now) but I find using IBKR even for minute and daily level data causes QR to become unstable, If i queue up stocks to IBKR to pull down and something else triggers a IBKR query the whole system becomes unstable (even other containers, not sure why) and i have to restart... its making the system unsuable at this point (and no i have no second data right now on QR)

Tick data could possibly be loaded as custom data, once that’s supported (hopefully soon).

Any ballpark figure on when that could be supported?

Not in the 2.4 release which will be out in a week or two but potentially in the release after that. Subject to change.

@Brian this would be hughe to get added. I've since added in a Polygon downloader to my local installation, and it's been a pain to say the least. At this point I'm only using Polygon to download the extended hours minute data, and then I'm stitching that data together with the intraday minute data from the us-stock data set at the time I request it within a backtest. While this totally works, I really would prefer to use Polygon as my primary data provider for all historic and real-time data.

Maybe even taking this a step further would be to simply make it easily to integrate any custom pricing data provider so that you don't have to built integrations for each. i.e. in my case, my plan is to fallback to iqfeed.net if Polygon.io goes away for any reason as I know they're still a funded startup.

FYI, I've also added a history downloader for Alpaca as I've just switched from Polygon to Alpaca in order to save nearly $200/mo. Feel free to ping me if anyone is interested; happy to share the code.

1 Like

@charles, I am in a similar situation to you! I would happily take you up on your code sharing offer.

1 Like