I am trying to ingest minute data using the following command:
quantrocket zipline ingest 'usstock-1min'
The problem is that it is taking too long, and my docker starts to hang before it finishes. Is there a way that I can ingest in batches to avoid this issue? Also, when the ingest fails, do the ingested securities get stored, or does that data get lost?
Data is synced to your hard drive, so a failed ingestion will run quicker the next time. You could ingest in batches by using the --sids or --universes params as described in the docs. But if Docker is hanging, I would make sure your system and Docker have adequate memory as usstock initial ingestion should just run in the background for 12-18 hours without much trouble.