How do I minimise or eliminate the use of csv files when building a universe, creating a database, and filling a Pandas data frame

Technical Help
#1

My current process is:

  • Collect STK and ETF listings asynchronously for the ASX exchange
  • Download the master data from quantrocket to csv file
  • Read csv file into a Pandas data frame called securities.
  • Create a universe using the csv file
  • Create a database for collecting history within the above universe
  • Collect history asynchronously into the database created above.
  • Create a list of tuples called symbol_dates, each containing a stock symbol and closing date.
  • Project a new data frame from securities above to contain the 3 columns:
    ** ConId,
    ** Symbol and
    ** Long Name
  • Create a Contracts object, containing this new data frame.
  • For each tuple in the list symbol_dates above:
    ** Search for the stock symbol in the Contracts list
    ** If the symbol is found, Retrieve the closing price from our universe database and Print the results
    **else Print which stock symbol was not found
#2

I'm sorry but It's not very clear to me what your question is or why you would want to avoid using CSVs. In general, the code library will be your best resource for learning recommended workflows for the tasks you describe. This forum is best used for reporting bugs or other specific problems.