Limit Max and Min Quantities creating double orders?

We have recently implemented Limit_position_sizes in our strategy files.
URL for reference: Force Moonshot to trade in quantities of 100 - #2 by Brian

While reviewing orders, I noticed that quantrocket was sometimes creating 2 orders for the same trade signal/candle.

I've isolated this incident, but I have many others if more data is needed. As a note, the order before the first one in this screenshot was at 19:40:10 and it closed out the previous position.

You'll notice that there were 2 orders, both created for the same strategy (we only have one paper trading atm) with the exact same submitted time. In reviewing the logs we've setup ourselves, there was in fact a -1 one sent for that candle and a 0 sent for the following candle. Because we had two 100 quantity shorts, QuantRocket sent a 200 long to close those out.

I am little confused as to how this is possible, and what we can do to address it.

another example of this happening at a different time:

Looking forward to any advice or fixes!

I can't say anything very helpful here because there isn't enough information to go on. I would suggest starting by looking at your order_stubs_to_orders method and maybe print what's coming into that method and what's going out. You can use review_date to re-run the strategy at the times you posted and see if you can reproduce the error, and if you can, add the logging and go from there.

I threw this onto the end of orders stubs to orders. to capture the logs. I'll report back the findings!

        ordersToLog = orders.copy()
        dateTime = pd.Timestamp.today()
        ordersToLog['datetime'] = dateTime
        ordersToLog.to_csv("/codeload/orders.csv", header=False, mode='a')

Here is the data capture from the order_stubs_to_orders function for today:

0 FIBBG000BCSST7 - BUY GRJ009-Live 100 MKT DAY 2024-03-07 19:20:10.708848
1 0 FIBBG000BCSST7 - SELL GRJ009-Live 100 MKT DAY 2024-03-07 19:21:11.630049
2 0 FIBBG000BCSST7 - BUY GRJ009-Live 100 MKT DAY 2024-03-07 19:23:10.941289
3 0 FIBBG000BCSST7 - SELL GRJ009-Live 100 MKT DAY 2024-03-07 19:24:10.893131
4 0 FIBBG000BCSST7 - BUY GRJ009-Live 100 MKT DAY 2024-03-07 19:25:10.540103
5 0 FIBBG000BCSST7 - SELL GRJ009-Live 100 MKT DAY 2024-03-07 19:27:10.793278

And heres the data for today from orders. It seems like something is going on with blotter thinking we need more positions than the expected 100. time from the above table should roughly match the time from the below table.

OrderId Broker Sid Action TotalQuantity Account OrderRef Submitted
31754 2f2486fb-4218-431c-987a-903398bed508 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-06T20:56:11+00:00
31755 8daa12bd-5e4e-4a16-98c4-fc69ca6f3290 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-06T20:56:11+00:00
31756 ab1cffec-330f-4842-9c69-6b716c2b24b7 alpaca FIBBG000BCSST7 BUY 200 PA337STNCV1D GRJ009-Live 2024-03-06T20:57:11+00:00
31757 1a2cd4bcdbfc11ee8f350242ac12000d alpaca FIBBG000BCSST7 BUY 200 PA337STNCV1D GRJ009-Live 2024-03-06T20:57:11+00:00
31758 3c71d95b-fbeb-4446-a29a-44a325e385c1 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:42:10+00:00
31759 e6b3ec2d-67f5-4c65-8221-08683dcad0d2 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:42:10+00:00
31760 167bc1bb-f9e4-4a76-b12b-5a0de61f9aaa alpaca FIBBG000BCSST7 BUY 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:43:09+00:00
31761 89770510-e3f4-4428-96d3-09e66dc6106c alpaca FIBBG000BCSST7 BUY 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:43:09+00:00
31762 6e589f74-7929-42f9-ae4c-1b91523d56f3 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:44:09+00:00
31763 4548a672-d43c-4ee1-bf93-07e65cb2420d alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T15:44:09+00:00
31764 1c4fd56e-f7ba-49e5-a04c-0734a45ed27c alpaca FIBBG000BCSST7 BUY 200 PA337STNCV1D GRJ009-Live 2024-03-07T15:45:10+00:00
31765 ae00e29edc9911ee8f350242ac12000d alpaca FIBBG000BCSST7 BUY 200 PA337STNCV1D GRJ009-Live 2024-03-07T15:45:10+00:00
31766 c2e43af9-6ef8-49f8-98d2-cda7205721f9 alpaca FIBBG000BCSST7 BUY 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:20:10+00:00
31767 32f06329-2a96-431e-a64e-66d608b91c56 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:21:11+00:00
31768 ecdf7f66-07b8-490b-a194-4fc1c964e118 alpaca FIBBG000BCSST7 BUY 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:23:11+00:00
31769 8d77ed4e-4988-4800-a8ae-b9e5694a9fa6 alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:24:10+00:00
31770 3718815d-fe12-4cc1-a9f6-d66c823c2da4 alpaca FIBBG000BCSST7 BUY 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:25:10+00:00
31771 8d5e65f3-23df-46ef-b00c-28a0cb8e15ac alpaca FIBBG000BCSST7 SELL 100 PA337STNCV1D GRJ009-Live 2024-03-07T19:27:10+00:00

For the record, I am using alpaca aggregate DB with minute candles as my data source and I have the following limit function:

 def limit_position_sizes(self, prices: pd.DataFrame):
        closes = prices.loc[self.CUST_CLOSE_DB_COLUMN]

        #create dataframe for longs detail the max share size for any one stock:
        max_quantities_for_longs = pd.DataFrame(100, index=closes.index, columns=closes.columns.sort_values())
        #Inverse it for short
        max_quantities_for_shorts = -max_quantities_for_longs

        return max_quantities_for_longs, max_quantities_for_shorts

the following order function:

def order_stubs_to_orders(self, orders: pd.DataFrame, prices: pd.DataFrame):        
        #Set Order type to LMT
        orders["OrderType"] = "MKT"   
        orders["Tif"] = "DAY"
        
        ordersToLog = orders.copy()
        
        dateTime = pd.Timestamp.today()
        ordersToLog['datetime'] = dateTime
        ordersToLog.to_csv("/codeload/orders.csv", header=False, mode='a')
        
        
        return orders

Brian, I am happy to email you my entire order file and any logs I have it will help you recreate the issue.

I'm not quite sure what issue you’re referring to. Looks like the last 6 rows from the blotter match the quantities in your logs, and I don’t see any double orders after you started logging.

Per the title of your post, I can at least say that you don’t need to worry that this line

return max_quantities_for_longs, max_quantities_for_shorts

is somehow creating double orders since it’s returning two variables - definitely not the case.

Recognizing that in the other post there was an issue in QuantRocket, in this case it would be premature to assume that at this point. Try to simplify the code to the point that you can isolate and reproduce the issue without all the surrounding noise.

Thank you for your response. I have been able to capture more data that will hopefully shed more light on the issue.

Below is the csv of all orders saved as a csv from the strategy file itself. As mentioned previously, this is coming directly from the dataframe passed to order_stubs_to_orders.It is my understand, that with limit_position_sizes() configured the way that it is, Quantrocket should only order in quantities of 100 and only ever allow 100 positions per stock

It is clear that two unexpected behaviors are happening:

  1. There are sometimes multiple orders happening for the same stock on the same minute candle (which I didn't know was possible). See orange and yellow rows.
  2. When the double orders on the same candle happen, the closing order for that position can sometimes be the matching quantity (look at index 16 and 17) or there can be multiple close orders on a single candle (see index 10-13).

It appears to be possible for these two behaviors to happen at the same time, creating index 16-17.


Validation of Issue:
I took a moment to consider what could cause the behavior we're seeing. The first thing that came to mind was "Maybe the orders aren't being filled for a minute or two."

I pulled all our orders from blotter, threw the csv in excel and cross sectioned the data to find the orders from previous image.

This provides the information needed to prove that quantrocket is sending multiple orders to the broker when the strange behavior takes place and that our orders are being filled when this multiple order behavior happens. The index column matches the index column from the previous image.


Recap:
The above behaviors shouldn't be possible what I understand about the features we're using. Quantrocket shouldn't be able to place multiple orders in one candle for the same stock and it shouldn't open continue to open/add to positions after 100 shares have been acquired (index 6-9).

I believe if we were able to get the above behaviors sorted, the order quantities being above 100 would resolve itself.

Please, correct my understanding if I am off base here. I am still not seeing any evidence of this being an issue within the strategy file itself.

Open to ideas!

Can you create a simple test strategy that reproduces the issue and share the code? One by one, start removing pieces from your code. If the issue persists, remove another piece. If you remove a piece and the issue disappears, add the piece back, and continue removing other pieces. Try to create a single stock strategy with as few pieces as possible that has the issue. A good reproducible example for something like this would be:

  • A simple, one-stock Moonshot strategy
  • the historical and realtime database configuration
  • the relevant scheduled commands from your crontab

One point to note pertaining to your comment is that limit_position_sizes determines the max position size, not necessarily the max order size. The order size is determined by diffing the desired position size and the existing position size.

Can you also make sure you're not somehow running the strategy from two different places - twice on the crontab, for example, or crontab plus manually?

Good call on the Crontab! I discovered that I had an overlapping trade job between 9am and 10am EST.

I used cronguru to set this up and misunderstood one of the jobs.

I used https://cronheatmap.com/ to visual the job and found the issue.

I will keep an eye on the orders the next few days and report back if I find something!