Hi All,

One feature I would very much like to see is the ability to ignore OddLot/SoldOutOfSequence ticks. For back testing on historical data, you will sometimes get a high/low that it insanely high. While not "bad ticks", these really mess up TA indicators and unlikely to trigger a stop loss order/etc.

It would be nice to have a flag on historical data request and streaming bars to ignore these ticks. Below is a picture of a oddlot tick before IQFeed removed it from their database per my inquiry.

A "fix" I devised was looking for large percentage changes +/- 13% on legs and trimming them. However, it has come to my attention that a number of smaller but still significant oddlot ticks are below that threshold and the "fix" doesn't work that well.

Sincerely,
Carson Flint
Edited by BlackSheep on Feb 21, 2018 at 07:03 PM

For comparison, here is IB's historical data. No such tick is registered and would not trigger a stop loss/limit order.

Trade conditions are sent with every tick so you should be able to filter those in your app as you see fit. The field to watch is Most Recent Trade Conditions. We also have trade conditions attached to each tick in history. But as to the minute and daily bars, we don't keep the underlying ticks to build from so it uses what we had from the exchange. I will write up the request though.

As to the previous issue, if I recall we added a tick to the wrong bar because it was soldoutofsequence and a fix to prevent that is part of a release that is in QA now, so it should be live within a couple weeks. Unfortunately, that is not something we can backfill due to the reasons above. But hopefully it will be better going forward.


Tim
Edited by DTN_Tim Walter on Feb 22, 2018 at 09:44 AM

Another note while I am thinking about it, you mentioned oddlots, but oddlots are not part of our minute data. Now oddlots only started a few years ago, and prior to that, a trade of 1 was sent by the exchange the same as a trade of 100, so they will be included in that older data. I wanna say that was 4 maybe 5 years ago.

But, if you think you are seeing oddlots in any recent minute data let me know and if I can I will confirm there isn't a problem I don't know about.

Tim

Thank you for your reply.

I was not aware you made the trade conditions available. I have built a tick to bar converter that excludes the E, O and S DTN bucket trade condition types. Since I am pulling much more data now, the process is much slower.

If you do add the filter trade conditions on historical bars, just giving me the ability to only include C trade conditions would be enough for me.

Carson
Edited by BlackSheep on Feb 24, 2018 at 06:44 AM

Good deal, glad it helped a little at least. Just to expand a little on your request, would the C trade only minute data be helpful if it was for only the last 8 calendar days? It would mean having to save off the data locally every week for longer term back studies, but that is an easier target for me to try and work for on your behalf.

I understand.
8 calendar days would be enough for me.

Good deal, thanks for the info.

Tim

After spending some more time looking at the data, I believe I actually do need the O (and possibly E) trades and need more than the C DTN trade conditions.

I looked at the hex codes from the STC command and all my problem ticks are of type "Sold Out Of Sequence"
25,SOLDOSEQ,Sold Out of Sequence,
49,STPD_SOLDOSEQ,Stopped Stock - Sold out of sequence,
58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

If I had to guess, the inconsistency between IB's data and DTN's data is IB filters out the "Sold Out Of Sequence" ticks since it is a broker and is going to be executing trades and the "sold out of sequence" ticks give a improper picture of the current market. I opened a support ticket with IB to see if that is the case.

So essentially, I would want the requested bars from the streaming/historical bar service to not include the above sold out of sequence ticks.

Sorry, if you had already started the other solution. I have created a solution for historical bars but I am more concerned about streaming bars. Will streaming bars also include "Sold Out Of Sequence" ticks?
Edited by BlackSheep on Feb 24, 2018 at 05:37 PM
Edited by BlackSheep on Feb 24, 2018 at 05:39 PM
Edited by BlackSheep on Feb 24, 2018 at 05:45 PM
Edited by BlackSheep on Feb 24, 2018 at 06:15 PM
Edited by BlackSheep on Feb 24, 2018 at 06:16 PM

I would not expect to see those codes in streaming bars once the fix makes it to live. I don't have a way to include O trades though. It is only C and E trades that are part of the streaming bars content.

Tim

Isn't 58 a Form-T trade and a E trade?
58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

http://www.iqfeed.net/dev/api/docs/HistoricalviaTCPIP.cfm
E - Extended Trade = Form T trade.
Edited by BlackSheep on Feb 25, 2018 at 08:01 PM

It is currently. But, due to it being sent out of sequence, it will now be marked as an other trade.

Tim

I see, will the change effect historical bars also?

Yes, it will be an O trade throughout the system.

Tim

I see, just to be clear, all sold out of sequence trade conditions will be marked as O (25, 49, 58) and thus no longer show up in streaming and historical bars?

Do you have a timeframe on when the change will go live and what is the best way for me to know when the change is live?

Lastly, I am very impressed with the support you guys have given me. You have a great product also.
Thank you for your attention with this matter.
Edited by BlackSheep on Feb 25, 2018 at 08:54 PM

The only place I think that you will see these in streaming or in history going forward will be as Other trades, unless....

The exchange sends us up to 4 trade conditions, the first being the one that is given the most weight. So they could do something weird, like send trade condition 01 (Regular or C trade) as the first condition and then 25,49,58 as the next 3, it would still be a regular trade. I hope this isn't something that should happen, but I just wanted to mention it was possible.

I'll have to follow up on time frame. I will need to check with the QA team on their schedule when they get in.

Tim

Do you have any updates on this?
I had a stock (MU) that came through today on streaming bars with a sold out of sequence tick so I assume it hasn't been implemented yet.

2eface9e-284b-41a8-9079-3de152ff21d5,2018-03-14 09:34:12.634561,60.6900,2000,5749398,58.0300,58.0700,4687,E,19,3A,
3A -> 58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

Good morning,

The hope is that the fix will be pushed out to live this weekend. To my knowledge this is all still on track. But, I'll check with QA and ask that they keep me in the loop for if anything changes.

Tim

Did the update go out this weekend? I still see Sold out of Sequence ticks in historical bar data.

Good morning,

The historical bars you mentioned, what time frame and symbol? The update will only fix the data for yesterday and all data going forward. It will not repair or change any previously received historical data.

Tim

Oh, in message 13 I asked if it would effect historical bars and in 14 you said it would.
>I see, will the change effect historical bars also?
>Yes, it will be an O trade throughout the system.

I truly believe that for backtesting purposes the historical bar data should not include these tick since they will not trigger a realized trade.

Sorry, I thought you meant on historical bars going forward. Unfortunately, since we do not have the underlying tick data in our system for the vast majority of the minute data that we provide, I cannot see what trades were used in the creation of a given minute. I also can't tell what to replace the filtered OHLC values with. I will escalate this to discuss further, but with what I have currently available I can't fix this for all of the minute data.

Tim

I understand now. I would imagine that most users that want historical bar data would want it for backtesting and thus the change makes sense. Please keep me updated on what is decided on.

Is the patch still live? I am seeing a sold out of sequence tick on a bar on GE on 3/29/2018

cc07452d-c1b5-40d7-853f-072b94c6b13f,2018-03-29 15:00:51.000000,15.4450,100,0,0.0000,0.0000,1146304591,C,5,19,

Sorry, this was a correction sent to us by the exchange and it is processed completely different than the live stream. I'll get this on the books for us to fix this as well. Sorry that we didn't catch this on the initial work.

Tim

No problem. Thanks for making the change in the first place.

Has this exchange correction problem been solved and is the patch live?