Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I used to have *******, but they are way more money for the same thing. I have had no probs with data from DTN since switching over." - Comment from Public Forum Post
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
"I just wanted to let you know how fast and easy I found it to integrate IQFeed into our existing Java code using your JNI client. In my experience, such things almost never go so smoothly - great job!" - Comment from Nate
"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"You have an excellent feed. Very few spikes for Spot Forex." - Comment from Public Forum Post
"You are much better than lawyers or the phone company because you answer the phone when I call! I just love your customer service." - Comment from Isreal
"The people at Nirvana have very nice things to say about your company and I can see why! Price and service is a potent combination." - Comment from Ed
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
Home  Search  Register  Login  Blogs Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTN_IQFeed on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Wish List »Ignore OddLot/SoldOutOfSequence ticks
Author Topic: Ignore OddLot/SoldOutOfSequence ticks (27 messages, Page 1 of 1)

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 21, 2018 07:02 PM          Msg. 1 of 27
Hi All,

One feature I would very much like to see is the ability to ignore OddLot/SoldOutOfSequence ticks. For back testing on historical data, you will sometimes get a high/low that it insanely high. While not "bad ticks", these really mess up TA indicators and unlikely to trigger a stop loss order/etc.

It would be nice to have a flag on historical data request and streaming bars to ignore these ticks. Below is a picture of a oddlot tick before IQFeed removed it from their database per my inquiry.

A "fix" I devised was looking for large percentage changes +/- 13% on legs and trimming them. However, it has come to my attention that a number of smaller but still significant oddlot ticks are below that threshold and the "fix" doesn't work that well.

Sincerely,
Carson Flint
Edited by BlackSheep on Feb 21, 2018 at 07:03 PM



File Attached: twx_iqfeed.png (downloaded 124 times)

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 21, 2018 09:05 PM          Msg. 2 of 27
For comparison, here is IB's historical data. No such tick is registered and would not trigger a stop loss/limit order.



File Attached: twx_ib.png (downloaded 122 times)

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 22, 2018 09:41 AM          Msg. 3 of 27
Trade conditions are sent with every tick so you should be able to filter those in your app as you see fit. The field to watch is Most Recent Trade Conditions. We also have trade conditions attached to each tick in history. But as to the minute and daily bars, we don't keep the underlying ticks to build from so it uses what we had from the exchange. I will write up the request though.

As to the previous issue, if I recall we added a tick to the wrong bar because it was soldoutofsequence and a fix to prevent that is part of a release that is in QA now, so it should be live within a couple weeks. Unfortunately, that is not something we can backfill due to the reasons above. But hopefully it will be better going forward.


Tim
Edited by DTN_Tim Walter on Feb 22, 2018 at 09:44 AM

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 22, 2018 10:15 AM          Msg. 4 of 27
Another note while I am thinking about it, you mentioned oddlots, but oddlots are not part of our minute data. Now oddlots only started a few years ago, and prior to that, a trade of 1 was sent by the exchange the same as a trade of 100, so they will be included in that older data. I wanna say that was 4 maybe 5 years ago.

But, if you think you are seeing oddlots in any recent minute data let me know and if I can I will confirm there isn't a problem I don't know about.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 24, 2018 06:24 AM          Msg. 5 of 27
Thank you for your reply.

I was not aware you made the trade conditions available. I have built a tick to bar converter that excludes the E, O and S DTN bucket trade condition types. Since I am pulling much more data now, the process is much slower.

If you do add the filter trade conditions on historical bars, just giving me the ability to only include C trade conditions would be enough for me.

Carson
Edited by BlackSheep on Feb 24, 2018 at 06:44 AM

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 24, 2018 06:32 AM          Msg. 6 of 27
Good deal, glad it helped a little at least. Just to expand a little on your request, would the C trade only minute data be helpful if it was for only the last 8 calendar days? It would mean having to save off the data locally every week for longer term back studies, but that is an easier target for me to try and work for on your behalf.

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 24, 2018 06:48 AM          Msg. 7 of 27
I understand.
8 calendar days would be enough for me.

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 24, 2018 06:51 AM          Msg. 8 of 27
Good deal, thanks for the info.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 24, 2018 05:32 PM          Msg. 9 of 27
After spending some more time looking at the data, I believe I actually do need the O (and possibly E) trades and need more than the C DTN trade conditions.

I looked at the hex codes from the STC command and all my problem ticks are of type "Sold Out Of Sequence"
25,SOLDOSEQ,Sold Out of Sequence,
49,STPD_SOLDOSEQ,Stopped Stock - Sold out of sequence,
58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

If I had to guess, the inconsistency between IB's data and DTN's data is IB filters out the "Sold Out Of Sequence" ticks since it is a broker and is going to be executing trades and the "sold out of sequence" ticks give a improper picture of the current market. I opened a support ticket with IB to see if that is the case.

So essentially, I would want the requested bars from the streaming/historical bar service to not include the above sold out of sequence ticks.

Sorry, if you had already started the other solution. I have created a solution for historical bars but I am more concerned about streaming bars. Will streaming bars also include "Sold Out Of Sequence" ticks?
Edited by BlackSheep on Feb 24, 2018 at 05:37 PM
Edited by BlackSheep on Feb 24, 2018 at 05:39 PM
Edited by BlackSheep on Feb 24, 2018 at 05:45 PM
Edited by BlackSheep on Feb 24, 2018 at 06:15 PM
Edited by BlackSheep on Feb 24, 2018 at 06:16 PM

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 25, 2018 01:06 PM          Msg. 10 of 27
I would not expect to see those codes in streaming bars once the fix makes it to live. I don't have a way to include O trades though. It is only C and E trades that are part of the streaming bars content.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 25, 2018 07:55 PM          Msg. 11 of 27
Isn't 58 a Form-T trade and a E trade?
58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

http://www.iqfeed.net/dev/api/docs/HistoricalviaTCPIP.cfm
E - Extended Trade = Form T trade.
Edited by BlackSheep on Feb 25, 2018 at 08:01 PM

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 25, 2018 08:00 PM          Msg. 12 of 27
It is currently. But, due to it being sent out of sequence, it will now be marked as an other trade.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 25, 2018 08:02 PM          Msg. 13 of 27
I see, will the change effect historical bars also?

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 25, 2018 08:04 PM          Msg. 14 of 27
Yes, it will be an O trade throughout the system.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Feb 25, 2018 08:54 PM          Msg. 15 of 27
I see, just to be clear, all sold out of sequence trade conditions will be marked as O (25, 49, 58) and thus no longer show up in streaming and historical bars?

Do you have a timeframe on when the change will go live and what is the best way for me to know when the change is live?

Lastly, I am very impressed with the support you guys have given me. You have a great product also.
Thank you for your attention with this matter.
Edited by BlackSheep on Feb 25, 2018 at 08:54 PM

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Feb 26, 2018 08:03 AM          Msg. 16 of 27
The only place I think that you will see these in streaming or in history going forward will be as Other trades, unless....

The exchange sends us up to 4 trade conditions, the first being the one that is given the most weight. So they could do something weird, like send trade condition 01 (Regular or C trade) as the first condition and then 25,49,58 as the next 3, it would still be a regular trade. I hope this isn't something that should happen, but I just wanted to mention it was possible.

I'll have to follow up on time frame. I will need to check with the QA team on their schedule when they get in.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Mar 14, 2018 06:46 PM          Msg. 17 of 27
Do you have any updates on this?
I had a stock (MU) that came through today on streaming bars with a sold out of sequence tick so I assume it hasn't been implemented yet.

2eface9e-284b-41a8-9079-3de152ff21d5,2018-03-14 09:34:12.634561,60.6900,2000,5749398,58.0300,58.0700,4687,E,19,3A,
3A -> 58,FORTMTSOLDOSEQ,Form-T Sold Out of Sequence,

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Mar 15, 2018 09:56 AM          Msg. 18 of 27
Good morning,

The hope is that the fix will be pushed out to live this weekend. To my knowledge this is all still on track. But, I'll check with QA and ask that they keep me in the loop for if anything changes.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Mar 19, 2018 03:33 PM          Msg. 19 of 27
Did the update go out this weekend? I still see Sold out of Sequence ticks in historical bar data.

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Mar 20, 2018 09:59 AM          Msg. 20 of 27
Good morning,

The historical bars you mentioned, what time frame and symbol? The update will only fix the data for yesterday and all data going forward. It will not repair or change any previously received historical data.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Mar 20, 2018 10:04 AM          Msg. 21 of 27
Oh, in message 13 I asked if it would effect historical bars and in 14 you said it would.
>I see, will the change effect historical bars also?
>Yes, it will be an O trade throughout the system.

I truly believe that for backtesting purposes the historical bar data should not include these tick since they will not trigger a realized trade.

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Mar 20, 2018 10:29 AM          Msg. 22 of 27
Sorry, I thought you meant on historical bars going forward. Unfortunately, since we do not have the underlying tick data in our system for the vast majority of the minute data that we provide, I cannot see what trades were used in the creation of a given minute. I also can't tell what to replace the filtered OHLC values with. I will escalate this to discuss further, but with what I have currently available I can't fix this for all of the minute data.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Mar 20, 2018 11:02 AM          Msg. 23 of 27
I understand now. I would imagine that most users that want historical bar data would want it for backtesting and thus the change makes sense. Please keep me updated on what is decided on.

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Apr 6, 2018 09:01 PM          Msg. 24 of 27
Is the patch still live? I am seeing a sold out of sequence tick on a bar on GE on 3/29/2018

cc07452d-c1b5-40d7-853f-072b94c6b13f,2018-03-29 15:00:51.000000,15.4450,100,0,0.0000,0.0000,1146304591,C,5,19,

DTN_Tim Walter
-DTN Guru-
Posts: 1158
Joined: Apr 25, 2006


Posted: Apr 9, 2018 08:20 AM          Msg. 25 of 27
Sorry, this was a correction sent to us by the exchange and it is processed completely different than the live stream. I'll get this on the books for us to fix this as well. Sorry that we didn't catch this on the initial work.

Tim

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: Apr 9, 2018 08:25 AM          Msg. 26 of 27
No problem. Thanks for making the change in the first place.

BlackSheep
-Interested User-
Posts: 17
Joined: Sep 22, 2016


Posted: May 5, 2018 06:06 PM          Msg. 27 of 27
Has this exchange correction problem been solved and is the patch live?
 

 

Time: Tue May 22, 2018 8:54 AM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003