Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"Can I get another account from you? I am tired of ******* going down so often" - Comment from George
"Thank you so much - awesome feed, awesome service!" - Comment from Greg via Email
"I have been using IQFeed now for a few years in MultiCharts and I have zero complaints. Very, very rare to have any data hiccups or anything at all go wrong." - Comment from Public Forum
"I was with ******* for 4 years at $230 a month, this is a huge savings for me, GOD BLESS YOU PEOPLE," - Comment from T.S. via Email
"IQ feed works very well, does not have all of the normal interruptions I have grown used to on *******" - Comment from Mark
"If someone needs the best quality data and backfill beyond what their broker provides at a rate that is the best in the industry, I highly recommend IQFeed." - Comment from Josh via Public Forum
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"I was on the phone with a friend who uses CQG and right after the Fed announcement, CQG was as much as 30 seconds behind DTN.IQ. Some quotes were off by as much as 15-18 cents. Your feed never missed a beat." - Comment from Roger
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
"It’s so nice to be working with real professionals!" - Comment from Len
Home  Search  Register  Login  Blogs Recent Posts

Information on Various DTN Products:
DTN IQFeed | DTN ProphetX | DTN Ag | NxCore
Follow DTN_IQFeed on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Tick data into OHLCV
Author Topic: Tick data into OHLCV (23 messages, Page 1 of 1)

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 28, 2016 04:09 PM          Msg. 1 of 23
Hello,

I have a custom application that collects tick data (realtime and history) into OHLCV bars by different timeframes and analyses the price levels. As a result I want to get the same price and volume as in your history interval data. Currently I'm testing my application on history and realtime ticks and want to have a clear understanding how should I filter ticks to get the same volume (period volume) and prices (open, high, low, close) as your historical interval data?
For example, currently I do not use the following ticks:
- Realtime ticks with "Message Contents" field that is not equal "C" (Last Qualified Trade)
- History ticks with "Basis For Last" field that is not equal "C" (Last Qualified Trade)
Should I implement any additional filtering?

Thanks.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 29, 2016 01:25 AM          Msg. 2 of 23
Good morning, we include C & E trades in our interval data. So you would want to add those in as well.

Tim Walter

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 29, 2016 05:00 AM          Msg. 3 of 23
Thank you Tim.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 29, 2016 05:10 AM          Msg. 4 of 23
Anytime, while I am thinking about it, also make note our intervals are such that the data from 11:00:00 to 11:00:59.999999 is placed in the 11:01 minute bar. This can cause some people an off by one error when comparing data.

Let me know if you have any questions,

Tim

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 30, 2016 04:38 AM          Msg. 5 of 23
Hello Tim,

I'm checking the data matching between ticks and interval and found some inconsistencies.
Dealing with BD#C, timeframe is 1 minute, date - 19 November 2015, time 2:02.

Request of interval history data was the following:
HIT,BD#C,60,20150329 030000,20160329 160000,,,,1,HIT_BD#C

Interval data from response:
HIT_BD#C,2015-11-19 02:02:00,156.66,156.61,156.63,156.65,2242,2238,

Next, requested tick data for the interval above:
HTT,BD#C,20151119 020100,20151119 020159,,,,1,HTT_BD#C

Response and volume calculations are in the attached file (ticks_BD#C_20151119_0202.xlsx). There are tick fields in the columns from "A" to "J". Look at column "T" please, there is summary volume (sum of "last size" fields) of the tick data and and interval (period) volume of the interval data.

Could you tell me the reason why these volumes are not equal?

Thank you.
Simon

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 30, 2016 05:13 AM          Msg. 6 of 23
Sorry, xlsx attachments are not supported. Wrapped in a zip archive.



File Attached: ticks_BD#C_20151119_0202.zip (downloaded 103 times)

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 31, 2016 02:28 AM          Msg. 7 of 23
Hi guys, i'm finding numerous inconsistencies between interval data and tick data, could someone please get back to me on the above question today? Trying to get to the bottom of this!

Really appreciate your help

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 31, 2016 09:03 AM          Msg. 8 of 23
Sorry, the file that you uploaded appears to have an issue. Could you re-upload your zip file for me to look over?

Tim

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 31, 2016 09:09 AM          Msg. 9 of 23
Zip-file is re-uploaded. Just in case, here is a link on dropbox: https://www.dropbox.com/s/c9t8cc08jc8p4k2/ticks_BD%23C_20151119_0202.zip?dl=0



File Attached: ticks_BD#C_20151119_02021.zip (downloaded 109 times)

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 31, 2016 09:15 AM          Msg. 10 of 23
Thanks much. Looking it over now.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 31, 2016 09:31 AM          Msg. 11 of 23
What protocol are you requesting the data under?

Note: If you are not calling SET PROTOCOL anywhere on this socket, it would be protocol 4.9.

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Mar 31, 2016 02:11 PM          Msg. 12 of 23
On startup my application connects to the Level1 and sets protocol v5.1.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Mar 31, 2016 02:14 PM          Msg. 13 of 23
Hello,

Sorry for the delay, but this one took me a little while to track down. It appears this is related to an issue that occurred back in 2015 during our initial roll-out of microseconds. This caused the discrepancies that you are seeing in our minute data and we are looking into what it will take to get it resolved. The tick data is correct though in all of the cases that I researched, so in the interim, if you build your bars from the underlying tick data, you will have the correct information.

I will try to get more information on this in the days to come. Let me know if you have any questions beyond this though and I will be happy to impart whatever I can.

Tim

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Apr 3, 2016 12:38 PM          Msg. 14 of 23
Thank you very much Tim.

Currently I'm testing my program with realtime (Level1) ticks. To build bars from tick I'm using the following felds of the Q messages: "Last", "Bid", "Ask", "Last Size", "Last TimeMS" and "Last Date".
Checks showed that value of "Last TimeMS" has a large difference sometimes (more than 5 minutes) comparing to current time. Should I use "Most Recent Trade", "Bid", "Ask", "Most Recent Size", "Most Recent TimeMS" and "Most Recent Date" instead? I want to build bars in realtime and the build bars should match your history interval data.

Thank you.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Apr 4, 2016 05:57 AM          Msg. 15 of 23
Most Recent TimeMS is what you would want. This time field only updates on a trade though, so having it go several minutes without changing, while the bid and ask values may be updating constantly can be completely normal. We would expect the time time value to update on each message that has a Message Contents of C, E, or O. Though, you should be filtering out trades with a message content value of 'O' if you want to match up to history.

Tim

semenchikus
-Interested User-
Posts: 12
Joined: Mar 28, 2016


Posted: Apr 4, 2016 09:25 AM          Msg. 16 of 23
Thank you again for quick reply!

So, in the other words, your historical ticks (response for any of HT* command) according to realtime ticks in the following way:
Last (history) = Most Recent Trade (realtime)
Last Size (history) = Most Recent Trade Size (realtime)
Time Stamp (history) = Most Recent Trade Date + Most Recent Trade TimeMS (realtime)
Bid (history) = Bid (realtime)
Ask (history) = Ask (realtime)

Is it correct?

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Apr 4, 2016 09:49 AM          Msg. 17 of 23
Assuming you are filtering out "O" trades and that you are using protocol 5.1, the new protocol usees different time fields.

Last (history) = Most Recent Trade (realtime) = true
Last Size (history) = Most Recent Trade Size (realtime) = true
Time Stamp (history) = Most Recent Trade Date + Most Recent Trade TimeMS (realtime) = true

A snapshot of these values are taken at the time of the trade message. The individual messages are not saved.

Bid (history) = Bid (realtime)
Ask (history) = Ask (realtime)

Tim

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Apr 4, 2016 12:45 PM          Msg. 18 of 23
'It appears this is related to an issue that occurred back in 2015 during our initial roll-out of microseconds'

Can you please expand on this...it kind of seems like a big deal.
How invalid is the post 2015 interval data?

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Apr 4, 2016 12:55 PM          Msg. 19 of 23
There were 2 days, on the 18th and 19th of November 2015, where we received trades with a second value of 60 or more from Eurex, so we are rebuilding those intraday bars, for that exchange.

Tim

Craig
-DTN Evangelist-
Posts: 214
Joined: Apr 16, 2010


Posted: Apr 4, 2016 01:19 PM          Msg. 20 of 23
Cheers.

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Apr 8, 2016 11:08 AM          Msg. 21 of 23
Hello,

Just wanted to let you know all relevant changes were made to the futures contracts affected.

Sorry for the inconvenience,

Tim

quickTick
-Interested User-
Posts: 34
Joined: Nov 17, 2013


Posted: Apr 8, 2016 02:10 PM          Msg. 22 of 23
By the way: Some minutes actually do have 61 seconds. The last one was apparently at "June 30, 2015 at 23:59:60 UTC".

https://en.wikipedia.org/wiki/Leap_second

How is this handled by IQFeed? Wouldn't want my app to crash in such a case.... :)

DTN_Tim Walter
-DTN Guru-
Posts: 1093
Joined: Apr 25, 2006


Posted: Apr 8, 2016 02:38 PM          Msg. 23 of 23
Yep, those happen every so often. They get rolled into the previous second, luckily they happen when most everything is closed.

Tim
 

 

Time: Tue December 12, 2017 7:15 PM CFBB v1.2.0 15 ms.
© AderSoftware 2002-2003