steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 7, 2012 05:12 AM
Msg. 1 of 26
Hi Steve,
I reported a similar problem earlier this year, but it seemed like the problem was on my end. However, I seem to be running into this problem again, and with more recent data, so I was wondering if you could verify if the problem is on my end, or if you see the same data that I'm seeing.
Some sample data is:
2012-09-06 15:59:28,836.20,1,102940,836.10,836.20,9328080,0,0,C, 2012-09-06 15:59:28,836.10,1,102939,836.10,836.20,9328061,0,0,C, 2012-09-06 15:59:28,836.10,1,102938,836.10,836.20,9328013,0,0,C, 2012-09-06 15:59:28,836.10,1,102937,836.10,836.20,9328013,0,0,C, 2012-09-06 15:59:28,836.10,1,102936,836.10,836.20,9328013,0,0,C, 2012-09-06 15:59:28,836.10,1,102935,836.10,836.20,9328013,0,0,C, 2012-09-06 15:59:28,836.10,1,102934,836.10,836.20,9328013,0,0,C, 2012-09-06 15:59:28,836.20,1,102933,836.10,836.20,9328002,0,0,C, 2012-09-06 15:59:28,836.10,1,102932,836.10,836.30,9327939,0,0,C, 2012-09-06 15:59:28,836.10,1,102931,836.10,836.30,9327939,0,0,C, 2012-09-06 15:59:28,836.10,1,102930,836.10,836.30,9327939,0,0,C, 2012-09-06 15:59:28,836.10,1,102929,836.10,836.30,9327939,0,0,C, 2012-09-06 15:59:28,836.10,1,102928,836.10,836.30,9327939,0,0,C, 2012-09-06 15:59:28,836.20,1,102927,836.10,836.30,9327895,0,0,C, 2012-09-06 15:59:28,836.20,1,102926,836.20,836.30,9327895,0,0,C, 2012-09-06 15:59:28,836.20,2,102925,836.20,836.30,9327895,0,0,C, 2012-09-06 15:59:28,836.30,1,102923,836.20,836.30,9327871,0,0,C, 2012-09-06 15:59:28,836.20,1,102922,836.10,836.30,9327823,0,0,C, 2012-09-06 15:59:28,836.20,1,102921,836.10,836.30,9327776,0,0,C, 2012-09-06 15:59:28,836.20,1,102920,836.10,836.30,9327760,0,0,C, 2012-09-06 15:59:28,836.20,1,102919,836.10,836.20,9327760,0,0,C, 2012-09-06 15:59:28,836.20,1,102918,836.10,836.30,9327682,0,0,C, 2012-09-06 15:59:28,836.20,1,102917,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,1,102916,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,2,102915,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,1,102913,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,3,102912,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,1,102909,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,4,102908,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.20,1,102904,836.20,836.30,9327671,0,0,C, 2012-09-06 15:59:28,836.30,1,102903,836.20,836.30,9327614,0,0,C,
I'm taking this data straight off the data stream from IQFeed, so I believe this data is correct. You can see that the tickId is being repeated many times across this same second.
Some other weird data is:
2012-08-29 20:55:00,812.40,1,9,812.10,812.90,5662,0,0,C, 2012-08-29 20:55:00,812.80,1,8,812.40,812.90,5662,0,0,C, 2012-08-29 20:55:00,812.90,1,7,0.00,0.00,5389,0,0,C, 2012-08-29 20:55:00,812.90,1,6,0.00,0.00,5389,0,0,C, 2012-08-29 20:55:00,812.90,1,5,0.00,0.00,5389,0,0,C, 2012-08-29 20:55:00,812.90,1,4,0.00,0.00,5389,0,0,C, 2012-08-29 20:55:00,812.90,2,3,0.00,0.00,5389,0,0,C, 2012-08-29 20:55:00,812.90,1,1,0.00,0.00,5389,0,0,C,
It appears that the tickId is the same for many ticks, and the bid/ask is 0.00. Is this what you see on your end?
In fact, the problem is pretty pervasive, I'm seeing literally tens of thousands of duplicate tickIds on each second, from 8/19/2012 to today.
If you can confirm that you're seeing the same data as me, I can give you the list of all the seconds that I'm seeing with duplicate tickIds, if that helps you.
thanks,
Steve
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 7, 2012 05:25 AM
Msg. 2 of 26
Just to clarify the above data is from @TFS#.
One more data point, the data appears to go weird as of 8/19/2012. I'm not sure if this date is relevant or not on your side.
Thanks for your help,
Steve
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Sep 7, 2012 08:10 AM
Msg. 3 of 26
Hi Steve1986,
I see the duplicate tick IDs that you mention and the 0's for bid and ask prices on 8-29-12. I will ask our Market Data Services to take a look and get back to you here with a response. Thanks.
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Sep 7, 2012 10:25 AM
Msg. 4 of 26
Hello Steve1986,
Thank you for bringing this to our attention. Our developers are looking at the way the data comes to us from the ICE and tell me that this situation looks more complicated than expected. We appreciate your patience and will keep you posted here.
|
DTN_Steve_S
-DTN Guru-
Posts: 2093
Joined: Nov 21, 2005
|
Posted: Sep 7, 2012 10:50 AM
Msg. 5 of 26
Just a slight update to this.
We have verified that the only information that is incorrect is the TickID field. All of the rest of the data is correct.
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 7, 2012 02:15 PM
Msg. 6 of 26
Okay great, thanks for the update Steve.
Do you know if this problem started 8/19/2012, or is this something that has been happening for a while?
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 7, 2012 02:18 PM
Msg. 7 of 26
Sorry, I mean thanks Todd and Steve :)
|
DTN_Steve_S
-DTN Guru-
Posts: 2093
Joined: Nov 21, 2005
|
Posted: Sep 7, 2012 02:32 PM
Msg. 8 of 26
Yes, this started on the weekend of the 19th and would affect all ICE exchanges (not just @TFS). We have identified the cause and are working on resolving it (hopefully this weekend).
The zero bid/ask issue on 8/29 is unrelated and is still being investigated.
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Sep 10, 2012 10:27 AM
Msg. 9 of 26
Steve1986,
The ICE exchange tells us that the 0's for bid and ask prices on 8-29 was correct. They further added: "During Pre-open the orders are not sent to the market. When the state change from pre-open to open you would receive the trades during the matching period and you would start to see bid and asks when the market transition to open state. This is as designed."
We will keep working on the other issue and report back here.
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 13, 2012 04:44 PM
Msg. 10 of 26
Thanks Todd, so if I understand it right, the ticks don't have bid or ask because they were pre-market orders that were matched on Open?
Would you happen to know if there's any progress with the duplicate tick ids?
thanks,
Steve
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Sep 14, 2012 10:51 AM
Msg. 11 of 26
Hi Steve1986,
Yes, that is correct. And we are still working on the issue of duplicate tick ids and hope to have it resolved this weekend.
I will keep you posted here.
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 14, 2012 08:00 PM
Msg. 12 of 26
Great, thanks for the update!
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Sep 28, 2012 10:08 AM
Msg. 13 of 26
Hello Steve1986,
I am glad to report that the issue of duplicate tick ids has been corrected. Thank you for your patience.
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 28, 2012 10:55 AM
Msg. 14 of 26
Great, thanks so much Todd!
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Sep 30, 2012 02:40 AM
Msg. 15 of 26
Hi Todd,
Unfortunately, I still see the same duplicate tickids for @TFS#. I just downloaded a fresh set of data going back to pre-8/19/2012, and I still see hundreds of thousands of instances of tickids being duplicated.
Can you double check to see if this problem was actually fixed?
thanks,
Steve
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Oct 1, 2012 08:12 AM
Msg. 16 of 26
Hi Steve,
The issue of duplicate tick ids has been fixed going forward, but we do not have a fix for the past tick data. If you see problems in the current data, we would be interested in knowing about that. Thanks again.
|
steve1986
-DTN Evangelist-
Posts: 119
Joined: Apr 13, 2009
|
Posted: Oct 2, 2012 05:16 PM
Msg. 17 of 26
Thanks Todd. Is there a plan to fix the past tick data, or is there no intention to fix it?
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Oct 3, 2012 07:48 AM
Msg. 18 of 26
Hi Steve1986,
There is no fix for the past tick data in this case. And, as Steve said above: We have verified that the only information that is incorrect is the TickID field. All of the rest of the data is correct.
Thanks.
|
mvvcorp
-Interested User-
Posts: 24
Joined: Apr 7, 2010
|
Posted: Oct 19, 2012 12:41 AM
Msg. 19 of 26
The same problem @CCZ12: 20.08.2012,040000,2442,1,1,0,0,714949,0,0,C, 20.08.2012,040000,2442,1,2,0,0,714949,0,0,C, 20.08.2012,040000,2442,1,3,0,0,714949,0,0,C, 20.08.2012,040000,2442,1,4,0,0,714949,0,0,C, 20.08.2012,040000,2442,1,5,0,0,714949,0,0,C
Is it bunch of numerous duplicates? should I delete\ignore them except first one?
14.09.2012,134904,2637,3,10127,2636,2639,6570900,0,0,C, 14.09.2012,134904,2636,2,10129,2636,2639,6570900,0,0,C, 14.09.2012,134904,2636,1,10130,2636,2639,6570900,0,0,C, <------------ 14.09.2012,134904,2636,1,10131,2636,2639,6570900,0,0,C, <------------ 14.09.2012,134904,2636,1,10132,2636,2639,6570900,0,0,C, <------------
Edited by mvvcorp on Oct 19, 2012 at 12:51 AM
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Oct 19, 2012 08:52 AM
Msg. 20 of 26
Hello mvvcorp,
You do not need to delete or ignore the above trades. The trades are legitimate - only the tick ids were in error. Thanks.
|
mvvcorp
-Interested User-
Posts: 24
Joined: Apr 7, 2010
|
Posted: Oct 22, 2012 11:51 AM
Msg. 21 of 26
Thx
Next question: I found multiple unordered ID's. Should I sort lines by ID or not ?
EPIX12 (raw data): 19.10.2012,161219,983.50,1,57462,982.75,983.50,12622342,0,0,C, 19.10.2012,161219,983.50,1,57463,982.75,983.50,12622585,0,0,C, 19.10.2012,161219,983.50,1,57464,982.75,983.50,12622587,0,0,C, 19.10.2012,161219,983.50,1,57465,983.50,983.75,12622584,0,0,C, 19.10.2012,161219,983.50,1,57466,983.50,983.75,12622600,0,0,C,
OR
EBK13 (raw data): 19.10.2012,115656,108.50,1,6332,108.50,108.53,17846918,0,0,C,/ 19.10.2012,115656,108.50,1,6333,108.50,108.53,17890254,0,0,C,\ --- --------------- should be here 19.10.2012,115656,108.50,3,6336,108.50,108.53,17890261,0,0,C, | 19.10.2012,115656,108.50,4,6340,108.50,108.53,17890267,0,0,C, | 19.10.2012,115656,108.50,1,6341,108.50,108.53,17890273,0,0,C, | 19.10.2012,115656,108.50,2,6343,108.50,108.53,17890279,0,0,C, | 19.10.2012,115656,108.50,3,6346,108.50,108.53,17890285,0,0,C, | 19.10.2012,115656,108.50,1,6347,108.50,108.53,17847488,0,0,C, -> unordered ID 19.10.2012,115701,108.41,1,6348,108.37,108.41,17899340,0,0,C, 19.10.2012,115710,108.40,1,6349,108.38,108.40,17903487,0,0,C,
But if I sort lines by ID, I'll get unordered cumulative volume...
So more important question: What is of greater priority: native line number OR tick ID OR cumulative volume? What should I do if they have conflict with each other?
thx
Edited by mvvcorp on Oct 22, 2012 at 12:20 PM
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Oct 22, 2012 04:07 PM
Msg. 22 of 26
Hi mvvcorp,
We are digging into this and expect to have an answer shortly. Thanks.
|
mvvcorp
-Interested User-
Posts: 24
Joined: Apr 7, 2010
|
Posted: Oct 22, 2012 08:11 PM
Msg. 23 of 26
@DXZ12: 26.09.2012,082229,80.000,1,4740,79.995,80.000,7592703,0,0,C, 26.09.2012,082229,80.000,1,4741,79.995,80.005,7656031,0,0,C, 26.09.2012,082230,80.000,2,4743,80.000,80.005,7668917,0,0,C, 26.09.2012,082243,80.005,3,4746,80.000,80.005,7585461,0,0,C, 26.09.2012,082243,80.005,1,4747,80.000,80.005,7666640,0,0,C, 26.09.2012,082243,80.005,2,4749,80.000,80.005,7668825,0,0,C, 26.09.2012,082243,80.005,2,4751,80.000,80.005,7669010,0,0,C, 26.09.2012,082243,80.005,1,4752,80.000,80.010,7669012,0,0,C, 26.09.2012,082243,80.010,1,4753,80.000,80.010,7233582,0,0,C, 26.09.2012,082243,80.010,1,4754,80.005,80.010,7665989,0,0,C, 26.09.2012,082243,80.010,1,4755,80.005,80.010,7670630,0,0,C, 26.09.2012,082243,80.015,1,4756,80.005,80.015,7500844,0,0,C, 26.09.2012,082243,80.015,1,4757,80.005,80.015,7579144,0,0,C, 26.09.2012,082243,80.015,2,4759,80.005,80.015,7584981,0,0,C, 26.09.2012,082243,80.015,2,4761,80.005,80.015,7585802,0,0,C, 26.09.2012,082243,80.015,3,4764,80.005,80.015,7606130,0,0,C, 26.09.2012,082243,80.015,1,4765,80.005,80.015,7631035,0,0,C, 26.09.2012,082243,80.015,1,4766,80.005,80.015,7666282,0,0,C, 26.09.2012,082243,80.015,1,4767,80.005,80.015,7666307,0,0,C, 26.09.2012,082243,80.015,2,4769,80.005,80.015,7670678,0,0,C, 26.09.2012,082243,80.015,2,4771,80.005,80.015,7670679,0,0,C, 26.09.2012,082243,80.015,2,4773,80.005,80.015,7670692,0,0,C, 26.09.2012,082243,80.015,3,4776,80.005,80.020,7670693,0,0,C, 26.09.2012,082243,80.020,1,4777,80.005,80.020,7579151,0,0,C, 26.09.2012,082243,80.020,2,4779,80.005,80.020,7585779,0,0,C, 26.09.2012,082243,80.020,2,4781,80.005,80.020,7606150,0,0,C, 26.09.2012,082243,80.020,3,4784,80.005,80.020,7624726,0,0,C, 26.09.2012,082243,80.020,1,4785,80.005,80.020,7666409,0,0,C, 26.09.2012,082243,80.020,1,4786,80.005,80.020,7668682,0,0,C, 26.09.2012,082243,80.020,19,4805,80.005,80.025,7670690,0,0,C, 26.09.2012,082243,80.010,1,4806,80.005,80.020,7670720,0,0,C,
|
mvvcorp
-Interested User-
Posts: 24
Joined: Apr 7, 2010
|
Posted: Oct 22, 2012 09:57 PM
Msg. 24 of 26
Well, and finally what should I do in this case? :
@CTZ12_0: 14.08.2012,211325,72.91,1,233,72.84,72.91,215704,0,0,C, 14.08.2012,211347,72.92,1,234,72.84,72.92,215718,0,0,C, 14.08.2012,211408,72.86,1,235,72.86,72.94,215737,0,0,C, 14.08.2012,211630,72.84,1,236,72.84,72.89,215821,0,0,C, 14.08.2012,210000,73.15,1,236,72.75,72.84,215922,0,0,C, <--------------!!! 14.08.2012,212115,72.78,1,237,72.78,72.84,216104,0,0,C, 14.08.2012,212156,72.77,1,238,72.77,72.82,216140,0,0,C, 14.08.2012,212241,72.82,1,239,72.78,72.82,216159,0,0,C, 14.08.2012,212247,72.84,6,245,72.82,72.84,216168,0,0,C, 14.08.2012,212254,72.84,8,253,72.82,72.84,216172,0,0,C,
Yes, we have ordered IDs but obviously there is wrong ID for the line starting with "14.08.2012,210000..."! Apparently this line should be moved to another place (much more earlier).
one more case: @ESZ12: 08.10.2012,104715,1451.00,1,461593,1450.75,1451.00,1817180,0,0,C, 08.10.2012,104715,1451.00,3,461596,1450.75,1451.00,1817181,0,0,C, 08.10.2012,104715,1451.00,3,461599,1450.75,1451.00,1817182,0,0,C, 08.10.2012,104716,1451.00,3,461602,1450.75,1451.00,1817183,0,0,C,<----------- ??? 08.10.2012,104715,1451.00,2,461604,1450.75,1451.00,1817184,0,0,C, 08.10.2012,104715,1451.00,1,461605,1450.75,1451.00,1817185,0,0,C, 08.10.2012,104715,1451.00,2,461607,1450.75,1451.00,1817186,0,0,C, 08.10.2012,104715,1451.00,1,461605,1450.75,1451.00,1817185,0,0,C,
M.NI3=LX: 03.04.2012,080346,18379.00,1,1416,18362.00,18382.00,42747,0,0,C, 03.04.2012,080346,18379.00,2,1418,18362.00,18382.00,42750,0,0,C, 03.04.2012,080346,18379.00,2,1420,18362.00,18382.00,42755,0,0,C, 03.04.2012,090414,18375.00,0,1420,18364.00,18379.00,42807,0,0,C,<----------- ??? 03.04.2012,080420,18379.00,1,1421,18363.00,18381.00,42815,0,0,C, 03.04.2012,080420,18380.00,1,1422,18363.00,18381.00,42816,0,0,C, 03.04.2012,080439,18381.00,1,1424,18362.00,18390.00,42883,0,0,C, 03.04.2012,080439,18382.00,3,1427,18362.00,18390.00,42884,0,0,C, 03.04.2012,080456,18390.00,2,1429,18370.00,18390.00,42896,0,0,C, 03.04.2012,090457,18380.00,0,1429,18371.00,18380.00,42901,0,0,C,<----------- ??? 03.04.2012,080531,18387.00,1,1430,18377.00,18390.00,42970,0,0,C, 03.04.2012,080551,18367.00,1,1432,18367.00,18390.00,42987,0,0,C, 03.04.2012,080951,18387.00,1,1433,18381.00,18394.00,43036,0,0,C, 03.04.2012,081000,18392.00,1,1434,18371.00,18394.00,43043,0,0,C,
MTV12: 17.09.2012,103007,3552.00,4,54306,3552.00,3552.50,491456,0,0,C, 17.09.2012,103010,3552.50,1,54307,3552.00,3553.00,491902,0,0,C, 17.09.2012,103010,3552.50,1,54304,3552.00,3553.00,491903,0,0,C, 17.09.2012,103033,3552.50,1,52531,3552.00,3553.00,492427,0,0,C, 17.09.2012,103033,3552.00,4,52535,3552.00,3553.00,492428,0,0,C, 17.09.2012,103106,3552.00,1,52536,3551.50,3552.50,493270,0,0,C, <----------- 17.09.2012,103053,3546.50,1773,54309,3552.00,3552.50,493291,0,0,C,<----------- 17.09.2012,103145,3553.00,5,50768,3553.00,3553.50,494112,0,0,C, 17.09.2012,103145,3553.00,5,50773,3552.50,3553.00,494114,0,0,C, 17.09.2012,103149,3552.00,1,50774,3551.00,3552.50,494449,0,0,C, 17.09.2012,103149,3552.00,1,50770,3552.00,3552.50,494451,0,0,C, 17.09.2012,103158,3552.00,1,50771,3551.50,3552.50,495049,0,0,C, 17.09.2012,103158,3552.00,1,50771,3552.00,3552.50,495051,0,0,C, 17.09.2012,103202,3552.50,1,50772,3552.00,3553.00,495169,0,0,C,
QCLJ12: 13.02.2012,140110,100.73,1,74293,100.73,100.75,1333152,0,0,C, 13.02.2012,140110,100.73,1,74294,100.71,100.73,1333261,0,0,C, 13.02.2012,140110,100.73,1,74295,100.71,100.73,1333275,0,0,C, 13.02.2012,140111,100.73,1,74296,100.77,100.79,1333374,0,0,C, 13.02.2012,140110,100.73,1,74297,100.77,100.79,1333394,0,0,C, 13.02.2012,140006,100.77,1,74296,100.78,100.80,1333706,0,0,C,<----------??? 13.02.2012,140014,100.77,1,74233,100.76,100.77,1333856,0,0,C, 13.02.2012,140025,100.76,1,74234,100.74,100.76,1334045,0,0,C, 13.02.2012,140110,100.73,1,74235,100.74,100.77,1334085,0,0,C, 13.02.2012,140111,100.73,1,74296,100.76,100.78,1334177,0,0,C, Edited by mvvcorp on Oct 23, 2012 at 08:34 AM
|
DTN_ToddH
-DTN Guru-
Posts: 287
Joined: Oct 6, 2011
|
Posted: Oct 25, 2012 01:05 PM
Msg. 25 of 26
mvvcorp,
Trades should not be arranged by Tick id as the tick ids are supposed to be unique to each trade for each day, but are not necessarily sequential or incremental.
|
mvvcorp
-Interested User-
Posts: 24
Joined: Apr 7, 2010
|
Posted: Oct 27, 2012 06:02 PM
Msg. 26 of 26
thx!
|
|
|
|