Joined: |
Jul 20, 2012 05:05 PM |
Last Post: |
Sep 21, 2023 05:04 PM |
Last Visit: |
Sep 22, 2023 04:40 PM |
Website: |
|
Location: |
|
Occupation: |
|
Interests: |
|
|
AIM: |
|
ICQ: |
|
MSN IM: |
|
Yahoo IM: |
|
|
aQuant has contributed to 49 posts out of 21251 total posts
(0.23%) in 4,443 days (0.01 posts per day).
20 Most recent posts:
I don't think adding orderID would solve it, and in any case, it means adding an optional field-which is what I suggested with the sequence ID. They already are parsing the MBO feed and therefore must be having access to sequence ID. Maybe it's a huge undertaking to store/pass to client as an optional field...
I have been running it for years from Sunday night to Friday night, i.e. a full trading week. No issues. The client has a very small memory footprint.
Great, I am glad it's being debated at least. Thank you.
Hello,
as I originally requested this (and subsequently inquired) I thought I would add this note. It's not true that MBO or any existing enhancements in 6.2 will 'obviate' this request that I posted years ago and never got any good response on it. I can provide specific real examples as to why it's impossible to synchronize the two distinct L1 and L2 (MBO or MBP) data feeds in the current approach-even if they all carry precise time stamps to microsecond (which they currently do for CME). I outlined (in my 10/23/2019 post) how this option could be offered relatively easily (with most burden offloaded to the interested developers-however, making it at least theoretically possible to synchronize the two feeds). Even more so confounding that you must already be parsing this field in the process. In summary, any sophisticated analysis that requires the knowledge of trade match events and their relation to order book changes does need this and to be able to do it deterministically and with guaranteed correctness the two feeds must be obtained with sequential IDs (even if on different channels/threads as is currently the case).
Is the depth data available for this product? I am only seeing a 'q,QMCLQ21' message in MBO feed.
Yes, but I am referring to real-time L1 data stream. I just pulled the raw data as received for ES, here it is for a few lines: Q,@ESU20,3229.50,881690,1,3229.50,3228.75,491,186,183,13:01:51.552581,27215001,13:01:50.321955,12:59:27.926377,C, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.552581,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,488,186,,13:01:51.552581,27215001,13:01:51.552818,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.564067,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.788146,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.794049,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.805153,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.964915,12:59:27.926377,b, Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:52.506262,12:59:27.926377,b, Q,@ESU20,3229.50,881691,1,3229.50,3228.75,490,186,183,13:01:52.787067,27215002,13:01:52.506262,12:59:27.926377,C, Q,@ESU20,3229.50,881691,1,3229.50,3228.75,489,186,,13:01:52.787067,27215002,13:01:52.787067,12:59:27.926377,b, Q,@ESU20,3229.50,881691,1,3229.50,3228.75,488,186,,13:01:52.787067,27215002,13:01:52.787329,12:59:27.926377,b, Q,@ESU20,3229.50,881691,1,3229.50,3228.75,491,186,,13:01:52.787067,27215002,13:01:53.235752,12:59:27.926377,b, Q,@ESU20,3229.50,881691,1,3229.50,3228.75,490,186,,13:01:52.787067,27215002,13:01:53.328002,12:59:27.926377,b, Q,@ESU20,3229.75,881693,2,3229.50,3228.75,490,186,173,13:01:53.354236,27215003,13:01:53.328002,12:59:27.926377,C, Q,@ESU20,3229.75,881693,2,3229.50,3228.75,491,186,,13:01:53.354236,27215003,13:01:53.354478,12:59:27.926377,b, Q,@ESU20,3229.75,881693,2,3229.50,3228.75,491,186,,13:01:53.354236,27215003,13:01:53.456365,12:59:27.926377,b, Q,@ESU20,3229.75,881693,2,3229.50,3228.75,490,186,,13:01:53.354236,27215003,13:01:53.465140,12:59:27.926377,b, Q,@ESU20,3229.75,881693,2,3229.50,3228.75,487,186,,13:01:53.354236,27215003,13:01:54.535890,12:59:27.926377,b,
This indicates bid and ask of 3229.50 and 3228.75 respectively, so crossed. Note that ask size is 'frozen' at 186 and not updating. It's a case of stale ask in this case.
I am seeing this issue again today in ESU0 contract, roughly between 11:40am and 12:07pm CST. Support, could you please look into it?
I noticed a related issue. At night (~7pm CST) the L2 feed often lags the L1 feed, this is at a time when the activity is fairly low. Did anyone notcice similar? I have seen this for a few nights in a row now.
Done.
I requested this 5 years ago... but I understand the separation of L1 and L2 feeds is probably important and convenient organization of the feed data.
However, can we request the messages to contain a new field, message sequence ID, that comes directly from the exchange for both L1 and L2 update messages? This would allow the client to synchronize on their end. And you must be already parsing those from say CME in your data processing anyway. You could just pass them along (as optional field) to the client. See https://www.cmegroup.com/confluence/display/EPICSANDBOX/MDP+3.0+FIX+Tag+Library and tag 83 message field described as: Tag: RptSeq FIX Type: Int Description: MD Entry sequence number per instrument update. Reset weekly.
Hi Stephen,
I emailed my request to support as instructed. Thanks.
I agree, IQFeed developers should be able to handle binary protocol just as easily and it should be much more efficient to transfer to clients too. Especially with the hopefully coming soon (?) Market By Order depth protocol this would seem quite desirable to limit the number of bytes sent over the network.
I am using 5.2.5.0.
In the post above yours I didn't show where it began, I would have to look for that. I can try to locate it later. I am definitely willing to supply a lot more information to anyone who will as a result help find a solution. But I would like to see some involvement of IQFeed support in this. I can help them nail this if they really want to solve it...
Based on the raw level1 messages it appears that one side stops updating, in the below case it was bid that got 'stuck' at 51.51, there may be other ways this happens, I didn't study them all:
Q,QCLH19,51.36,465114,2,51.51,51.37,1,46,183,11:19:10.391894,11321247,11:17:37.737313,11:19:10.391882,C, Q,QCLH19,51.36,465114,2,51.51,51.37,1,47,,11:19:10.391894,11321247,11:17:37.737313,11:19:10.392009,a, Q,QCLH19,51.36,465115,1,51.51,51.37,1,47,183,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392009,C, Q,QCLH19,51.36,465115,1,51.51,51.37,1,48,,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392418,a, Q,QCLH19,51.36,465116,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C, Q,QCLH19,51.36,465117,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C, Q,QCLH19,51.36,465117,1,51.51,51.37,1,49,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392538,a, Q,QCLH19,51.36,465117,1,51.51,51.37,1,50,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392633,a, Q,QCLH19,51.36,465117,1,51.51,51.37,1,51,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393159,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,6,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393887,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,7,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395510,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,8,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395553,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,10,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395590,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,13,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395627,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,14,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395727,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,15,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395840,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,18,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395916,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,20,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396142,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,22,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396215,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,23,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396250,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,24,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396328,a, Q,QCLH19,51.36,465117,1,51.51,51.36,1,25,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396725,a,
It hasn't, for example today (2/11/2019) it occurred in CL, ES, 6E between 10:18-10:23am CST.
Here is an example of a few situations from yesterday where best bid is reported above best ask (MD01 updates below). This is from 1/28/2019 CLH19 instrument.
QCLH19,MD01,52.99,52.89,4,5,01:41:26.756957,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD02,52.98,52.9,10,1,01:41:26.756957,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD03,52.87,52.91,57,13,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD04,52.86,52.95,67,8,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD05,52.85,52.96,70,3,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD06,52.84,52.97,53,9,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD07,52.83,52.98,90,3,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD08,52.82,52.99,81,3,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD09,52.81,53.,48,35,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,F, QCLH19,MD10,52.8,53.01,109,23,01:41:26.757060,2019-01-28,52,01:41:26.757060,T,T,T, QCLH19,MD01,52.99,52.89,4,6,01:41:26.756957,2019-01-28,52,01:41:26.757075,T,T,T,
QCLH19,MD01,52.85,52.84,31,2,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD02,52.82,52.85,27,56,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD03,52.81,52.86,55,33,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD04,52.8,52.87,50,42,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD05,52.79,52.88,36,26,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD06,52.78,52.89,40,31,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD07,52.77,52.9,38,70,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD08,52.76,52.91,34,31,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD09,52.75,52.92,47,42,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,F, QCLH19,MD10,52.74,52.93,37,47,07:03:29.613372,2019-01-28,52,07:03:29.613372,T,T,T, QCLH19,MD02,52.82,52.85,27,6,07:03:29.613372,2019-01-28,52,07:03:29.613386,T,T,F, QCLH19,MD03,52.81,52.86,55,22,07:03:29.613372,2019-01-28,52,07:03:29.613386,T,T,T, QCLH19,MD03,52.81,52.86,55,4,07:03:29.613372,2019-01-28,52,07:03:29.613403,T,T,T,
QCLH19,MD01,52.42,52.41,3,2,09:07:55.677653,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD02,52.41,52.43,10,15,09:07:55.677676,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD03,52.39,52.44,37,45,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD04,52.38,52.45,61,41,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD05,52.37,52.46,41,49,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD06,52.36,52.47,60,46,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD07,52.35,52.48,98,42,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD08,52.34,52.49,40,33,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD09,52.33,52.5,47,37,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,F, QCLH19,MD10,52.32,52.51,62,24,09:07:55.677684,2019-01-28,52,09:07:55.677684,T,T,T,
Yes, I am aware of the Bid/AskInfoValid flag. Note this line though:
QCLH19,MD10,53.05,54.27,90,14,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,T,
which indicates end of group updates. It still has the 54.27 ask (this time all are flagged valid). This ask is definitely not valid, about 100 ticks off of correct price.
I often see incorrect/nonsensical L2 data right after a large trade takes out a few levels on the DOM (depth of market). For example, this is from CLH9 1/22/2019 at 06:01.44.16260am EST the ask price is off by almost hundred ticks for a few updates in a row. Then at 06:01:44.162895 it 'comes back' to normal.
I have other examples, where crossed bid/ask MD0x levels are reported for a number of updates (typically following a trade that sweeps a few levels). Is this something that can be fixed?
QCLH19,MD01,53.13,53.15,10,18,06:01:44.162613,2019-01-22,52,06:01:44.021207,T,T,F, QCLH19,MD02,53.12,53.16,34,25,06:01:44.162613,2019-01-22,52,06:01:43.970119,T,T,F, QCLH19,MD03,53.11,53.17,54,22,06:01:44.162613,2019-01-22,52,06:01:43.970383,T,T,F, QCLH19,MD04,53.1,53.18,76,28,06:01:44.162613,2019-01-22,52,06:01:43.883353,T,T,F, QCLH19,MD05,53.09,53.19,32,28,06:01:44.162613,2019-01-22,52,06:01:43.874302,T,T,F, QCLH19,MD06,53.08,53.2,37,22,06:01:44.162613,2019-01-22,52,06:01:44.011207,T,T,F, QCLH19,MD07,53.07,53.21,80,23,06:01:44.162613,2019-01-22,52,06:01:44.037856,T,T,F, QCLH19,MD08,53.06,53.22,38,29,06:01:44.162613,2019-01-22,52,06:01:44.037856,T,T,F, QCLH19,MD09,53.05,53.23,90,18,06:01:44.162613,2019-01-22,52,06:01:43.874302,T,T,F, QCLH19,MD10,53.04,53.24,39,24,06:01:44.162613,2019-01-22,52,06:01:43.874302,T,T,T, QCLH19,MD01,54.18,54.18,0,3886,06:01:44.162620,2019-01-22,52,06:01:44.162620,F,T,F, QCLH19,MD02,53.13,54.19,10,9,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD03,53.12,54.2,10,23,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD04,53.11,54.21,41,26,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD05,53.1,54.22,76,15,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD06,53.09,54.23,32,14,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD07,53.08,54.24,37,8,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD08,53.07,54.25,80,22,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD09,53.06,54.26,38,8,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,F, QCLH19,MD10,53.05,54.27,90,14,06:01:44.162620,2019-01-22,52,06:01:44.162620,T,T,T, QCLH19,MD01,53.11,53.12,49,19,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD02,53.1,53.13,95,1,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD03,53.09,53.15,32,18,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD04,53.08,53.16,37,25,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD05,53.07,53.17,80,22,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD06,53.06,53.18,38,28,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD07,53.05,53.19,90,28,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD08,53.04,53.2,39,22,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD09,53.03,53.21,55,23,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,F, QCLH19,MD10,53.02,53.22,48,29,06:01:44.162895,2019-01-22,52,06:01:44.162895,T,T,T,
Today was particularly bad, CL and currency data (CL, 6E, 6A,6J) all had this issue for hours after 9:31am central time.
|
|