I have been seeing invalid and stale market data in various insturments (CL, ES) in the last month or so (could be slightly longer). It occurs at seemingly random times and usually lasts about 10 minutes sometimes up to an hour.
Example from today: 10:08-10:20am, 3:05-3:15pm CST in ES. Here is a snapshot of L1 updates from one of the periods:

Q,@ESZ18,2738.25,3859981,2,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859982,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859983,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@ESZ18,2738.25,3859984,1,2738.00,2733.75,64,1,183,16:08:31.779471,271740673,16:08:31.770457,16:05:00.164499,C,
Q,@EDU20,96.7650,270309,2,96.7650,96.7700,8665,15942,,16:07:28.306149,9268532,16:08:31.734942,16:08:31.768035,a,
Q,@EDM19,96.9800,405656,2,96.9800,96.9850,7411,30599,,16:00:00.036589,9268244,16:08:31.753186,16:08:31.768255,a,
Q,@EDU20,96.7650,270309,2,96.7650,96.7700,8665,15824,,16:07:28.306149,9268532,16:08:31.734942,16:08:31.768255,a,
Q,@EDH20,96.7650,342864,2,96.7650,96.7700,4985,27436,,16:07:28.306149,9268541,16:08:31.734935,16:08:31.768255,a,
Q,@EDM20,96.7600,348765,3,96.7600,96.7650,6401,20909,,16:07:28.306149,9268536,16:08:31.733403,16:08:31.768255,a,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,13,1,,16:08:31.779471,271740673,16:08:31.779481,16:05:00.164499,b,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,14,1,,16:08:31.779471,271740673,16:08:31.779658,16:05:00.164499,b,
Q,@ESZ18,2738.25,3859984,1,2738.25,2733.75,15,1,,16:08:31.779471,271740673,16:08:31.779676,16:05:00.164499,b,

In the above (and with my message structure) the best bid/ask data are crossed and indicating 2738.25,2733.75 for bid and ask for ES, clearly nonsensical. This same stale/invalid bid/ask info stays such for the period in question. Notice that ED above (eurodollar) is just fine as are most other contracts. This has been happening for over a month and I alerted iqfeed support to it. Initially they expressed it was due to testing MBO feed. When I contended those reasons they claimed they had some issues with CME feed. Whatever it is, this is not getting fixed and I am concerned this is not getting the priority it deserves. Any advanced developer depends on full/complete/correct data-as the feed proclaims to provide. Please correct this issue. I can provide more details if needed.

Good morning,

I apologize that this continues to happen and for my poor communication of our concern over it. We are very concerned with the quality of our data. To that end, here we have added the CME trade aggressor flag to our level one data. This flag tells if the trade occurred at the bid or the ask, so not only will you save on the processing, but you will have a definite answer on every trade of where it occurred. We hope to have this available to our developers later this month as a beta, but 4th quarter for sure. We have also developed better ways to identify when these situations occur, so that we can ensure quality and strive towards a continuous improvement to our data.

I know that the people that are needed to insure the continued priority of this work actively watch the forums and I know that your concerns will only help to keep the focus where it needs to be. I will try to keep you more in the loop in the future when related updates occur.

Tim

I am also still seeing this issue. Sometimes restarting the instance of iqfeed resolves the issue, but often times it does not. I've been noticing that for many instruments that trade on the CME, a bid or ask will get stuck at a particular price for several hours.

Constantly having these bad prices is making the data effectively unusable. Do you have an ETA on when the CME data will become usable again?

Today was particularly bad, CL and currency data (CL, 6E, 6A,6J) all had this issue for hours after 9:31am central time.

I agree that Friday was particularly bad. The entire strip November expiring ESZ8 options was giving me bad prices, which was throwing off calculations we use to drive our model.

Hello, we are continuing to monitor and make changes to address this issue. Friday was particularly bad since we had several different issues contributing to make the problem worse. We made some configuration changes on our servers over the weekend and continue to actively work with our providers to get the issue resolved.

Has this been resolved?

---

MAC

It hasn't, for example today (2/11/2019) it occurred in CL, ES, 6E between 10:18-10:23am CST.

I believe I saw it at about the same time today, for ES.

Today as in 2019-02-11 CST, that is.

As of right, now, I also see the gap in the historical tick data. Or maybe that's where I saw it in the first place.

The quotes got stuck at 2707.50, trades do continue (in the historical tick data).

aQuant or quickTick,

Could you explain how you detect this? I want to start monitoring.

Is there a complete lack of Q messages arriving with 'b' or 'a' updates?

or

Are the Q messages coming in for 'b' or 'a' but the prices don't change?

I have disabled IQFeed from my production deployment and will run it in the background for monitoring only.

I was about to add NYMEX, COMEX, CBOT and EUREX to my subscriptions. I will wait until this has been resolved.

---

MAC

Based on the raw level1 messages it appears that one side stops updating, in the below case it was bid that got 'stuck' at 51.51, there may be other ways this happens, I didn't study them all:

Q,QCLH19,51.36,465114,2,51.51,51.37,1,46,183,11:19:10.391894,11321247,11:17:37.737313,11:19:10.391882,C,
Q,QCLH19,51.36,465114,2,51.51,51.37,1,47,,11:19:10.391894,11321247,11:17:37.737313,11:19:10.392009,a,
Q,QCLH19,51.36,465115,1,51.51,51.37,1,47,183,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392009,C,
Q,QCLH19,51.36,465115,1,51.51,51.37,1,48,,11:19:10.392111,11321248,11:17:37.737313,11:19:10.392418,a,
Q,QCLH19,51.36,465116,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,48,183,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392418,C,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,49,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392538,a,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,50,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.392633,a,
Q,QCLH19,51.36,465117,1,51.51,51.37,1,51,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393159,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,6,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.393887,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,7,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395510,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,8,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395553,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,10,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395590,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,13,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395627,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,14,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395727,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,15,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395840,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,18,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.395916,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,20,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396142,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,22,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396215,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,23,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396250,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,24,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396328,a,
Q,QCLH19,51.36,465117,1,51.51,51.36,1,25,,11:19:10.392432,11321249,11:17:37.737313,11:19:10.396725,a,

@ aQuant

I'm not sure about which field is which, but it doesn't seem to show the beginning.
Do quotes get crossed because they are stuck, or do they get stuck because they are crossed?

aQuant,

I am now monitoring the following CME symbols (AD,BP,CD,JY,EC,NG,ES).

I didn't see anything out of the ordinary for trader date 2018-02-20. However, my monitoring is only gather stats and not saving the raw messages. There is a possibility I have a bug in my stats.

If it's not too much to ask, could you post here when you see one of these symbols get stuck or cross? I can then verify with my monitoring.

If I can verify, we can all push for a solution.

Thanks in advance.

---

MAC

In the post above yours I didn't show where it began, I would have to look for that. I can try to locate it later.
I am definitely willing to supply a lot more information to anyone who will as a result help find a solution. But I would like to see some involvement of IQFeed support in this. I can help them nail this if they really want to solve it...

aQuant,

I spoke too soon. I see the bid frozen from 2018-02-21 02:00:00 to 02:17:00 ET on all the products I'm monitoring except JY. It looks like my simple stats are able to catch this.

I haven't coded up the HT* command (request tick history). Will this show the same behavior? If so, surely IQFeed won't be able to refute the problem.

I will call them this morning.

---

MAC

Attached is the HTT response for @NQH19.

Note that the bids do not change for 17 minutes.

---

MAC

We continue to actively monitor this thread and the information you are providing is being utilized by our network, development and operations teams. aQuant, regarding your comment “I would like to see some involvement of IQFeed support in this… if they really want to solve it” – the answer is absolutely yes. We are here to support our customers and continually improve our products. Feedback from each of you is critical to our continued success.

There is a lot of action going on behind the scenes at DTN to further optimize our service and increase resiliency. This involves not only our staff, but teams in the middle and all the way through to the exchanges. In fact, this morning we received word that the CME is performing maintenance on fiber connections that we believe could be responsible sporadic dropped packets that may result in the reported stuck bids. Please continue to share your findings by sending them directly to developer support, and we will continue to work with you to resolve any issues you are experiencing.

@ mac

In your text file, the bid keeps the value it first acquires at 02:01:12.834878, at the value of 7091.75.
That's about a screen down from the beginning.
Right in the next line, the bid gets crossed by both ask and trade. Perhaps that is not a coincidence that it happens so close.

Perhaps the bid gets stuck because bid and ask cross. For example, somewhere might be code like "if ask < bid", perhaps even as a safety guard, and that code causes it to get stuck at that value. If so, bid and ask probably shouldn't cross in the first place, but that might be just a delay in the delivery of one vs the other.

Just a suggestion, since I don't see that on my own screen displaying the historical tick data for NQ, as far as I can tell.

@quickTick,

It's interesting that you don't see this. The attached file is the raw text data from the socket. I've attached the full response to the HTT request run at 2018-02-21 21:20:00 ET. It's a bit longer but still shows the frozen bid.

I should note, I have yet to upgrade to IQConnect v6. I will do that now and see if it fixes the problem.

Below is the HTT command I issued.

HTT,@NQH19,20190221 015000,20190221 022000,0005000,000000,235959,1,REQ_HST_TCK0000000,0000500

Same result without including BeginFilterTime and EndFilterTime in the request

HTT,@NQH19,20190221 015000,20190221 022000,0005000,,,1,REQ_HST_TCK0000000,0000500

---

MAC

I just upgraded to IQConnect v6 and the bid is no longer frozen. I am still using 5.2 protocol. (see attached file)

@aQuant, which version of IQConnect are you running? If v6, I will assume the frozen bid I saw 2am on Feb 21 was version related.

I will continue to monitor with IQConnect v6.

I hope this finding will shed some light on the issue.

---

MAC

I am using 5.2.5.0.

I am seeing this issue again today in ESU0 contract, roughly between 11:40am and 12:07pm CST. Support, could you please look into it?

Do you mean the symbol @ESU20? In looking at the historical data for the time you give, I don't see any cases where the Bid exceeds the Ask. Can you give an example?

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Yes, but I am referring to real-time L1 data stream. I just pulled the raw data as received for ES, here it is for a few lines:
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,491,186,183,13:01:51.552581,27215001,13:01:50.321955,12:59:27.926377,C,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.552581,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,488,186,,13:01:51.552581,27215001,13:01:51.552818,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.564067,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.788146,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.794049,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:51.805153,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,489,186,,13:01:51.552581,27215001,13:01:51.964915,12:59:27.926377,b,
Q,@ESU20,3229.50,881690,1,3229.50,3228.75,490,186,,13:01:51.552581,27215001,13:01:52.506262,12:59:27.926377,b,
Q,@ESU20,3229.50,881691,1,3229.50,3228.75,490,186,183,13:01:52.787067,27215002,13:01:52.506262,12:59:27.926377,C,
Q,@ESU20,3229.50,881691,1,3229.50,3228.75,489,186,,13:01:52.787067,27215002,13:01:52.787067,12:59:27.926377,b,
Q,@ESU20,3229.50,881691,1,3229.50,3228.75,488,186,,13:01:52.787067,27215002,13:01:52.787329,12:59:27.926377,b,
Q,@ESU20,3229.50,881691,1,3229.50,3228.75,491,186,,13:01:52.787067,27215002,13:01:53.235752,12:59:27.926377,b,
Q,@ESU20,3229.50,881691,1,3229.50,3228.75,490,186,,13:01:52.787067,27215002,13:01:53.328002,12:59:27.926377,b,
Q,@ESU20,3229.75,881693,2,3229.50,3228.75,490,186,173,13:01:53.354236,27215003,13:01:53.328002,12:59:27.926377,C,
Q,@ESU20,3229.75,881693,2,3229.50,3228.75,491,186,,13:01:53.354236,27215003,13:01:53.354478,12:59:27.926377,b,
Q,@ESU20,3229.75,881693,2,3229.50,3228.75,491,186,,13:01:53.354236,27215003,13:01:53.456365,12:59:27.926377,b,
Q,@ESU20,3229.75,881693,2,3229.50,3228.75,490,186,,13:01:53.354236,27215003,13:01:53.465140,12:59:27.926377,b,
Q,@ESU20,3229.75,881693,2,3229.50,3228.75,487,186,,13:01:53.354236,27215003,13:01:54.535890,12:59:27.926377,b,

This indicates bid and ask of 3229.50 and 3228.75 respectively, so crossed. Note that ask size is 'frozen' at 186 and not updating. It's a case of stale ask in this case.

DTN had a brief internal issue receiving CME data today, at this time. I will add your report to our internal investigation.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Hi, is it normal that some fields arrive empty? Is this related to the internal CME issue?

Q,QHGN20,2.9050,11:04:24.794165,07/29/2020,2.7450,,193,1
Q,QNGQ20,1.882,14:29:24.179276,07/29/2020,1.615,,3645,1

I'm not sure what you mean by "empty". But those two ticks are settlements, not trades. The symbols are options that expired at the end of the day. When that happens, the exchanges send a "settlement" transaction, which shows up as a tick. The Most Recent Trade Conditions field tells you if a tick represents a trade, or something else.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist