This message was accidently posted to the "IQFeed developer wishlist forum" so I have duplicated it here.
-------------------------------------------------------------------------------------------
I have been trying for days to correct NASDAQ data (in the form of update records) but have been foiled at every turn. As anyone who has dealt with exchanges around the world will know, data that is not filtered in any way (eg the IQFeed service) will have incorrect data. Common errors include values such as zero for a bid/ask/trade and a value such as $223.1 instead of $22.31 (misplaced decimal point). Some data feeds implement correction algorithms and I am attempting to do this. However, to do this effectively when the data is being delivered via an inherently unreliable medium such as the Internet you need to know when some transactions have been missed.

IQFeed supplies two fields that can assist:

field 56 (number of trades today) - I'll call this trade_count
field 7 (total volume) - I'll call this day_volume

trade_count appears to be generated locally. I am unsure whether day_volume is server generated (I hope it is).

The problem is that both of these fields can jump around unexpectedly. For example, I have seen trade_count jump backwards from 2741 to 2034. It then continues normally (ie 2035, 2036, ...). Day_volume can also jump around, both forwards and backwards. I have seen it jump forward a large amount, proceed normally for a few hundred updates, and then jump back to the earlier value! They usually don't jump around at the same point. These unusual movements are NOT related to missed updates. Unfortunately, from the perspective of tick correction, these jumps often occur at the same time as incorrect data is received......I don't think this is a coincidence.

I have just got to the point where I can deal with the high level of corrupt data (discussed elsewhere in these forums) that is dished out by the IQFeed client and this is just one more setback. Tick correction is VITAL to active traders as incorrect data can easily trigger a BUY or SELL order.

So IQFeed developers, WHERE is this data coming from? Why is it jumping around?

Thanks, Dennis

PS please don't tell me to get a faster machine or increase my buffer size, etc

As a follow up to my previous posting, I have additional information that sheds light on the issue.

The update record has an additional field 50 (Previous days volume) that also changes sometimes between ticks. As this should never happen, further analysis revealed that the Update record had MERGED DATA from different symbols!!! Therefore I could get the price for one symbol but a trade count, daily volume and previous days volume for another symbol.

On Dec 7th I had internet issues which resulted in a huge number of these types of errors. Similar to the corrupt data problems reported earlier (bug number 2750) it seems that the IQFeed client falls over badly under load or when it receives bad data. Can someone from DTN/IQFeed explain to me why this bug isn't considered CRITICAL? Having a bug list is fine but without proper prioritisation and escalation it's not that helpful.

CAN I PLEASE GET A RESPONSE FROM DTN/IQFEED. I have lost over a month of analysis work due to this problem.

dhakme

PS I have IQFeed client logs (level 5) as well as my own logs of every update received.

The problem you are seeing is due to the amount of data you're watching and the speed of your connection. Currently if the server cannot send a full update (full receive buffer at the client) it discards the remainder of the message. This is done to prevent performance and resource impact on the servers from clients. It means that the buffer in IQFeed will contain an update fragment. When the next message is received it will concatenate to the message fragment and result in a corrupt message.
We are currnetly working on a change in both the servers and IQFeed to correct this.
Incidently we are also working on tick correction, unfortunately I don't have a hard release date yet.

---

Lonnie Shumate
Development Manager, IQ Systems
DTN Market ACCESS