I have noticed that the historical tick data does not match the ticks I have collected in real time.
I know that the real time ticks are complete because there is a running aggregated volume per symbol as well as a counter (per symbol). However, there are often additional trades in the historical data (never the other way round).

Where are theses trades coming from?
They don't seem to exist in any form in the streaming data.


Dennis

Dennis,

If you could give me an example of a symbol you see this happening on, I can look into this.

Thanks,

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Tim,
An example,
Date: October 12, 2004
symbol: IACI
timestamp: 13:00

The historical trades include three trades in a row of 2200, 2460 and 2800 shares.
These trades do not exist in the real time data. In the data below, the first three trades
are okay and then things go wrong:

REAL TIME DATA
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,600,3046534,4663
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046634,4664
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046734,4665
2004-10-12 13:00:30,IACI,20.59,20.60,20.60,100,3046834,4666
2004-10-12 13:00:33,IACI,20.57,20.58,20.58,100,3046934,4667
2004-10-12 13:00:38,IACI,20.57,20.58,20.57,885,3047819,4668

DOWNLOADED HISTORICAL
IACI,0,20041012,13:00,20.600000,600
IACI,0,20041012,13:00,20.600000,100
IACI,0,20041012,13:00,20.600000,100
IACI,0,20041012,13:00,20.600000,2200
IACI,0,20041012,13:00,20.600000,2460
IACI,0,20041012,13:00,20.600000,2800
IACI,0,20041012,13:00,20.590000,300
IACI,0,20041012,13:00,20.590000,200
IACI,0,20041012,13:00,20.590000,885

Dennis,

Looking at our stored historical data vs what you sent, I see that those trades received were regular trades. Moreover, the volume on these trades (which is part of the historcal retrieval) is higher than the volume you have.

The obvious reason for this is that data is being dropped passing between our servers and you. The "why" of the dropped data is what we need to pinpoint now; I'll need a bit more information from you.

- What type of connection are you using to the Internet?

- What type of machine is your application running on?

- If possible, can you send me a private message with the list of symbols you're watching? This will give me a better idea of the bandwidth involved.

Now, some background on why these things can impact your data reception.

Our quote servers, after upgrades last month, have proven to be more than capable of keeping up with our data feed and transmitting it to customers in a timely manner. However, memory on these machines is not unlimited - we cannot queue data on our end waiting for the remote clients to be ready to receive it. Doing this would put our servers at the mercy of any client who wanted to break our servers just by not receiving data for an extended period of time. Timely (and complete) delivery of data is entirely dependant on the remote end's ability to read the data fast enough.

While I'm not certain what symbols you are watching, with 1300 symbols being watched, I'm pretty sure this is generating a significant amount of data.

Get back to me with your information, and we'll try to work from there.

Thanx,

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Tim,
My trading depends on a high level of liquidity so the 1300 NASDAQ symbols I am watching are the top 1300 ordered by average traded value excluding the top two symbols (INTC & MSFT). This still leaves me with plenty of very busy symbols. (I will send my list via email)

In answer to your questions:
- Internet connection is high speed cable (in Australia). I can easily sustain 7mbps downloads.
Ping times to your login servers are around 200ms
- I am running the app on an HP xw8000 workstation with dual 3.2GHz Xeons, 4GB RAM, 15,000rpm SCSI drives, etc Windows XP SP2. IQFeed client is v2.3.0.2

I am intrigued by your response. The real time feed had no missing counts. Does that mean that these are generated by the IQFeed client rather than your server?
BTW, the historical data was downloaded by a separate application called Dynastore rather than our application.

At present, my Java application is simply receiving the data stream and writing out the data to disk. At market open the download peaks at 5.5Mbps and then settles down to around 2Mbps. The strange thing here is that the amount of data written to disk is only slightly larger than what is being received. This implies minimal compression from the servers.
I have also had problems with data corruption (missing fields, truncated records, and things like "U,F" in some fields) which may be related. There is a separate thread on this.

Dennis

- 7MB sustained should be more than enough - at 9am EST, I'm seeing output at just under .2Mb on your watchlist.

- Nice machine =) As I'm sure you guessed, no problems there.

- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)

Btw, as far as compression goes: I'm seeing 21KBytes/second on our server output, vs 160Kbytes/second (again, with your watchlist).

I'm actually suprised about the "U,F" you're seeing in some fields - this is something you should NEVER see - it's part of our server feed before it's converted to the text feed that IQFeed puts out. I'll pass this on to the IQFeed guys to look at.

Could be this message corruption is the root of your problem, though.

Thanks for the input,

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Dennis - I've reposted your email here so that everybody can benefit.

I was under a slight delusion as to how IQFeed handled outgoing data - basically I looked at code that has NOT been completed yet, that fixed a problem that is most likely causing your symptoms.

Basically (and I hope you understand the basics of TCP/IP communication):
- IQFeed sets TCP send / recv buffers to 256 Kbytes.
- When incoming data is processed, a message is created for the client.
- This message is sent to the client. No checking is (currently) being done to ensure that this message is sent in its entirety. This is fine so long as the 256k TCP buffer doesn't get full.
- After the TCP buffer gets full, attempts to write to the socket will fail.
- You will get corrupted data / lose data.

The obvious problem here is that your code may not be processing messages from IQFeed fast enough. A couple of solutions:

- Make your TCP recv buffer bigger. In java.net.Socket, I believe this is accomplished via the setReceiveBufferSize() method.

- Read data from the incoming socket in one thread, and pass this data to another thread - via a queue or whatever - to be processed.

Sorry for failing to notice this earlier. The recv buffer size fix is often offered, but I wasn't aware of why until I got one of our client guys to help me find the problem in the code.

If this helps you, let me know.

Now, for the amount of data you're seeing - are you getting this via the IQConnectionManager? If not, where are you getting these values? Only curious, as what I'm seeing going to your computer doesn't account for that much data.

Thanks,

---

Tim Russell
Software Engineer
DTN IQ & FinWin

"- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)"

Without having seen his code, I imagine thet he is using the field "Number of Trades Today" (field 56 in the tcp buffer) to indicate the sequence number of the print.

Where does this field come from? From the description of this problem it appears that it is generated by the IQFeed client, not the server. This makes it significantly less useful, if not useless.
Edited by skunk on Oct 21, 2004 at 08:26 AM

Ah - trades today. Okay.

That value's maintained by the server - it's returned to you when you request it. However, it's not transmitted on a tick-by-tick basis, since it's easily calculable on the remote end.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

Okay......you can forget about all my concerns to do with high bandwidth as it was my fault. My application logs just about everything and I recently changed the location of the log to a networked drive. The bandwidth measurements were being skewed by this.

Tim, I have to agree with skunk on the value of the counters. If you can't tell when you've missed a trade then what's the point. If you look at the real time data example I gave earlier then it appears that I haven't missed anything. Clearly the IQFeed client wasn't aware either.

I've made the changes to the way I read from the socket connection and increased the java buffer sizes as suggested. I now have a separate thread that takes the data stream from IQfeed and place it in a separate buffer for the application to read. It seems to be working very well with only a small number of error packets so far. I'll analyse it better at the end of trading.
I am somewhat stunned that a buffer overflow can result in corrupt data. I can understand missed message or even truncated messages, but corrupt individual fields within messages???

The trade counters were originally for internal use, and used by our FinWin product - since it's a web service, it doesn't do streaming data, and this information is very useful to those customers. The reason we made this available in the streaming feed was because there seemed to be a bit of interest in it.

However, it is NOT designed to be a method of letting you know data has been dropped in the feed.

As for the corrupt messages - let me know what you see.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

So, to summarize (and please correct any misunderstandings)

As of 2.3.0.2 and earlier there is NO flow control between IQConnect and API socket clients. The assumption is that the client will empty the receive buffer fast enough so there are no buffer overruns.

The "num trades today" field is useless to detect missed update messages beween your server farm and IQConnect. It might be useful to detect lost messages between IQConnect and the client app though.

Is there any way to detect update messages lost beween your servers and IQConnect?

And yes, this is a thinly veiled request for server (or even exchange) generated timestamps and sequence numbers to be transmitted to API clients.

Okay,
I've now done the analysis of all data received from IQFeed (thank god for Linux utilities) and have mostly bad news.
At the start of market open, I received an average of 1400 messages (trades,bids,etc) per second and had no problems at all. This is the highest data rate for the day and suggests that there are no buffer overrun problems. However, I received a large number of invalid messages starting at EXACTLY 12:00pm. Now if the corruption was due to buffer overruns or lack of bandwidth or CPU then you would expect all symbols to have corrupt messages. The below analysis of these invalid messages is very telling:

Entries containing the string "U,F|" in place of expected data:
1 ADEX
1 AMZN
14821 BMET
1 BRCM
2001 CMVT
1 CNCT
2 COCO
186 EBAY
1 EMMS
6 FAST
1 FHRX
1 FRED
785 FXEN
1 JAKK
1 MLNM
2 MRCY
1 NCRX
4149 PMCS
1038 PWAV
1 PZZA
2 QLGC
13641 RSAS
1 RTIX
1 SAFC
2620 SAPE
1 TSAI
34868 VRTS
53309 YHOO
=======
127444


Trades containing more than 59 commas (invalid):
1 ADEX
1 AMZN
3391 BMET
1 BRCM
654 CMVT
1 CNCT
1 COCO
1 CSCO
169 EBAY
1 EMMS
2 FAST
1 FHRX
1 FRED
116 FXEN
1 JAKK
1 MLNM
2 MRCY
1 NCRX
1 PCAR
1510 PMCS
422 PWAV
1 PZZA
2 QLGC
4752 RSAS
1 RTIX
1 SAFC
816 SAPE
1 TSAI
13609 VRTS
18306 YHOO
=======
43768

As you can see, the 170,000+ errors affected only a handful of the 1300 symbols I'm watching. Some very busy symbols such as CSCO are not represented.
Futhermore, the error in the message is in excatly the same position for a given symbol.

eg
Q,YHOO,F,35.3800,0.89,0.025804581,16679769,100,3537U,F|,34.9000,35.3700,35.3800,3400,600,,175,3502.1,12:56b
,,35.3700,34.4900,0.01,,,,p,N,,,,10/21/2004,,35.4000,,,,0.91,-0.01,90.8,-0.212661364,0.,0.,0.01,0,98.985302
431,48137214.260000005,14,4,,16009207,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,37928,,,35.3464,

The above has a U,F| which makes it invalid. All YHOO corrupt messages have this in the same position.

This seems to point the finger directly at the IQFeed client software. Perhaps it has separate buffers for each symbol........it's just speculation.

TIM: you mentioned that you're more of a server developer. Can we get some input from a client side developer please. I am now heading off to the "IQFeed Data Wishlist" forum to ask Santa for my early Christmas present.
http://forums.iqfeed.net/index.cfm?page=forum&forumID=15

Dennis,

Thank you for the analysis. I will forward this on to our software engineering staff who are already investigating the "U,F|" issue.

Steve Grunberg
IQFeed Developer Support
DTN Market Access

Steve,

You can also send them this one which I received today (never seen this before):

TL|=8800|>18.85|?1.46|@VOZ,YYT|B585.6|C28.9|D06/01/04|E172.5|F69586|H307000,20040423|I144400,20040830|J307000,20040423|K144400,20040830|k0.50,10/13
/99|}

It looks like the output to the log.txt file from the IQFeed client.

Dennis

Steve,
Have you heard anything from the software developers yet?
I'm keen to get acknowledgement that the problem is a bug with the IQFeed client and some indication of when a fix can be expected.
To blame this problem on an application that is too slow or has insufficiently large buffers would be unacceptable as that should result in lost messages, not corrupt messages.

Dennis

Dennis,

The reported issue of the character string "U,F|" occasionally appearing in streaming data has been logged in our IQFeed API Bug Database as item #2750. You can monitor its status by going out to the IQFeed Developer website and selecting the menu item "View Current IQFeed API Bug Database.

Our software support team will need to investigate it to see if they can replicate its occurence, prior to resolving it. No estimated resolution time has been set.

Steve Grunberg
IQFeed Developer Support
DTN Market Access