Today the NASDAQ and NYSE were particularly busy at the start of day and I observed data clipping on the IQFeed data stream. I have two machines, one watching 1300 symbols on NASDAQ the other 1300 symbols on NYSE. I have observed this apparent data clipping before and mentioned this in the forum. Jay (of DTN/IQ) replied that there is no bandwidth throttling and that perhaps our Internet connection was the problem. The two machines were originally in Australia but are now in a colo in Boston and I am still observing the clipping, primarily on NASDAQ where the message rate is much higher. The interesting thing is the clipping occurs at around 600kbps on NASDAQ and 450-500kbps on NYSE. When this occurs, we start getting a large number of corrupt messages. Nasdaq typically clips for 1-2 minutes at the start of each day.
If bandwidth throttling is not being implemented by IQ then what is causing the clipping? Could there be another limitation such as message/sec rate or some, non-deliberate firewall setting?

Additional info
Both machines are running dual Xeons with plenty of RAM and are not heavily loaded. The Internet connection is 100Mbps. Ping times to the IQ servers are typically 45ms.

Below is a graph of the NASDAQ bandwdith usage at start of day. Our NYSE feed is still clipping (with a large number of corrupt messages) almost an hour after market open.

Just some more info. We have been running the NASDAQ machine for two years and at the start we often had download rates of 700-900 kbps at the start of the day. Since March 2006, when we first noticed (and reported) the clipping we have never had data rates over 600kbps.

Dennis, can you post which servers each of your machines are connected to (IP in the IQ Connection Manager)?

---

Jay Froscheiser
DTN - Trading Markets

Nasdaq: 66.112.148.226
nyse: 66.112.148.227

nyse stopped clipping at around 10:32am.

I did disconnect nyse and reconnect but it made no difference (it sometimes improves). Unfortunately I didn't take note of the first server IP.

Dennis, I remember troubleshooting this when the original report was made. Can you email me (at developer support) the symbol lists you are currently using so that I can setup some more testing?

Steve,
I've just sent the symbol lists.

When you tried to troubleshoot this last time there was no obvious cause. As we were planning on relocating our machines to a colo in the US we didn't pursue it. Most of the time it doesn't cause us a problem but some days, like today, it can be bad enough that we shut down our day trading.

As it's just after 3am here in Melbourne, Australia so I'm off to bed but will respond to any other requests in the morning.

Steve,

Continuous clipping on NYSE this morning (server=66.112.156.223). Average download speed is 498kbps.
No clipping on NASDAQ, however the bandwidth usage is less than 600kbps where clipping normally occurs.

Dennis

Steve,
Thanks for your help with this problem both in this forum and via email.

We have examined the problem in detail and have determined that the problem is related to CPU utilisation. When we last looked at CPU utilisation 12 months ago we observed momentary peaks at around 70%. Things change.

Here is a breakdown of CPU utilisation on our NYSE machine during the busiest periods. The last two days have seen around 50-60% more data than normal. The machine is a Dell dual 3.0GHz Xeon server with 2GB RAM & mirrored SCSI drives.

IQConnect.exe 45%
IB TWS 8% (broker application and API)
Automated trading app 40% (total)

When we also run VNC for remote control of our machine it pushes it over the edge.

Our ATS is actually made up of several processes of which the most CPU intensive is the 'collator' which parses the IQ data, performs tick correction, saves raw data and tick data to compressed files and creates 10 second slot data for our trading application. It uses a separate thread to quickly accept data from the IQ client. We are attempting to improve the efficiency of the collator. We are also reducing the number of symbols we are watching.

The IQConnect client is using almost 1 CPU (3GHz P4) to itself to cope with watching 1300 NYSE symbols on a busy day. However, IQ staff mention watching 2500+ symbols and coping with even bigger data streams.

We are using v4.1.2.0 of the IQ client, are the later (or earlier) versions more efficient?

What hardware are you using, or do you recommend?

I have discovered that IQConnect.exe is a processing cycle hog. When I am watching 1100 tickers, on busy days, IQConnect uses up such a high fraction of the processing duty cycle that my application's recv does not have enough processing cycles to just get the data from the socket to a buffer. This happens when I just ignore the buffer and run a tight loop calling select with a 100 millisec time limit and recv. Note that select will return when data is available - it does not wait fo rthe 100 millisec in this case.

By looking at the system milliseconds, it is clear that the recv call itself is being preempted so IQConnect.exe can have the cycles it needs. Thus recv takes so long to execute that the loop cannot keep the pipeline from filling up indefinitely. (Normally recv for a typical IQFeed packet of 17KB takes just a few microsoeconds, but when the market gets busy recv for teh same number of bytes takes up to 100 or 200 milliseconds.

There clearly is no way the application can work around this problem - IQConnect.exe will always hog all the cycles when the market is busy and you are watching a lot of tickers. So the only solution I can think of is for the application to detect this condition and cut back in the number of tickers watched when it happens. IQConnect.exe does not appear to discard older data in the socket pipeline or merge prints or do other things to avoid this overflow condition. (Perhaps it is best to leave this to the app - he may not want to lose old data, but the app developer needs to understand the problem in order to deal with it).

The way I detect this vicious cycle of data backing up is when I need new data, I look at how many bytes I can get from the socket with a loop of recv calls before recv returns 0 indicating I have all the bytes. If this number of bytes is more than 1,000,000 then I send r<ticker>\r\n 's to reduce the number of tickers being watched.

When things are running normally, the number stays under 250,000 bytes.

The processor I use has a big effect as you would expect. For example on a 1.3 MHz laptop
the problem occurs pretty much regardless of how busy the market is. But in a HP dual core 3 MHz Xeon, the problem only occurs on busy days and it runs OK most of the time.

Larry

---

L Miller

Larry,

I'm not sure whether I agree that IQConnect is a cycle hog, but it strikes me as being somewhat inefficient. When streaming data, it basically just uncompresses the incoming data, creates the appropriate data structure and buffers it before feeding it through the API.

By contrast, our collator saves the received data to a compressed file, parses the data and performs tick corrections, saves ticks to a file, and creates 10 second tick summaries to pass to our analysis software, as well as saving these summaries to a file. (the tick corrections consume lots of cycles on busy days)

Without knowing the IQ clients internals I would say our collator performs a lot more work yet consumes much less CPU and is written in Java.

We would like to offer IQ/DTN to write an alternative version of IQClient in Java. We would be happy to pass ownership of this code to DTN. This would also allow DTN customers to run on any platform that supports Java (Windows,Linux,Mac,Unix,etc). Could someone from DTN let me know if you would like to accept our offer.

Hi,
On a dual core 3 GHz Xeon, when the market is fast, IQConnect uses close to 100% of the cycles. (I don't know if it is using one or both cores). For this testing, I am simply calling recv in a tight loop and ignoring the data. recv itself tests out to not take a significant amount of processing. I can run my processing on a second CPU so I am not concerned about that. But if IQConnect needs more cycles per second than a CPU can provide, it will get behind and discard data, corrupt data, disconnect (this has happened to me) or do something unpleasant.
At the point wher the data gets progressively farther behind, I am getting about 3000 watch records (P, and Q,) per second. Thus it takes 333 microseconds of cpu time for IQConnect to process one record on the average. That is one million machine cycles to uncompress the approximately 250 byte record, add unchanged data and whatever. This strikes me as something that could be made MUCH faster (in any language), as your letter suggests.
I think it probably should take no more than 200 cycles per output byte, which is 50,000 cycles.
It seems that the various high performance API data providers all do something to keep you from talking directly to their servers over the internet. However, if you can get DTN to let you write a client that does this it would be very good. Alternatively, if they would make IQConnect more efficient it would be very good.

---

L Miller

Larry,

Data providers will always require you to talk to their servers through a provided API. I don't disagree with this policy as it protects all users from rogue programs that may affect server performance. I would be happy to sign a non disclosure agreement with DTN in order to write an alternative Java client for them. Naturally, they would need to own this software in order to maintain and enhance it in the future.

I too would be happy with a more efficient IQclient, but the appeal of a Java client, I'm sure, would be widespread. In my case, it would allow me to run Linux which is the native environment for all of our development work.

I love Linux. It handles preemptive multithreading and multiprocessing a lot better than Windows. However last I looked the major automated trading brokerages all run their API only on Windows. I would like to discover one whose API runs on Linux.

---

L Miller

We use Interactive Brokers. Their TWS client (which includes an API) is written in Java and works just fine under Linux.

That is good to hear that IB will run on Linux. Do you have a way to automatically start up TWS in the morning?
I was considering Genesis Day Trading as an automation broker. Any experience with them?
Larry

---

L Miller

I believe there is a way to start TWS as you request. Check the TWS API group on Yahoo.