Hi there,

recently, I've spent some time modifying my trading application in order to keep a full real-time visual snapshot or "view" of the DOM landscape. While the visual result is very fancy and all of that (d'oh!), in the process of toying around with IQFeed's L2 stream I've come across a problem which I'd like to share with you.

Thing is, L2 can be at times a really busy stream, even more so if you happen to pick a busy symbol (NQ, ES, YM...). I've come to record incoming L2 message ratios well in excess of 1500 L2 messages per second coming from IQFeed during busy market hours (stock maket open at 09:30EST, close at 16:00EST, etc.. Normal, "quiet" periods tend to average 200-500 messages per sec.

The problem I'm experiencing is that, the moment the L2 message ratio reaches aound the 1000 mps (messages per second) mark, my IQClient app starts losing L2 messages, and it will disconnect / reconnect to your L2 servers. Here's an example from the logs:

------------
STATUS Information 36 0 2020-06-16 16:21:38 No data from Level 2 server in 7s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:21:38 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:21:43 Connected to Level 2 server.

STATUS Information 36 0 2020-06-16 16:40:03 No data from Level 2 server in 43s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:40:03 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:41:09 Connected to Level 2 server.

STATUS Information 36 0 2020-06-16 16:41:48 No data from Level 2 server in 39s. Reconnecting.
STATUS Information 36 0 2020-06-16 16:41:48 Attempting to reconnect to Level 2 server.
STATUS Information 36 0 2020-06-16 16:42:11 Connected to Level 2 server.
----------

Sometimes, it will recconect itself in mere seconds (3-4 seconds), while other times one or two minutes can pass until it will recconect itself to the L2 server.

I know that if my end cannot keep up with IQFeed's server, your host will likely drop my connection, hence these disconnections could be due to my system not being able to cope with one of these "L2 storms" which it's happening during market busy hours.

The problem I see here is that I already tried to make a very simple, stripped-down L2 client just to make a proof-of-concept of how many L2 messages per second I would be able to catch. Even with a simple read-and-discard strategy (no L2 message storage, no ascii parsing, no dynamic structures used, no thread locking, no synchronization locks, etc) and then, I keep getting kicked out of your L2 servers during _Really_ busy market events (opening, close, etc.).

So, if the software isn't blocking your stream flow... could it be just a latency problem ? My servers have a not-so-nice 180-190ms ping round trip to IQFeed servers. With as high a latency as that, am I facing a structural problem here, and no matter how fast my software is, I won't be able to keep up with a fast moving L2 stream ? (unless I opt for leaving Europe and collocating my servers within USA, of course..)

thanks for your time,
Edited by jmv on Jun 16, 2020 at 10:18 AM
Edited by jmv on Jun 16, 2020 at 10:18 AM
Edited by jmv on Jun 16, 2020 at 10:20 AM

There are no hard-and-fast limits on the number of L2 messages you can receive at once, or anything else that should be causing this behavior. Can you tell me what programming language you are using, and the list of symbols you're requesting that's causing the problem? You may send this information to the IQFeed API support email or support chat if you'd prefer.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

The language I'm using is Java (openJDK 14), and I'm asking for seven symbols : ES,NQ,YM,6E,6J,6B, BTC .

I'm just toying a bit with the system's granularity right now... I'm checking the receiver buffer a bit less often (every 25ms instead of 5ms which was before). I'll let you know if this eases the problem.

However, the real test will come when markets close (i.e., 16:00EST ) since this is one of the busiest events in the day, from a L2 perspective. So in three and a half hours we will be able to see whether we still hit this problem again.

For me, it's highly suspicious that the problem only manifests itself in the busiest market moments (i.e., when the L2 message flow is at its peak). Either my receiving software(*) is the bottleneck, or my latency is causing the problem, I'm afraid.

(*)note: as for my software, I've also tried raising the socket buffer size, just to be sure that I wasn't discarding L2 messages due to a network bufer overrun... but that doesn't seem to be the problem, either.
Edited by jmv on Jun 16, 2020 at 11:30 AM
Edited by jmv on Jun 16, 2020 at 11:33 AM
Edited by jmv on Jun 16, 2020 at 11:36 AM
Edited by jmv on Jun 16, 2020 at 11:37 AM

If you are willing, we can test your app in-house. This will help eliminate network or machine as possible causes of the delay.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Thanks for your kind offer ,

as of yesterday's experiments, I've found out that tinkering with granularity (i.e., thread locking frequency) helps yielding better results. Yesterday, I got only a final (and single) L2 disconnection event, at market's close (16:00EST) which was to be expected, since it's the busiest moment of day (along with the stock market opening at 09:30EST, this is). This is much better than previous days, when I was getting several L2 disconnections *per hour*.

This tells me that the problem is at my end (most probably, excessive thread locking is still my problem). I think I can fix this by toying a bit more with my system's granularity. Should this not be the case, I will send you the source code of my stripped-out as-simple-as-possible test mule barebones L2 receiver, so that you can check it out for yourself.

thanks for you time and great support,

If you decide you would like us to review it, you can get in touch with me via any of the support channels. Ask to speak to Gary (me), and I'll handle everything from there.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Hi Gary,

just a couple of lines in order to let you know that, by means of optimizing our code (mostly trying to avoid race conditions and time-costly locks), I've been able to keep the L2 disconnections at a minimum (perhaps twice or three times a day, in the busiest day moments).

So the problem is _nearly_ solved. However, there's one thing I forgot to mention in my initial problem description: for years, I've been running iqconnect.exe via wine from linux, which I am quite sure it gives some (if not quite a) performance penalty.

So, after all, I'm quite sure right now that those two or three "spikes" would be handled correctly by iqconnect.exe , should it be natively run from within a Windows machine. Within the next few days, I will set up a Windows Sever 2016 instance in order to try whether this is the case.

Other than that, we're pretty satisfied with our performance right now.

thanks for your time and support !

Thanks for the update! IQFeed is a native Windows program, and I can't offer a lot of advice on how to run it under Linux or any non-Windows OS. You might also look for any Wine errors or log messages at the times you're having these spikes; in some cases that can clue us in to what's causing it.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist