I'm using IQFeed in an automated trading system to fetch the bid, ask and last price after I receive an alert from an external system. I really just need the SummaryMessage so as soon as I get this I stop watching the symbol yet after I get a hundred or so alerts, I start getting the error message:

S,SYMBOL LIMIT REACHED

I'm not subscribed to any symbols. I haven't gotten any quote or summary events for over five minutes so I'm pretty sure that the unwatch request is successful, yet it looks like something has "remembered" which symbols I watched and is blocking any new requests. I have to shut down my application and restart it periodically to get around this.

Does anyone have any ideas on what is going on and how I can work around it?


Here's my code.

When I get an alert I call WatchSymbol:

            log.Debug("handling alert for symbol=" + symbol + ", from " + defn.WindowName);
            iqFeed.WatchSymbol(ref symbol);

When I get a summary message, I get the data I need and remove it from my watch list:

void iqFeed_SummaryMessage(object sender, AxIQFEEDYLib._DIQFeedYEvents_SummaryMessageEvent e) {
                iqFeed.RemoveSymbol(ref symbol);
// ...

Why am I still getting the SYMBOL LIMIT REACHED error?

You should only be getting the symbol limit reached error when the connection manager shows that "Number of Symbols" is equal to "Maximum Symbols". When you get the error, does it show these two numbers equal? If so, then there is something causing the symbols to not get unwatched.

To properly replicate, I may have to wait until Tuesday as this problem typically occurs in the morning when I get most of my signals.

Just looking at the IQ Connection Manager now, I see the Maximum Symbols is 10. Can this really be so low? I thought we were able to watch many more than this. I also see that the number of symbols is 2 even though I would expect it to be zero so maybe there is a problem getting symbols unwatched.

Is there a way of figuring out which symbols are being watched? In my application I print a log message every time the SummaryMessage, QuoteMessage (and SystemMessage) events are received and they aren't printing anything so whichever symbols the manager thinks I'm watching, it isn't passing any of the events back to my client.

Max symbols of 10 is an indication that you are not launching the feed with your registered productID in the call to RegisterClientApp.

There is a system message you can send from the socket connection watching the symbols to know which symbosl you are currently watching. However, this system message is not supported in the IQFeedY activeX control (so it is not possible without making some major changes to your app).

You probably want to start by checking the return value for the call to RemoveSymbol. This will tell you if the command is actually getting sent to IQConnect or if the RemoveSymbol call is failing.

Another option will be to turn on logging within IQConnect. You can turn on logging via the diagnostics app (before launching the feed). The logfile will output all communication sent to and received from your app. Make sure you turn logging off when not debugging since it will potentially create very large logfiles and hinder the performance of the feed.

Steve,

Thanks for the debugging tips. I'll go ahead and do them on Tuesday to see what's happening.

Re the registered productID, I double-checked and I'm using the only product ID that I've been given. I'm also connecting to IQFeed via QuoteTracker - could that be affecting things?

Looks like I didn't have to wait. I've been running my app for an hour and the Connection Manager is showing me at 5 symbols (out of 10!). In the app, I added code to print out an error message any time that RemoveSymbol() returns false and so far none of them have done so:

                bool res = iqFeed.RemoveSymbol(ref symbol);
                if (!res) {
                    log.Error("Call to RemoveSymbol for " + symbol + " was unsuccessful.");
                }

You are correct, it does appear that you are using your registered productID in your app. However, it also appears that you used the default demo productID for your initial login (which caused the max symbols = 10). All of the example apps and dev tools included with the feed use the demo productID. Make sure you launch the feed initially with either your app or QuoteTracker and your symbol limit will be the normal amount. You can run the example apps and tools after you are already connected but make sure the initial login is with your app.

As for not unwatching symbols, I would definately recommend at this point taking a look at the logging in the feed to see if you can determine what is happening (they should be pretty self explanatory). Failing this, I will need to be able to duplicate the issue you are seeing internally in order to offer more help. Is there anyway that you can email me a copy of your app that I can test internally to help debug this further? You mentioned that it is triggered from an external system that I would assume cannot be sent. Can this behavior be emulated in a test app that demonstrates the problem?

Steve - I'll do the logging thing on Tuesday and if I can't sort it out, I'll try to package some sort of app to replicate it. It's not going to be easy since it appears to take about 50 signals to generate one glitched symbol. Situations like that are always a huge pain.

On the plus side, I shut down all apps using IQFeed and then started my app before QuoteTracker and now the Connection Manager is showing 500 max symbols! That means that at the rate I was going I should be able to accumulate dead or glitched symbols without maxing out. Not ideal but a good start. Thanks for your help, will let you know what I find with full logging.

I've enabled logging for "All data flow". When I try to view the log it says "The log file c:\Program Files\QuoteTracker\\IQConnectLog.txt could not be found!" I'm not sure why it is looking in the QuoteTracker directory or why there are two "\\" before the file name. Where should I look to find the log and verify everything is working?

I think I may have one app connected to IQFeed already but my bot just connected. Do I need to have a clean restart? It's showing a 500 symbol limit so it's not running in demo mode any more.

My apologies. You will need to make sure you launch the feed using your software after turning on the logging. The reason is that the QuoteTracker software automaticly turns logging off in IQFeed every time it launches and redirects the logfile to the QT install directory.

I'm not sure about the extra \ in the path and not entirely sure it will cause a problem.

You can manually edit the path of the logfile via the registry setting if necessary:
HKEY_CURRENT_USER\Software\DTN\IQFeed\Startup\LogPath

Keep in mind that IQFeed only reads these settings at startup so you will need to make sure that IQConnect is shut down (no systray icon), turn logging back on, launch the feed (without using QT to launch it) and then go about your testing.

I tried changing the registry but for some reason it is stuck in the QuoteTracker folder. The log file is being generated so that's good enough. I made the mistake of letting it run for a full day before checking the log and now it's over 3Gb! I also didn't delete it yesterday and so the log file contains a mix of extraneous data. I'll clean it up after trading today and try to save a copy after 10 or 15 minutes of trading to keep the size to a minimum.

In the meantime, I ran HJSplit to chop it into 5Mb chunks and have been going through them to try to see if there's anything I can glean. My goal is to figure out why I remove some symbols and they are still in my "watch" list. I'm just guessing, but I think this is what the good path looks like:

(Edit: I'm adding line breaks in the log files to make them easier to read)

FROM CLIENT (LEVEL1)=2009-02-24 06:46:44: wDECK
TO CLIENT (LEVEL1)=2009-02-24 06:46:44: F,DECK,F,10.6,521000,146.6000,46.2700,85.9000,50.6900,,,,,,,,,,,5.23,7.16,42.69,12,,
    DECKERS OUTDOOR CORP,GVK QUK,,2.20,,301.6,71.4,09/01/2008,,13069,,,,,p,14,4,3021,76.43,1,1,04/28/2008,11/20/2008,01/06/2009,02/02/2009,79.87,,,,,
P,DECK,F,54.9800,-0.23,-0.004165912,35604,100,55.2000,54.9400,54.8200,54.9900,200,
    200,,,0.26,09:46:29t,,55.1800,55.2100,0.17,,,,p,,,,,02/24/2009,,54.9800,,,,-0.23,0.,10.6,-0.931662188,,,-0.2,0,0.004728992,718533.62,14,4,,
    730454,BSE-CSE-CHX-NYSE-PSE-NMS,,,,,249,,,55.0189,0,N,,

FROM CLIENT (LEVEL1)=2009-02-24 06:46:44: rDECK

Again, just guessing, I think this is a request to watch DECK ("wDECK"), then a line for the fundamental data ("F" type), then a summary ("P" type) and immediately after the summary, the client removes DECK ("rDECK").

Sometimes I get a string of quotes ("Q") lines in between and sometimes I can see that I'm getting quotes after the remove request:

FROM CLIENT (LEVEL1)=2009-02-24 06:46:56: rFDX
TO CLIENT (LEVEL1)=2009-02-24 06:46:56: Q,FDX,D,44.2900,-0.26,-0.005836139,305974,400,44.8900,44.2400,44.2800,44.3200,
    300,300,,,0.65,09:46:55b,,44.8900,44.5500,0.04,,,,p,,,,,02/24/2009,,44.2900,,,,-0.26,0,13.5,-0.914817929,0.,0.,-0.6,0,0.014675999,13787122.68,
    14,4,,3984230,CSE-CHX-NYSE-PSE-NMS,,,,,1631,,,44.5301,0,N,,
Q,FDX,D,44.2900,-0.26,-0.005836139,305974,400,44.8900,44.2400,44.2800,44.3200,400,
    300,,,0.65,09:46:55b,,44.8900,44.5500,0.04,,,,p,,,,,02/24/2009,,44.2900,,,,-0.26,0,13.5,-0.914817929,0.,0.,-0.6,0,0.014675999,13787122.68,14,4,,
    3984230,CSE-CHX-NYSE-PSE-NMS,,,,,1631,,,44.5301,0,N,,
Q,FDX,D,44.2900,-0.26,-0.005836139,305974,400,44.8900,44.2400,44.2800,44.3200,300,
    300,,,0.65,09:46:55b,,44.8900,44.5500,0.04,,,,p,,,,,02/24/2009,,44.2900,,,,-0.26,0,13.5,-0.914817929,0.,0.,-0.6,0,0.014675999,13787122.68,14,4,,
    3984230,CSE-CHX-NYSE-PSE-NMS,,,,,1631,,,44.5301,0,N,,

Though in this case, the FDX quotes stopped quickly, within a second.


What I don't see are any responses from the remove request to indicate which symbols are being watched and which aren't. I'm already logging any quote and summary messages when they come and after a remove request, they stop coming. What more should I be looking at?
Edited by adrian on Feb 25, 2009 at 09:24 AM

adrian, the second example certainly seems suspicious to me. We have code in IQConnect that should prevent you from getting messages for a symbol after we process a remove command. At this point I am inclined to say that the problem we are seeing in the log exists in the logging code for IQFeed (it is acutally logging the info out of order) but I will look into this to doublecheck since it could point towards the problem you are experiencing.

Realisticly what you are looking for is exactly this type of behavior. Something that would indicate that a symbol is not getting unwatched when you send your request. This would show up either as your app sending the request but it not showing up in the IQConnectLog (meaning IQConnect never received/processed it) or as something like the above but where you continue to get messages for a symbol after it is unwatched.

Additionally, you might want to add to your app a request for the current watches. You can send a S,REQUEST WATCHES to the feed and it will tell you what symbols are being watched on the client connection which the command is sent from (and this info would be output in the IQFeed Log File as well).
Edited by DTN_Steve_S on Feb 25, 2009 at 09:44 AM

Steve,

I don't see a way to send raw commands like this via the C# OCX so I modified a Java app. Will the REQUEST WATCHES let me know all of the symbols all clients are watching or only those that the current socket client is watching?

When I send a command "S,REQUEST WATCHES\r\n" on the socket the only response I get is "E,!SYNTAX_ERROR!,". I can use the same structure to request historical data (HTT, HDX, HIT, etc) with no problems so I'm not sure what's going on. Are there any docs on the REQUEST WATCHES command?

I forgot that you are using the IQFeedY control. There is no way to get the watches using the control. Unfortunately, there is also no way to get a list of all symbols being watched from another socket connection either.

As for the syntax error, it sounds like you are sending the command to the lookup port. This only works for the Level 1 data port that you get the streaming data from (you don't issue watches on the lookup port so the command is not supported).

Is is possible for you to email a copy of your app to developer support so that we could run it internally? It might be easier for us to track this down if we can see first hand what is happening.

Steve - Thanks for the offer. I know how time consuming it can be to debug a strange new application so let me help out by removing the external dependencies, generating some fake signals and getting an app which can replicate this problem quickly & easily. Hopefully that'll make it easier for everyone. I'll e-mail it when I get something ready.

Steve - I haven't found an easy way to strip out the InteractiveBrokers code so I did the next best thing and rewrote the IQFeed code to use a socket directly. I got it working and the same problem is happening. I even added code to call RemoveSymbol() from the QuoteMessage event but this doesn't seem to have any effect.

To debug, I call "S,REQUEST WATCHES" but I don't see anything returning. Perhaps I'm missing it. Is there any documentation to describe the return values?

Also I am pushing remove watch messages (r[symbol]<CR><LF>) whenever I get a Quote message and nothing seems to happen. I continue to get quotes. I have streams of lines in my log file where I request to remove QQQQ, get a quote for QQQQ anyway, request remove, get a quote, request remove, get a quote, request remove, and on and on.

I've put in the work to get a socket-based client so that I can have more control over the feed. Can you help me figure out what more I can do to debug this problem?

Adrian, in your previous emails, you mentioned that if you remove the IB calls, the problem went away. This was when you were still using the ActiveX control. Is this still the case now that you are using sockets?

The request watches command is documented on the System Messages page of the Documentation. You should be getting a system message in return for this even if IQFeed doesn't report that you are watching anything. An easy way to test this (so you can see what you should be getting back) is to open up a telnet session to the localhost on the level1 port and issue the commands manually.

Steve - I found the System Messages page, thanks. I'm getting the WATCHES messages now and they look good.

I can't say whether the bug is present or not. It seems to be highly dependent on quote volume or time of day. I haven't been able to replicate it since 11 EST today which is frustrating. Hard to work on it when it can't be replicated.

I was getting the problem using sockets when I was sending orders to IB this morning and I didn't check to see if the problem went away when I turned IB orders off. Something to check tomorrow morning. Will let you know when I figure something out...

As an update: after changes to make this problem clearer, it has ironically gone away. I've kept backups of my test suite so when I get a chance I'll go back to earlier versions to try to see if it's still present or if I've accidentally fixed the problem. Since it's working, tracking the bug is low priority but when I get some new info, I'll post it here.

Thanks Steve for helping me through some debugging techniques.