Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I am keeping IQFeed, much better reliabilty than *******. I may refer a few other people in the office to switch as well." - Comment from Don
"Everything is working great with the API. I love it." - Comment from Calvin
"With HUGE volume on AAPL and RIMM for 2 days, everyone in a trading room was whining about freezes, crashes and lag with *******, RealTick, TS and Cyber. InvestorRT with IQFeed was rock solid. I mean SOLID!" - Comment from Public IRC Chat
"You are much better than lawyers or the phone company because you answer the phone when I call! I just love your customer service." - Comment from Isreal
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"Very impressed with the quality of your feed - ******* is a real donkey in comparison." - Comment from A.C. via Email
"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Socket problems
Author Topic: Socket problems (13 messages, Page 1 of 1)

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Oct 25, 2005 09:41 AM          Msg. 1 of 13
I have been hesitant to report the following problem for quite some time, since I assumed it was the fault of my own programming. But in the past few weeks, I have added copious logging to my own application, have attempted to debug it scrupulously, and now I am coming to the conclusion that perhaps the problem lies with IQFeed, not my own app.

A description: my app uses the TCP/IP (sockets) interface to IQFeed. The app runs great for some period of time. Then, at some interval which may be a few hours, or may be ten minutes, it stops working great.

Specifically, what happens is that while my app continues to receive Q, P, and F update lines perfectly well, IQFeed apparently stops listening to watch requests. I know this because symbols which are perfectly good simply don't appear in the message stream, even after multiple attempts to "watch" them. Once this condition starts, it's terminal ... no matter which symbols I attempt to add to my watch list, nothing gets added. The only way to deal with this is to shut everything down and restart it.

To address the first question DTN will ask: I am not bumping up against the symbol limit in my account. I confirm this both via my own internal application's counting scheme, and through the Connection Manager Feed Stats.

I wonder if anyone else out there is using sockets programming w/ IQFeed and facing a similar issue. Has DTN heard of anything similar? If neither answer is yes, then it may very well be a problem in my code, but if so - I can't fathom what it can be - after all, I can sit and watch the incoming message stream continue to roll into my app from IQFeed... but no new symbols get added.

Can anyone help?

Matthew

Matthew Klein

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1753
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 25, 2005 09:51 AM          Msg. 2 of 13
Matthew,

I haven't heard this issue before, and you answer the first question I would have asked

The next question I would have is when you place the watch command, do you see the connection manager increase the symbol count?

Also, maybe open a telnet session to port 5009, enter S,KEY then enter your watch command and see if the other socket is able to watch the symbol you are having issues with.

Can you give any more details as to timing, how many symbols you're watching, etc that may help. Are you churning through a large number of symbols throughout the day?

I know Natalie has identified an issue with numerous history requests, but I am not aware of anything on watching level I data.

Jay Froscheiser
DTN - Trading Markets

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Oct 25, 2005 10:21 AM          Msg. 3 of 13
Hmm. For what it is worth, I actually do also make numerous history requests, but for minute data, not daily data. I remember reading in these forums that the problem Natalie has mentioned relates to daily history requests, right?

Let me ask this: is it possible these are both the same problem? Is one of the symptoms of the "too many history requests" problem that IQFeed stops listening to socket 'watch' requests, but it still continues to deliver updated quotes via the socket for items already in the watch list?

I would consider this great news, because it would mean the problem I am reporting is a problem that has already been reported, and has a known (or soon-to-be-known) cause, and is something which IQFeed is already presumably working on.

Matthew

Matthew Klein

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Oct 26, 2005 09:18 AM          Msg. 4 of 13
Jay:

OK. I did some more investigation and have the following to report.

I set an alarm to sound when the socket problem occurs. (That is, when IQFeed will not respond to any more 'watch' requests, but is still delivering update messages for symbols which have already been watched on already-open socket).

When I heard the alarm this morning, I followed Jay's instructions. I telneted into port 5009, and then manually issued 'watch' commands. Lo and behold, the new socket worked fine. New symbols could be watched an unwatched.

Jay and Natalie: any ideas what this means? Any other ways I can help you investigate this?

Finally, I wonder: could I possibly be the *only* person to have this problem? If so, does this indicate that I am a lousy programmer, and I am doing something wrong; or is it merely an indication that I am one of only a few who uses the TCP/IP interface?

Matthew

Matthew Klein

DTN_Jay_Froscheiser
-VP, Product Operations-
Posts: 1753
Joined: May 3, 2004

DTN IQFeed/DTN.IQ/DTN NxCore


Posted: Oct 26, 2005 09:28 AM          Msg. 5 of 13
As much as I would like to say this issue is because you are a lousy programmer , I always take the tact that we need to troubleshoot this together. If I were to blow you off and tell you its your problem, we may miss finding a bug somewhere in our code that only you have been able to identify so far. At this time we have had no other reports of this kind of problem. That doesn't mean nobody has seen it, it just means nobody has reported it. One thing I can assure you of is that we have hundreds of developers using Sockets. Only a very small number of developers are using the COM interface for IQFeed. I would guess this is more of an issue of what you are doing is different than what others do, and that is why nobody else has the issue.

The fact that you opened the socket manually and were still able to watch symbols tells us that IQFeed itself is still responding to watch commands, so we can assume its not an issue with the core IQFeed "not working". However, there may be something in the socket that you are using that is getting corrupt and thus isn't seeing the watch commands any longer.

Can you give some more specifics on how many watches/unwatches you issue and for what kind of symbols, before you see th issue occur. Does it happen during market hours or only afterhours? Again, more detail the better.

Jay Froscheiser
DTN - Trading Markets

DTN_Natalie_H
-DTN Evangelist-
Posts: 175
Joined: May 10, 2004

DTN Market Access, LLC.


Posted: Oct 26, 2005 09:29 AM          Msg. 6 of 13
Hi Matthew,

Thanks for further investigating the issue for us.

The history issue was present on all request types...not just daily. That issue would completely lock IQFeed up (if I remember correctly), so I don't think that is your issue. I believe that we have quite a few people developing to the socket interface, but that doesn't really mean anything. You could be exercising IQFeed in a different way than others do.

So, you could really help me out by giving me specific steps to recreate the problem. If I can see it happening, then I can fix it. If you have a test app that recreates the problem, you can send it to the developer support email, and I can run it to reproduce the issue. Other wise, you can just tell me the steps that you are doing, and I can write my own test app.

Thanks!

Natalie

Natalie Hannan DTN Market Access, LLC.

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Oct 26, 2005 10:54 AM          Msg. 7 of 13
I beginning to suspect perhaps that the problem is caused by the unusual way I use IQFeed.

First, some background. My own app is used for internal trading and trade-tracking purposes, so it doesn't need to keep a lot of symbols on the watch list at any one time. Another way to say this is that I have no need to do a lot of "scanning" of the stock universe, looking for various conditions among a large universe of symbols. Instead, I simply want to be able to pull up an up-to-date quote for a stock/future I am interested in trading. When I have looked at the quote and fundamental data, my app is generally "done" with the symbol, and doesn't need it any longer.

I tell you this to explain why I use IQFeed the way I do. I have read elsewhere on this forum that IQFeed frowns upon users doing a lot of "watching" and then "unwatching" of symbols, since each new request requires more bandwidth than a simple update message.

So when I first designed my app, I tried to be a good software citizen. If I needed a quote, I would start to watch it, pull up the quote, then process it. But I would leave the symbol in my watchlist and so continue to receive update messages for it.

But I noticed some distrubing things. First, the more quotes I had in my watchlist, the longer the latency of getting a new quote for a brand-new symbol. What's more, when I studied the IQFeed Connection Manager, I saw that the Internet Bandwidth used by my app grew quite substantially as the number of symbols in the watchlist grew. (I suppose this is obvious.) There reached a magic point (60, 70, 80 symbols, I think) where the latency for requesting a new quote on a new symbol grew too oppressive ... I suppose because my bandwidth / DTN's CPU cycles were occupied by delivering update messages for symbols I didn't really need.

So I began to experiment with an alternative software design. I would keep the number of symbols in my watch list quite low (say, 5). Thus I would ask for 5 symbols, but when the sixth was needed, I would unwatch the first one. Since I don't need to keep quotes in memory, and since I need symbols only periodically and don't need to watch them over time, this was fine with me. In fact, I believe this set-up was better from DTN's point of view, since my Internet Bandwidth usage was kept quite low (.75 K/sec during the trading day, even during periods where symbols get updated frequently).

The result of this design is: (1) I don't use a lot of DTN bandwidth (so don't get mad at me), and (2) I don't suffer latency when I need a new quote on a new symbol. In general, I can ask for a quote on a symbol, by watching it, and have the update and fundamental messages arrive almost instantly.

However, the implication of this design is that I do indeed do a lot of watching and unwatching of symbols. This might be the reason (I hypothesize) that I am one of the few who has reported a problem in which an attempt to watch a new symbol finally just fails, even though the IQFeed core is still running fine.

So, if you want to emulate my software, I would suggest you simply write a small app which will keep 5 random symbols in a watchlist, and -- when you want to watch a sixth, will unwatch symbol number 1 first. Just have the app ask for a new sixth symbol repeatedly. After some random amount of time (sometimes hours, alas!) the next attempt to watch a symbol is simply ignored, even if the watch request is repeated.

Matthew

Matthew Klein

DTN_Natalie_H
-DTN Evangelist-
Posts: 175
Joined: May 10, 2004

DTN Market Access, LLC.


Posted: Oct 26, 2005 11:01 AM          Msg. 8 of 13
Hi Matthew,

Thanks for the explanation. I will attempt to recreate the issue, and will let you know what I find. Is you app doing anything else with IQFeed when you are able to recreate the issue? News, History, etc. or are you able to reproduce it with just the Level 1 socket requests? Thanks!

Natalie

Natalie Hannan DTN Market Access, LLC.

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Oct 26, 2005 11:41 AM          Msg. 9 of 13
Natalie:

My app will also request recent minute-bar history for symbols that are "of interest." No other IQFeed functionality is used -- no news, no optionchains, etc.

Regarding the history requests: I will sometimes get a failure when requesting history, but I believe this is caused by the too-many-history-requests bug, which is different than the problem I am reporting.

To repeat: the symptom of the problem I am reporting is that new "watch" requests are ignored, and the requested symbol never gets added to the message stream.

Matthew

Matthew Klein

DTN_Natalie_H
-DTN Evangelist-
Posts: 175
Joined: May 10, 2004

DTN Market Access, LLC.


Posted: Nov 2, 2005 12:35 PM          Msg. 10 of 13
Hi Matthew,

I haven't been able to reproduce the problem. How many symbols are you watching when this issue occurs? Have you seen the issue occur with various amounts of symbols? Thanks!

Natalie

Natalie Hannan DTN Market Access, LLC.

matthewklein
-Interested User-
Posts: 40
Joined: May 20, 2004


Posted: Nov 3, 2005 10:15 AM          Msg. 11 of 13
Natalie:

I'm adding some more extensive logging to my app, so I will be able to send you something that is intelligable to you, and helpful. As soon as I have done this (and as soon as the problem occurs again), I will email you the log. (Please let me know the email address where I should send it.)

To answer your question, though, I'm watching exactly 5 symbols when the problem occurs. However, it *seems* (need to confirm this) to happen when those five symbols are very heavily traded, during peak hours, and thus I get a very fast stream of update messages (so, the watch list might include: @YMZ5, @ESZ5, @ER2Z5, etc. during peak trading hours). What seems to happen is that my wSYMBOL requests are ignored by IQFeed when this fast and furious stream of update messages is being shoved into the socket.

Again, I'll try to capture this in a useful log for you. Hopefully I'll have something for you in the next day or two.

Matthew

Matthew Klein

swhitney
-Interested User-
Posts: 21
Joined: Nov 3, 2005


Posted: Nov 24, 2005 01:33 PM          Msg. 12 of 13
Are you using threads? (you should be)

And if so are they synched? (they have to be)

IQ will hog a single threaded application message que when the data is flying.

Sounds like you may need to set up a blocking to squeeze your outbound message in and through.

Just a thought..

Scott

DTN_Natalie_H
-DTN Evangelist-
Posts: 175
Joined: May 10, 2004

DTN Market Access, LLC.


Posted: Nov 28, 2005 09:09 AM          Msg. 13 of 13
Sorry, I should have posted that Matthew and I came to a resolution via an IQFeed test build. I will get the build to QA this week. Thanks!

Natalie

Natalie Hannan DTN Market Access, LLC.
 

 

Time: Fri April 3, 2020 12:18 AM CFBB v1.2.0 0 ms.
© AderSoftware 2002-2003