Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I am keeping IQFeed, much better reliabilty than *******. I may refer a few other people in the office to switch as well." - Comment from Don
"Everything is working great with the API. I love it." - Comment from Calvin
"With HUGE volume on AAPL and RIMM for 2 days, everyone in a trading room was whining about freezes, crashes and lag with *******, RealTick, TS and Cyber. InvestorRT with IQFeed was rock solid. I mean SOLID!" - Comment from Public IRC Chat
"You are much better than lawyers or the phone company because you answer the phone when I call! I just love your customer service." - Comment from Isreal
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"I've never had DTN go out on me since switching. ******* would go down a couple times every month when I was using them." - Comment from Bryce in AL.
"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"Very impressed with the quality of your feed - ******* is a real donkey in comparison." - Comment from A.C. via Email
"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Lookup Socket receives only one single data point even when setting "DataPointsPerSend" to 2500 or 5000
Author Topic: Lookup Socket receives only one single data point even when setting "DataPointsPerSend" to 2500 or 5000 (14 messages, Page 1 of 1)

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 03:20 AM          Msg. 1 of 14
I wonder what I am doing wrong but my Lookup socket only receives one single line (1 datapoint) no matter to what I set DataPointsPerSend when submitting the request. What is wrong and how can this be solved? I would expect a much larger load to be forwarded by the IQConnect gateway as described in the developer documentation.

Thanks
Matt

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 03:41 AM          Msg. 2 of 14
Hello Matt,

Are you seeing the ENDMSG coming through the socket after your request? The rest of the data may be on the socket waiting to be read. If not, send us an email with your ID and I will check the logs to see what the requests are and how much data we are sending you.

Tim

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 03:45 AM          Msg. 3 of 14
Tim Walter,

I do see the ENDMSG coming through but its the last message. Each single tick is reported in a separate message. I would expect to see 500 or 2500 data points to arrive at my socket at once but that is not the case. Each single message contains only one single data point , such as

myquestId,2014-03-31 05:00:00,1.37701,1.37540,1.37570,1.37660,81355,12575,0,

the last message contains ENDMSG at the end.

What I would expect to see are multiple datapoints in one packet.

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 04:25 AM          Msg. 4 of 14
Bbmat,

I can find no instances where we returned only one datapoint to any of your requests, they are all 25, 500, 1000, or 0. Have you tried to turn on logging of the history socket so you can compare what you are seeing with what IQFeed is returning? This is starting to feel like a parsing/socket issue.

Tim

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 04:27 AM          Msg. 5 of 14
OK, could you please help me locate where to turn on loggin in the gateway?
thanks

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 04:32 AM          Msg. 6 of 14
Right click the IQFeed icon in the system tray, under support, launch diagnostics you will see a logging tab where you can enable and disable as you choose. I would just turn on all Lookup items for while you are testing. Just don't forget to turn it off when your done.

Tim

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 04:55 AM          Msg. 7 of 14
Here is what I am seeing:

FROM CLIENT LookupRequest 9296 2014-03-31 17:51:46 HIX,EURUSD.COMP,3600,500,0,159c0467-29b0-4402-a945-f9ea09132d7f,5000,s

I attached the logFile could you please take a look, it still looks like as if the gateway sends each message (each data point) individually, or is this to be understood as one single response message?

I just wonder, if this was a socket issue then why do I get each message containing exactly one full datapoint with ending CF flag? If the buffer was set too low in my socket then I would understand but then each message should be truncated which it is not.

Edited by bbmat on Mar 31, 2014 at 04:56 AM
Edited by bbmat on Mar 31, 2014 at 04:57 AM
Edited by bbmat on Mar 31, 2014 at 04:58 AM



File Attached: IQConnectLog1.txt (downloaded 790 times)

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 05:24 AM          Msg. 8 of 14
Ok, well the data is being returned, the line above is the request, and

the TO CLIENT LookupData 9296 lines below that are the data being returned.

As an example, from the server we will say we receive the string

ABC<CR><LF>DEF<CR><LF>GHI<CR><LF><!ENDMSG!><CR><LF>

In your first buffer you could get everything through the first letter G.

ABC<CR><LF>DEF<CR><LF>G

You would store the G, everything to the right of the terminating characters to a temp var.

Then you would process the complete lines that you have,

ABC<CR><LF>
DEF<CR><LF>

Then you will read from the buffer again, you would receive
HI<CR><LF><!ENDMSG!><CR><LF>

You would want to prepend the G you saved off to get your full messages and then clear your temp var.

GHI<CR><LF>
<!ENDMSG!><CR><LF>

If the message was larger you would continue your loop until you process the ENDMSG Line.

You should be able to step through your debugger and see this as well. Also, if you are using C#, make sure your BeginReceive and EndReceive logic is lined up right, so that the socket is not tossing data. I know some people have had trouble in that area in the past.

Tim

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 05:31 AM          Msg. 9 of 14
Tim Walter, thanks for your effort but I guess you misunderstand; My issue is that my socket only receives incoming data packets with the EXACT size and content of 1(!!!) single data point, one line (ending with "\r"). That is all each packet contains. I expect to receive hundreds of data points in one packet. It still looks to me that the gateway sends out line by line rather than all 500 requested data points at once.

So, currently I do not need to store anything in temp variables because I receive exactly one data point and it is easy to parse that into a POCO. My problem, however, is that I want to get many datapoints (optimally) the whole request in one or few large packets. I double checked and my socket logic does not impose any limits on incoming data sizes.

My socket logic seems perfectly fine. I receive exactly what I request, not one single data point is dropped. But my socket receives each single data point in an individual packet rather than having many data points lumped together in one packet. And right now I still have to suspect that it is the Gateway IQConnect which is the speed throttle here -> Sending each data point individually slows down things to a large degree.


Edited by bbmat on Mar 31, 2014 at 05:33 AM
Edited by bbmat on Mar 31, 2014 at 05:36 AM

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 05:55 AM          Msg. 10 of 14
Ok, so if you open the C# sample code for history socket and set a breakpoint on the HistorySocketForm.cs and request GOOG, what do you get? I got 24 bytes the first pull, one record as you mentioned, but about 25k on subsequent hits of this breakpoint with tick requests being closer to 250k bytes received.

Does your behavior match this?

Tim

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 05:57 AM          Msg. 11 of 14
Line 156 for the breakpoint, I was checking the received bytes values.

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 05:57 AM          Msg. 12 of 14
will need to double check and get back to you on this, but good idea to isolate whether the gateway is involved or not.

bbmat
-Interested User-
Posts: 30
Joined: Mar 31, 2014


Posted: Mar 31, 2014 08:00 AM          Msg. 13 of 14
My apologies,

problem solved! I wrote an AsyncSocket class some time ago and completely forgot that I internally already manage potentially split messages. Sorry to occupy your time with this issue. I verified and the Gateway indeed sends messages containing multiple data points, constrained by the "dataPointsPerSend" parameter.

DTN_Tim Walter
-DTN Guru-
Posts: 1238
Joined: Apr 25, 2006


Posted: Mar 31, 2014 08:07 AM          Msg. 14 of 14
Great, glad we got it figured out.

Tim
 

 

Time: Fri April 3, 2020 12:09 AM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003