Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I was on the phone with a friend who uses CQG and right after the Fed announcement, CQG was as much as 30 seconds behind DTN.IQ. Some quotes were off by as much as 15-18 cents. Your feed never missed a beat." - Comment from Roger
"I used to have *******, but they are way more money for the same thing. I have had no probs with data from DTN since switching over." - Comment from Public Forum Post
"This is an excellent value, the system is generous (allowing for 500 stocks) and stable (and really is tick-by-tick), and the support is fantastic." - Comment from Shirin via Email
"And by the way, have to say this. I love the IQFeed software. It's rock solid and it has a really nice API." - Comment from Thomas via RT Chat
"I use IQ Feed, Great stuff as far as data analysis information, storage and retrieval is concerned." - Comment from Public Forum
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"I ran your IQFeed DDE vs. my broker vs. a level II window for some slow-moving options. I would see the level II quote change, then your feed update instantaneously. My broker's DDE, however, would take as much as 30 seconds to update. I am not chasing milliseconds, but half a minute is unacceptable." - Comment from Rob
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
"As a past ******* customer(and not a happy one), IQ Feed by DTN is a much better and cheaper product with great customer support. I have had no problems at all since switching over." - Comment from Public Forum
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »Corrupted structure for historical data
Author Topic: Corrupted structure for historical data (6 messages, Page 1 of 1)

ovolansky
-Interested User-
Posts: 9
Joined: Apr 18, 2018


Posted: Apr 27, 2018 08:59 AM          Msg. 1 of 6
Hello,

I'm trying to receive historical data for approx 2000 symbols and after a while some records which I receive are not in correct structure.

It looks like the last row overwrite the first row and the message begins with: 238,!ENDMSG!,
Could you please advice me, what could be the reason? ( I tried to increase socket buffer, but without any improvement)

Thank you,
Ondrej

DTN_Steve_S
-DTN Guru-
Posts: 2090
Joined: Nov 21, 2005


Posted: Apr 27, 2018 09:22 AM          Msg. 2 of 6
Hello, this is almost certainly a case of either not properly dealing with partial messages you read off the socket or not properly clearing resources between requests.

By "partial messages", I'm referring to when you read from a socket into a buffer, there is no guarantee that the last characters read will be at a message break which means you have to save off the partial message you received and prepend it to the beginning of the data on the next socket read.

ovolansky
-Interested User-
Posts: 9
Joined: Apr 18, 2018


Posted: Apr 27, 2018 09:42 AM          Msg. 3 of 6
Hello,

I reused the script from your example, please see enclosed file. The strange thing is, that ENDMSG replaces the beginning of the original message, it is not just in the wrong order - so it looks that it happen already somewhere in the socket - as OnReceive only concatenate strings (in my case it is overwritten).

It happens only when I ask for longer period and only in very few cases. 1 of 300.



File Attached: vb_net.txt (downloaded 238 times)

DTN_Steve_S
-DTN Guru-
Posts: 2090
Joined: Nov 21, 2005


Posted: Apr 27, 2018 01:46 PM          Msg. 4 of 6
There are a couple changes to that file that could affect the reliability of the code (I am unable to tell since the changes call functions that aren't included). However, by modifying the same example app to adding a simple re-request loop that triggers when the ENDMSG is received, I was able to cycle through ~27K requests without error before I stopped it.

I'm not discounting the idea that the example code might have a bug in it that causes this. The example code is designed to simply demonstrate the communication and isn't very robust. Of course it is also possible that it could be a bug in the feed itself although that would be much less likely.

My recommendation to continue troubleshooting would be for you to examine the calls to BeginReceive and EndReceive to make sure they are always paired correct. For every call to BeginReceive, there should be exactly one call to EndRecieve and it should happen before the next call to BeginReceive.

I would also add an extra Array.Clear on the socket buffer array before calling EndReceive just to be 100% sure that any data you are processing is new off the socket instead of somehow leftover from the previous read.

You might also want to turn on logging in IQConnect (you can do this via the diagnostics app) to see what the feed is actually sending and compare this to what your app is receiving. Unfortunately, the logging in IQConnect will certainly affect performance of the feed and will likely result in a very large text file so it could be troublesome to work with.
Edited by DTN_Steve_S on Apr 27, 2018 at 01:48 PM

ovolansky
-Interested User-
Posts: 9
Joined: Apr 18, 2018


Posted: May 2, 2018 03:34 AM          Msg. 5 of 6
Hello,

I will try to do some testing on my side based on your advice.

Thank you,
Ondrej

ovolansky
-Interested User-
Posts: 9
Joined: Apr 18, 2018


Posted: May 3, 2018 02:20 AM          Msg. 6 of 6
Hello,

Finally I found an issue in the way I called the socket - I called it twice. Now it works without any problems.

Thank you for your support
 

 

Time: Thu May 23, 2019 2:14 PM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003