I'm just getting back into some iqfeed development and testing. I have 4.1 installed. I notice there is still an occasional hiccup with historical data transfers. Once upon a time, Natalie had sent a version with some extra debugs in it. Due to time constraints, I was unable to tackle it. Can we try this again and see if we can get the last of the bugs worked out with multi-threaded historical data transfers?

I second this request. When I request data from the the historical server from multiple connections, I get no speed increase whatsoever. Instead, it seems that the requests are queued and delivered in what seems to be sequential order.

Is there anyway to get the data faster? I have over 1000 symbols I need to query daily. Right now it seems as if Yahoo! is faster for EOD day data than the IQ service I'm paying for!

Also, does this problem occur on port 5009 (the update server)? I am thinking of assigning 50 symbols per connection to hopefully eliminate any kind of bottlenecks that having 500 symbols updating over one socket may incur. Is this a good idea?

As a plus, 4.1 seems to be much more stable than 4.0. At least the data won't just mysteriously stop transferring :-).


Thanks,

Brandon

Brandon, are you sure you are making requests simultaneously through multiple port 9100 connections? That is what I'm doing, and I can see that my bandwidth goes up dramatically as I do this. Downloads are quite fast.

This may not be a concern anymore. I modified my buffering code to account for the extraneous error message that will occur when a history request comes up empty (please see my post at http://forums.dtnma.com/index.cfm?page=topic&topicID=922. Retrievals appear much cleaner now. I have one final clean up problem to resolve. I'm hoping all is ok now.
Edited by stargrazer on Jan 8, 2006 at 03:54 PM

I am positive that I am making the requests simultaneously. I have no idea why it is behaving like this.

Just to clarify:
* What language are you programming in?
* Do you have multiple socket connections open to port 9100?
* Perhaps you have a snippet of code showing how you are doing your retrieval?

I am coding in C on the UNIX platform. I have multiple socket connections open to port 9100 through the form of multiple threads. Each thread has a connection to the server on port 9100.

I don't have any snippets to show atm, but I do know it's not any faster and I am not sure why. Have you experienced that at all?

Mid last year, I did multiple connections with Perl on an XP machine. Seemed to work ok. Nowadays I'm doing it with C# on XP. I pretty much maxed out my bandwidth in either situation.

However, as far as I know, IQFeed runs on a windows platform only. So.... what you are attempting to do isn't supported, I don't think.

Anyway, without code snippets or debug traces, it is kinda hard to help point the way.
Edited by stargrazer on Jan 8, 2006 at 06:42 PM

I did some more troubleshooting to see if there were any further issues with my code. My current test scenario consists of running through about 250 symbols and requesting 5 days of 1 minute bars, 5 days of 3 minute bars, and 200 days of daily data. I use a series of 75 buffers and socket connections in asynchronous mode using C#. I do a for loop to open 75 connections and issue the retrieval commands for the first 25 symbols and their three data requests each.

As each request completes, I wait for a group of three buffers to free up before starting the next symbol and its three data requests.

Each buffer has a use count so I know for how many requests it was used.

When nearing the end of the run, I end up with about 20 buffers still waiting for data, and my interface bandwidth graph goes down to zero. There is a pause of a few minutes and I may get buffer completions for another five or so buffers.

I end up with about 12 or so buffers that don't get completed.

The incomplete buffers have processed a varying number of lines. In this case I only had incompletions on Daily data and 3 minute bar data. All my 1 minute bar requests completed successfully. (this could be a red herring or not--later indications are that this is a red herring).

The other interesting fact is that the symbols not completed are from the intial list of 25 symbols I used to do a fast open (from within one thread) of the 75 port connections.

This would tend to indicate there are some asynchronous issues with opening ports quickly. The list of symbols outstanding changes with each run.

Here is a list what I get at the end:
Columns:Symbol, 0=Daily/180=3min,number of lines recieved,buffer#, useage count

  
  AAP    0  81 15 1
  ABC  180 224 29 1
  ABD    0   1 33 1
  ABFS   0 122 36 1
  ABI    0 122 42 1
  ABK    0 122 45 1
  ABM    0  41 48 1
  ABMD 180  75 50 1
  ABN    0  41 54 1
  ABP    0  41 57 1
  ABV  180 222 68 1 

Here are the first 25 symbols requested:

AA
AACC
AACE
AAI
AAP
AAPL
AATI
ABAX
ABB
ABC
ABD
ABFS
ABGX
ABI
ABK
ABM
ABMD
ABN
ABP
ABS
ABT
ABTL
ABV
ABX
ABY

I hope this is useful for tracking down the issue. If you need further info, please let me know.
Edited by stargrazer on Jan 8, 2006 at 07:02 PM

As a follow up, I did the request with the 25 symbols only. I ended up with 22 incomplete requests. ... While writing this, I am now down to 14 incomplete requests. The responses appear to be issued sporadically. Each time I had a burst of buffer completions, I had bursts on my Internet connection. This would tend to indicate that there is some issue between your servers and your client on my workstation. ... I was about to abort. I just had another internet burst and am down to 8 incomplete buffers.

One would think that everything should complete in one big chunk. Are we getting TCP timeouts?

... Ok, I'm still at 8 incomplete buffers. Hopefully this info helps. Please let me know if you need further info.
Edited by stargrazer on Jan 8, 2006 at 07:14 PM
After waiting 'long enough', this is the final list of outstanding buffers:

 ABAX   0  81 24 1
 ABC    0 122 30 1
 ABFS   0 162 36 1
 ABMD   0 162 51 1
 ABTL  60 374 64 1
 ABTL 180 136 65 1
 ABV  180 285 68 1
 

There are some similarities with the first, but there are differences as well... tending to indicate something random somewhere.
Edited by stargrazer on Jan 8, 2006 at 07:20 PM
Further testing and update: If I put a 100 ms timeout between each port open action, the last few ports 'drag' a bit at the end to complete. If I put a 200 ms timeout between each port open action, all ports appear to complete in a reasonably timely fashion. Summary: there is definitely some timing issues on the client and/or server side when opening a number of ports in a tight loop.
Edited by stargrazer on Jan 8, 2006 at 08:32 PM
Another point of information which may or may not be of use: I open the 75 sockets in a tight loop. I think proceed to generate the history requests. When generating the initial 25 sets of 3 requests with no inter-request delays, some requests don't complete. When I put a 200 ms delay between each set of three requests, requests will complete as expected.
Edited by stargrazer on Jan 8, 2006 at 09:36 PM
I redid my 2600 symbol run. 38 requests did not complete. ... do you have enough info to find a solution?
Edited by stargrazer on Jan 9, 2006 at 05:04 AM

I believe this is either the same issue or a related issue to the one posted here.

http://forums.iqfeed.net/index.cfm?page=topic&topicID=877

Do the problems in that thread seem to be the same as what you are experiencing?

I'm not exactly sure if it is the same thing or not. I seem to be able to do large transfers ok. It is when I try to initiate a number of transfers simultaneously is when I get my problems. Try initiating 75 simultaneous transfers through 75 ports and see what you get. I don't get data corruption, I just get incomplete transfers (which, in some cases, does appear to be the same thing as in the other thread). I can recreate the incomplete transfers quite readily.
Edited by stargrazer on Jan 9, 2006 at 09:25 AM

After doing some initial testing, I am fairly positive that these are different issues. I am seeing some similar results as you are however i want to make sure before digging into this too far.

Have you tried turning on the IQConnect Logs?

( see the developer documentation for information on turning logging on)

The log file will show all commands recieved from and sent to the client from IQConnect.

Is it possible for you to run a small symbol set and email that to us?

from my testing each set of 25 symbols (with 3 requests per symbol) produces about 2.5-3mb of log file so a 25symbol set would probably be best.

Thanks
Edited by DTN_Steve_S on Jan 19, 2006 at 02:17 PM
Edited by DTN_Steve_S on Sep 19, 2011 at 11:00 AM

After further testing, I am able to get the same results you describe by slowing down the connection requests to get better results.

however, I am getting 100% response from IQConnect on requests that IQConnect recieves.

You should be able to verify this through the IQConnect Logging. It should show that for every command recieved from the client, a data set (terminated by !ENDMSG!) is returned.

The key here (as you mentioned) is how quickly the sockets are opened. It does take a few milliseconds from the time you issue a Connect() until the time that the Listening Socket on IQconnect is ready to accept another connection attempt. Upon Initial review i believe this to be simply the nature of socket communication rather than a bug in the code. Unfortunatly, to my knowledge there isn't a way to determine from your end if the socket actually connected.

From my internal testing (which will vary for you based upon your operating envirnment) a 50ms delay between Connect() attempts provided optimal results.

My workstation here is a P4 3.00Ghz with 1GB RAM running winXP.

You mentioned that a run of 2800symbols resulted in 38requsts not returning. This to me would indicate that one of your sockets didnt actually connect (38*75 = 2850).

If you would Like, as you mentioned in your initial post, we can get you a build of iqfeed with more verbose logging to verify that this is the issue.

In clarification, there are actually two stages:
1) I use a loop to open 75 async sockets
2) I then use a loop to send the requests

So, just to confirm, the opening of the sockets doesn't appear to be the problem, as I think you are indicating above. So.. I should have 75 fully opened and waiting and ok sockets to your local client after I've performed stage 1.

The problem seems to be when I do the second stage and perform the requests. If I do 75 requests without any inter-request delays, random requests will return partial data. If I put in 100 - 300 ms delay between each set of three requests, things appear to work much better.

I'll pull together some logs later on today.



When you mention 'client' in your message above, are your referring to my application or your local iqfeed client to which my application talks?

The 2800 was a guesttimate. I think my current runs request 2602 symbols, with three durations each.

Take a look at my logs and see what you see. If we need more info, send me a verbose client and I'll collect additional info.

Thanx.
Edited by stargrazer on Jan 10, 2006 at 01:31 PM

Let me further Clarify.

I am referring to your app being the client.


I was actually Indicating that from my testing, opening the sockets Is the problem.

When my test app loads, I Create() and Connect() 75 sockets in a loop.

I then loop through the 75sockets and send History requests (25 symbols, 3 requests each symbol).

I found that when I delayed between the History requests, it had no affect on how many returned data.

When I delayed for 50ms in between each set of Create(), and Connect() calls, It resolved the issue.

We will continue to look into this from our end as well. Until then, as a temporary fix, you shoudl be able to add a small delay in the socket creation loop and get good data from your app.

The logs you provide will hopefully be able to clear up any confusion that still exists.

I put the delays in. For short to medium runs, things seem to be working fine. For long runs, I still have some sort of issues with buffers not completing.

The log files, to my untrained eyes, are gibberish. I think you have major threading problems. There are missing characters, extra characters, and funny characters. In addition, you need to add some sort of line markers to distinguish which line belongs with which port/thread/symbol. Might this indicate problems in your main stream code?

Here is an exerpt:

 
TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-10 11:42:00,25.35,25.35,25.35,25.35,4100,500



TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-09 14:09:00,55.2300,55.2300,55.2300,55.2300,93388,
TO CLIENT 
TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-09 15:54:00,26.70,26.65,26.65,26.70,1851300,15600

045


TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-06 14:30:00,85.28,85.24,85.24,85.28,1254900,4400


TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-09 09:56:00,9.8900,9.8900,9.8900,9.8900,26688,100

30642



TO CLIENT (LOOKUP)=2006-01-10 20:54:19: 2006-01-09 13:29:00,27.04,27.04,27.04,27.04,497100,1000


3562

 

I'm now trying something different. I ran my HD, HM, and HM for each symbol separately using only 25 ports. The HD and HM(1) worked well. My HM(3) stopped with 5 of 20 buffers non completed. In this case the symbols that did not complete were random. I don't think the timing stuff you mentioned to try is working in all cases.

   
  PWAV 180 345 2 89
  CAMD 180 316 5 21
  PL 180 354 8 95
  HBC 180 358 12 47
  ADRX 180 306 13 3

From what I see in my software, it appears to be getting and processing all data it recieves from the local IQFeed engine. But I can't prove that unless I have a log format change for your software which will allow me to match log lines with ports and symbols. In addition, automated log file matching won't work until the 'garbage' characters in the log file are cleaned out.

Edited by stargrazer on Jan 11, 2006 at 06:02 AM

I will look into the bad characters in the Log file...I have not seen this issue before

Would it be possible for you to email me a copy of your program so I can do some testing using your program while runnign IQFeed in debug?
Edited by DTN_Steve_S on Sep 19, 2011 at 11:00 AM

Unfortunately, the programs will not run self contained. They are dependant on SQL and QuantDeveloper from SmartQuant. I can send you some of the source modules if you would like to take the relevant bits out, fix them and make it all right. It is written in MSVS C#.

I now remember a thought I had earlier today which I was reminded of from the other data handling problem thread: http://forums.iqfeed.net/index.cfm?page=topic&topicID=877&start=16. My code handles partial buffers and intermittent sends. I'm wondering if the IQFeed engine performs any buffer flushes with the '!ENDMSG!' command to make sure all data has been sent to the client.

If it is necessary to put delays in like this, it is usually a sign of two
problems in the dtn server, (i.e. the dll that we connect to as clients)
that dtn could fix if motivated:

(1) when you create your server, there is an option for specifying
the number of server connection attempts that are queued. I think
the default is 5. Of course, boosting this to 50 would help alot.

(2) you are supposed to do as little work as possible after an
accept() returns, and as quickly as possible create a new thread
to handle the request. This way, accept requests are handled
quickly, and fewer requests build up in the queue.

We were already are aware of the buffer. This is essentially what is causing the problem. However, after testing, it was determined that we already have it set at the optimal length for overall server performance. If we simply increase the buffer, we will not be resolving the issue since there is no buffer size that will work for all cases. As such, it was determined that, currently, the best solution for all parties involved is to handle this client side.

You are correct about handling accept request and have essentially described the function we have in place.

As for testing if the issue in the thread is caused by this, or by something else, the primary concern I have is verifying that all of the sockets are getting a good connection to IQConnect.

The current logging should suffice for these purposes.

Can you verify for me that the number of requests recieved from the client equals the number of sockets you are opening? You should see a entry similar to the following for every request you issue that IQConnect recieves.

FROM CLIENT (LOOKUP)=2006-01-12 09:15:33: HM,CSCO,5,3

I'm pretty certain that the client requests I recieve is the same as the number of sockets being opened. I DO receive data on all sockets, and I believe it arrives on the correct socket. It is just that at some point in time I do not obtain all lines. I may get the first 25% or 50% or 75% or some such, and then for some reason, the remainder of the lines never show up for what ends up being 10 to 30 requests out of 7000 to 8000 requests.

I'd like to reiterate my request that the (LOOKUP) line has a port number or something in it so it can be uniquely identified and tracked.
Edited by stargrazer on Jan 12, 2006 at 10:34 AM

If you are recieving data on all ports then the socket is obviously making a good connection (my apologies if you mentioned this before). This leavess us with a similar situation as to what skearns23 is reporting in the other thread which we are currently looking into.

We will look into adding more functionality to the log file, however, it may not be able to get included until the next release.

When these changes are done, we will get you a build that you can use until the fixes are implemented in an official release.

Thank you for all the information you have provided. Please continue to keep us updated on any new information you have on the issue.