Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I just wanted to say how happy I am with your service. I was able to download the API docs last week and I was able to replicate Interactive Brokers historical bar queries and realtime bar queries over the weekend. That was about one of the fastest integrations that I've ever done and it works perfectly!!!!" - Comment from Jason via Email
"I have been using IQFeed now for a few years in MultiCharts and I have zero complaints. Very, very rare to have any data hiccups or anything at all go wrong." - Comment from Public Forum
"If you want customer service that answers the phone, your best bet is IQFeed. I cannot stop praising them or their technical support. They are always there for you, and they are quick. I have used ****** too but the best value is IQFeed." - Comment from Public Forum
"Thank God for your Data Feed as the only Zippers I see are on my pants (LOL), and no more 200 pip spikes to mess up charts." - Comment from Spiro via Email
"Thanks for all of your help. Great customer service deserves to be recognized which one the reasons I've been a customer of DTN for over 10 years!" - Comment from Stuart
"Just a thank you for the very helpful and prompt assistance and services. You provided me with noticeably superior service in my setup compared to a couple of other options I had looked at." - Comment from John
"I've been using Neoticker RT with IQFeed for two months, and I'm very happy with both of the products (I've had IQFeed for two years with very few complaints). The service from both companies is exceptional." - Comment from Public Forum
"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"I had always used ******* but for the past 2 weeks have been trying DTN IQFeed. Customer support has been extraordinary. They call just to make sure your problem hasn't recurred." - Comment from Public Forum
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »Product Support »Data and Content Support »IQFeed Change To Simultaneous Requests (sudden "E,Too Many Connections" errors)
Author Topic: IQFeed Change To Simultaneous Requests (sudden "E,Too Many Connections" errors) (14 messages, Page 1 of 1)

DTN_Gary_Stephen
-DTN Evangelist-
Posts: 180
Joined: Jul 3, 2019


Posted: Dec 22, 2020 08:40 AM          Msg. 1 of 14
Effective immediately, IQFeed has made a change to how it handles history requests.

Previously, IQFeed would allow up to 15 simultaneous active history requests. "Active" meant they had to still be returning data. When one request finished returning data, the number of active requests decreased by one, and another request could begin without exceeding this limit.

Going forward, iqfeed will allow up to 25 historical requests in a second, with no regard to whether past requests have completed. If your application makes a lot of small history requests at a time, this may cause you to start receiving the "E, too many requests" error message.

Whenever you make a new history request, your number of active requests is increased by 1. Every 1/25th of a second (40 milliseconds) this number decreases by one. As long as you don't exceed a rolling average of 25 history requests per second, you will avoid the "too many requests" error. In other words: time your requests such that you do not exceed 25 per second.

This change was made in hopes of setting a more consistent, and easier to understand, bar for historical requests. It was difficult to know whether you had exceeded the old "15 simultaneous active requests" standard; a limit of 25 per second regardless of how long each request takes to run is much easier to work with. Also, it sets a consistent standard for history requests of all sizes, that will work with our server.

These changes pertain to request sent to the history server (port 9100) only. Level 1 (port 5009) requests, and other requests, are not affected. If you have any questions about this change, please ask us!

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

DTN_Gary_Stephen
-DTN Evangelist-
Posts: 180
Joined: Jul 3, 2019


Posted: Dec 23, 2020 04:56 PM          Msg. 2 of 14
UPDATE: The new limit has been raised from 25 historical requests in a second, to 50 historical requests in a second. We are all also sending out emails to users with more information about this change.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Dec 23, 2020 07:15 PM          Msg. 3 of 14
Hi, Gary. I'm confused about the purpose and implementation of the new limit.

Each day after the main trading session (at 7PM Eastern) I fetch tick history for all the symbols I'm screening. Currently there are 4090 of them.

My first reaction on hearing about the new rate limit was "That's fine; I'm OK with a lower limit, though I doubt I'm hitting 50 requests per second as it is." I run 8 threads, each of which performs history requests sequentially, so there are at most 8 requests in flight at any given time, and it takes a while for each one to complete and a new one to be issued. (The data has to be received, reformatted, compressed, and written to disk before a new request is issued.)

Yesterday, for example, the history fetch took 339 seconds, for an average of about 12 completed requests per second.

Today the history fetch took 2031 seconds, for an average of 2 completed requests per second. Wow; that's quite a change. It's slower than the days when I had a 6Mbps DSL line.

My best guess is that the majority of history requests finish quickly, so they're hitting the rate limit and are being delayed by as much as a factor of 6. However, a fair number of requests take a long time to finish, which drags down the average way below the nominal rate limit. High peak rate, low average rate.

So I'm curious as to whether your intent is to limit the rate at which requests are initiated, or the bandwidth demand on the servers. If the latter, the current approach may be too drastic; my connection is idle more than 80% of the time.

mathpaquette
-Interested User-
Posts: 22
Joined: May 18, 2018


Posted: Dec 27, 2020 03:59 PM          Msg. 4 of 14
Quote: Hi, Gary. I'm confused about the purpose and implementation of the new limit.

Each day after the main trading session (at 7PM Eastern) I fetch tick history for all the symbols I'm screening. Currently there are 4090 of them.

My first reaction on hearing about the new rate limit was "That's fine; I'm OK with a lower limit, though I doubt I'm hitting 50 requests per second as it is." I run 8 threads, each of which performs history requests sequentially, so there are at most 8 requests in flight at any given time, and it takes a while for each one to complete and a new one to be issued. (The data has to be received, reformatted, compressed, and written to disk before a new request is issued.)

Yesterday, for example, the history fetch took 339 seconds, for an average of about 12 completed requests per second.

Today the history fetch took 2031 seconds, for an average of 2 completed requests per second. Wow; that's quite a change. It's slower than the days when I had a 6Mbps DSL line.

My best guess is that the majority of history requests finish quickly, so they're hitting the rate limit and are being delayed by as much as a factor of 6. However, a fair number of requests take a long time to finish, which drags down the average way below the nominal rate limit. High peak rate, low average rate.

So I'm curious as to whether your intent is to limit the rate at which requests are initiated, or the bandwidth demand on the servers. If the latter, the current approach may be too drastic; my connection is idle more than 80% of the time.
--- Original message by taa_dtn on Dec 23, 2020 07:15 PM
Hi @taa, can I see your implementation? I mean, 50 requests per second means 3000 requests per minute. How do you get 2 requests per second?

Mathieu

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Dec 27, 2020 04:28 PM          Msg. 5 of 14
Hi, Mathieu. I don't object to showing you the code (other than embarrassment because some is old and crufty), but it's straightforward. This is a C++ socket-based app that has existed for more than a decade being run on a capable modern desktop machine with an AT&T gigabit fiber connection and no recent changes to the code or to its environment, so an overnight factor-of-six slowdown suggests to me that the new rate-limiting on the feed might be having more effect than intended.

Until the new rate-limiting started, I was averaging 12 completed requests per second, which seems pretty reasonable to me. Some loss of performance is caused by round-trip latency (for simplicity and error recovery, as well as compliance with previous IQFeed policy, each thread keeps only one request in flight at a time). But I suspect the main reason for the low average rate is that for some equities, an entire day worth of ticks takes a significant amount of time to prepare, transmit, and process. It's not just a matter of the maximum rate at which the app is allowed to issue history requests.

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Dec 27, 2020 05:27 PM          Msg. 6 of 14
Following up with some data from my logs. Minor correction; I see there are 7 active threads, not 8; the code says I chose that to stay under half of IQFeed's suggested limit of 15.

On 2020-12-22, my history fetch showed mean latency (time from sending request to arrival of first data record) of 0.22 seconds. Min was about 0.1, median about 0.24, max about 0.6. Mean data transfer time (from arrival of first data record to arrival of last data record) was 0.09 seconds. Min was 0, median about 0.04, max about 10.2.

So, some equities generate a lot of data. For data transfer alone, the best average completion rate I could ever achieve would be about 80 requests per second. On the other hand, some equities trade very thinly, so their transfer time is essentially zero, and the peak request rate could be extremely high. This is what I mentioned in my earlier postings.

On 2020-12-23, the mean latency was 1.22 seconds (!); min 0.4, median 1.0, max 3.5. Mean transfer time was 1.9; min 0, median 0.04, max 118 (!!!).

Things got really slow on 2020-12-23 for both latency and throughput. Maybe it was just an odd coincidence that this happened the same day we received notice of the new rate limiting algorithm. I'm curious to see what happens tomorrow.

DTN_Gary_Stephen
-DTN Evangelist-
Posts: 180
Joined: Jul 3, 2019


Posted: Dec 28, 2020 09:50 AM          Msg. 7 of 14
Quote: So I'm curious as to whether your intent is to limit the rate at which requests are initiated, or the bandwidth demand on the servers.


It's to limit the rate at which requests are initiated. If your software works as you describe - by not sending the next request until the last one is complete - then you should never afoul of the limit.


Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Edited by DTN_Gary_Stephen on Dec 28, 2020 at 09:50 AM

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Dec 28, 2020 10:39 AM          Msg. 8 of 14
Hi, Gary. Thanks for the update!

Just to clarify, is the rate limit per-connection, or per-IP-address (or some other key)? Since I have multiple threads issuing requests on separate connections, there are conditions under which more than 50 requests could be issued over the course of a second for all the connections in aggregate, though not for any individual connection.

I implemented a request-rate limiter last week, just in case. Maybe the observed slowdown was unrelated to the changes at your end. We'll see what happens today.

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Dec 28, 2020 06:12 PM          Msg. 9 of 14
Back to normal performance levels today.

mathpaquette
-Interested User-
Posts: 22
Joined: May 18, 2018


Posted: Dec 30, 2020 09:31 PM          Msg. 10 of 14
Hello everyone,

Just to inform you that I released IQFeed.CSharpApiClient 2.6.0 with the required change in order to limit 50 requests per second to avoid potential exception.

https://www.nuget.org/packages/IQFeed.CSharpApiClient/2.6.0

Code:
https://github.com/mathpaquette/IQFeed.CSharpApiClient/commit/cc28fa80b009e048065cb9f6cccf1a97a9482132


Please review or comment.
Mathieu

freemind
-Interested User-
Posts: 7
Joined: Jun 27, 2017


Posted: Jan 16, 2021 09:37 AM          Msg. 11 of 14
My changes with 2.6.0 works well but you need to limit to 8 concurrent clients. For some reasons, passed this limit, even with the throttler in place, I'm getting ErrorMessage: Too many simultaneous history requests., MessageTrace: E,Too many simultaneous history requests.,

Hope this helps.
Mathieu
Edited by freemind on Jan 16, 2021 at 09:38 AM

DTN_Gary_Stephen
-DTN Evangelist-
Posts: 180
Joined: Jul 3, 2019


Posted: Jan 18, 2021 04:40 PM          Msg. 12 of 14
For the record, the number of concurrent clients shouldn't be a factor. The server limits you to a total of 50 requests per second, no matter how many concurrent clients the requests are spread across. Let me know if you want me to dig deeper on this.

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

freemind
-Interested User-
Posts: 7
Joined: Jun 27, 2017


Posted: Jan 20, 2021 05:20 AM          Msg. 13 of 14
That's strange. I believe your limit is a little bit behind 50 requests per second. I mean it should be less than 50 not equal to 50. My code has been well tested and I'm using high precision timers in order to throttle the output and using 50 triggers the exception. Received confirmation from others that adjusting it a little below fixed the issue.

I'll let you know once I have more info.
Thanks,
Mathieu

taa_dtn
-DTN Evangelist-
Posts: 133
Joined: May 7, 2004


Posted: Feb 4, 2021 10:10 PM          Msg. 14 of 14
Just FYI, today I was seeing the response '!ERROR! Unknown Server Error code 0.' during history fetch from servers in the IP range 66.112.156.0/24. This occurred repeatably. I worked around it by adding a small delay between history requests.
 

 

Time: Wed April 21, 2021 5:26 AM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003