Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I just wanted to let you know how fast and easy I found it to integrate IQFeed into our existing Java code using your JNI client. In my experience, such things almost never go so smoothly - great job!" - Comment from Nate
"IQFeed version 4 is a real screamer compared to anything else I have seen." - Comment from Tom
"I just wanted to let u know that your data feed/service is by far the best!!! Your unfiltered tick data is excellent for reading order flow and none of your competitors delivers this quality of data!" - Comment from Peter via Email
"Can I get another account from you? I am tired of ******* going down so often" - Comment from George
"Very impressed with the quality of your feed - ******* is a real donkey in comparison." - Comment from A.C. via Email
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
"I'm satisfied with IQFeed. It's the most reliable and fastest quote feed I have ever used. Although I'm a resident in China, it's still very fast!" - Comment from Xiaofei
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
"Thank you so much - awesome feed, awesome service!" - Comment from Greg via Email
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
Viewing User Profile for: lrmiller9
About Contact
Joined: May 13, 2005 03:20 PM
Last Post: Mar 1, 2007 11:30 PM
Last Visit: Mar 1, 2007 11:30 PM
Website:  
Location:
Occupation:
Interests:
AIM:
ICQ:
MSN IM:
Yahoo IM:
Post Statistics
lrmiller9 has contributed to 11 posts out of 20020 total posts (0.05%) in 5,778 days (0.00 posts per day).

20 Most recent posts:
IQFeed Developer Support » Streaming data bandwidth throttling? Mar 1, 2007 11:30 PM (Total replies: 15)

That is good to hear that IB will run on Linux. Do you have a way to automatically start up TWS in the morning?
I was considering Genesis Day Trading as an automation broker. Any experience with them?
Larry

L Miller

IQFeed Developer Support » Streaming data bandwidth throttling? Mar 1, 2007 10:31 PM (Total replies: 15)

I love Linux. It handles preemptive multithreading and multiprocessing a lot better than Windows. However last I looked the major automated trading brokerages all run their API only on Windows. I would like to discover one whose API runs on Linux.

L Miller

IQFeed Developer Support » Streaming data bandwidth throttling? Mar 1, 2007 09:50 PM (Total replies: 15)

Hi,
On a dual core 3 GHz Xeon, when the market is fast, IQConnect uses close to 100% of the cycles. (I don't know if it is using one or both cores). For this testing, I am simply calling recv in a tight loop and ignoring the data. recv itself tests out to not take a significant amount of processing. I can run my processing on a second CPU so I am not concerned about that. But if IQConnect needs more cycles per second than a CPU can provide, it will get behind and discard data, corrupt data, disconnect (this has happened to me) or do something unpleasant.
At the point wher the data gets progressively farther behind, I am getting about 3000 watch records (P, and Q,) per second. Thus it takes 333 microseconds of cpu time for IQConnect to process one record on the average. That is one million machine cycles to uncompress the approximately 250 byte record, add unchanged data and whatever. This strikes me as something that could be made MUCH faster (in any language), as your letter suggests.
I think it probably should take no more than 200 cycles per output byte, which is 50,000 cycles.
It seems that the various high performance API data providers all do something to keep you from talking directly to their servers over the internet. However, if you can get DTN to let you write a client that does this it would be very good. Alternatively, if they would make IQConnect more efficient it would be very good.

L Miller

IQFeed Developer Support » Streaming data bandwidth throttling? Mar 1, 2007 05:15 PM (Total replies: 15)

I have discovered that IQConnect.exe is a processing cycle hog. When I am watching 1100 tickers, on busy days, IQConnect uses up such a high fraction of the processing duty cycle that my application's recv does not have enough processing cycles to just get the data from the socket to a buffer. This happens when I just ignore the buffer and run a tight loop calling select with a 100 millisec time limit and recv. Note that select will return when data is available - it does not wait fo rthe 100 millisec in this case.

By looking at the system milliseconds, it is clear that the recv call itself is being preempted so IQConnect.exe can have the cycles it needs. Thus recv takes so long to execute that the loop cannot keep the pipeline from filling up indefinitely. (Normally recv for a typical IQFeed packet of 17KB takes just a few microsoeconds, but when the market gets busy recv for teh same number of bytes takes up to 100 or 200 milliseconds.

There clearly is no way the application can work around this problem - IQConnect.exe will always hog all the cycles when the market is busy and you are watching a lot of tickers. So the only solution I can think of is for the application to detect this condition and cut back in the number of tickers watched when it happens. IQConnect.exe does not appear to discard older data in the socket pipeline or merge prints or do other things to avoid this overflow condition. (Perhaps it is best to leave this to the app - he may not want to lose old data, but the app developer needs to understand the problem in order to deal with it).

The way I detect this vicious cycle of data backing up is when I need new data, I look at how many bytes I can get from the socket with a loop of recv calls before recv returns 0 indicating I have all the bytes. If this number of bytes is more than 1,000,000 then I send r<ticker>\r\n 's to reduce the number of tickers being watched.

When things are running normally, the number stays under 250,000 bytes.

The processor I use has a big effect as you would expect. For example on a 1.3 MHz laptop
the problem occurs pretty much regardless of how busy the market is. But in a HP dual core 3 MHz Xeon, the problem only occurs on busy days and it runs OK most of the time.

Larry

L Miller

IQFeed Developer Support » Manual entry of username and password Feb 20, 2007 03:41 PM (Total replies: 1)

I want to use the IQFeed sockets API with C++ VS2005.
It looks like when my app calls RegisterClientApp, a window pops up where the user enters name and password. Is there a way for my app to automatically enter the name and password so manual interaction is not necessary?
Larry

L Miller


Jay,
Thank you. I will follow your guidelines. I am glad that your daily, weekly and monthly data are split adjusted.
I also will eagerly await your new server code. Being able to get snapshots efficiently will make it possible for for me to provide a capability that allows users to do custom screens for stocks that rise more than a specified percentage from the market open to the present time.
Larry Miller

L Miller


Split Adjustment

Are the stock prices (in particular the close prices) split adjusted in the history data?

If not, does IQfeed provide any information about when splits (or stock dividentd) occur?

Thanks,

Larry

L Miller


One more equestion:

Would the HD,MSFT,1;

command given during market hours get me the previous day's open and close or the present day's open and current price?

Larry

L Miller


Jay,

Thanks for your reply. Lt me see if I can clarify exactly the best approach to use.

I need to get the current price of about 50 stocks every 10 minutes between 11:00 AM and 11:59 AM. I also need to do this for a different set of stocks between 3:00 PM and 4:00 PM.

Also, once every 15 minutes between 10:00 and 11:00 AM market time, I need to get the current price of about 500 stocks. The set of 500 stocks that I am interested in will change each 15 minutes. Thus 2000 different stocks wil be checked during the one-hour period.

I also need to do this 500 stock check at 15 minute intervals between 2:00 PM and 3:00 PM market time.

If I do all of the above using the streaming data, then I will be ignoring the great majority of the streaming data.

Is there some way to slow down the rate of delivery of the streaming data? or specify that I only want it once?

I can certainly just ignore it but it seems wasteful of bandwidth. Also, I would worry that the data might overflow somewhere in the pipeline. I can certainly do socket receives continually and efficiently dump the data at the receive end, but I don't have control of intermediate steps such as the flow control between IQConnect and the server.

Alternatively, is there a command that I can give to just get the current price of a stock? or a set of stocks? Would it be more efficient to issue this command for each of the 2000 stocks that I need data for between 10:00 and 11:00 AM? (and do a similar thing between 2 and 3 PM).

I know that I can issue a lot of w<ticker> and r<ticker> commands to the streaming feed system to do what I need to do but as I said, it seems wasteful of internet bandwidth.
In fact I could issue w<ticker> for a set of tickers and as I get a data back beck for each ticker, I can send r<ticker> for that ticker. Then the next 10 or 15 minute interval interval I can do this again - send a bunch of <w<ticker> commands (up to 500 I understand) and then send r<ticker> commands aftern I get a packet for each ticker. Is this tie best way to do it?

Thanks.

Larry Miller

L Miller

IQFeed Developer Support » When will 2.3.0.2 be out of beta? May 13, 2005 04:06 PM (Total replies: 3)

All,
Is 2.3.0.2 more stable than 2.3.0.1 at this time?
Thanks,
Larry Miller

L Miller


I just subscribed to IQfeed and I am writing code to use the TCP/IP interface with Visual C++ to get Minute History data for US stocks.

From the postings on the forums, it looks like there are a lot of occurrances of bad data and long delays, interleaved data from multiple requests, missing !ENDMSG! codes, and disconnects that cannot be automatically reconnected (the system crashes or hangs when trying to reconnect).

A lot of these problems sound very difficult or impossible to work around particularly because the API does not seem to use the usual flags that you would expect in an asynchronous client server system running over the internet, like the data not being tagged with the request even though data from multiple requests may be interleaved.

This sounds awful and scary to me. Would it really be possible to run for 24 hours (or 16 hours) getting Minute History without getting into a lot of these problems. I would really like my program to be able to stay connected overnight and run for more than one day.

If not, does anyone know of another service that is more useable for getting minute hisroties on US stocks?

Thanks,
Larry Miller
miller@scientificinvestor.com

L Miller


Time: Sun March 7, 2021 2:30 AM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003