|
|
Home
Search
Register
Login
Recent Posts
| Viewing User Profile for: lrmiller9 | |||||||||||||||||||||||
| About | Contact | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| Post Statistics | |||||||||||||||||||||||
|
lrmiller9 has contributed to 11 posts out of 21185 total posts
(0.05%) in 6,924 days (0.00 posts per day).
20 Most recent posts:
IQFeed Developer Support »
Streaming data bandwidth throttling? Mar 1, 2007 11:30 PM
(Total replies: 15)
That is good to hear that IB will run on Linux. Do you have a way to automatically start up TWS in the morning? I was considering Genesis Day Trading as an automation broker. Any experience with them? Larry L Miller
IQFeed Developer Support »
Streaming data bandwidth throttling? Mar 1, 2007 10:31 PM
(Total replies: 15)
I love Linux. It handles preemptive multithreading and multiprocessing a lot better than Windows. However last I looked the major automated trading brokerages all run their API only on Windows. I would like to discover one whose API runs on Linux. L Miller
IQFeed Developer Support »
Streaming data bandwidth throttling? Mar 1, 2007 09:50 PM
(Total replies: 15)
Hi, On a dual core 3 GHz Xeon, when the market is fast, IQConnect uses close to 100% of the cycles. (I don't know if it is using one or both cores). For this testing, I am simply calling recv in a tight loop and ignoring the data. recv itself tests out to not take a significant amount of processing. I can run my processing on a second CPU so I am not concerned about that. But if IQConnect needs more cycles per second than a CPU can provide, it will get behind and discard data, corrupt data, disconnect (this has happened to me) or do something unpleasant. At the point wher the data gets progressively farther behind, I am getting about 3000 watch records (P, and Q,) per second. Thus it takes 333 microseconds of cpu time for IQConnect to process one record on the average. That is one million machine cycles to uncompress the approximately 250 byte record, add unchanged data and whatever. This strikes me as something that could be made MUCH faster (in any language), as your letter suggests. I think it probably should take no more than 200 cycles per output byte, which is 50,000 cycles. It seems that the various high performance API data providers all do something to keep you from talking directly to their servers over the internet. However, if you can get DTN to let you write a client that does this it would be very good. Alternatively, if they would make IQConnect more efficient it would be very good. L Miller
IQFeed Developer Support »
Streaming data bandwidth throttling? Mar 1, 2007 05:15 PM
(Total replies: 15)
I have discovered that IQConnect.exe is a processing cycle hog. When I am watching 1100 tickers, on busy days, IQConnect uses up such a high fraction of the processing duty cycle that my application's recv does not have enough processing cycles to just get the data from the socket to a buffer. This happens when I just ignore the buffer and run a tight loop calling select with a 100 millisec time limit and recv. Note that select will return when data is available - it does not wait fo rthe 100 millisec in this case. By looking at the system milliseconds, it is clear that the recv call itself is being preempted so IQConnect.exe can have the cycles it needs. Thus recv takes so long to execute that the loop cannot keep the pipeline from filling up indefinitely. (Normally recv for a typical IQFeed packet of 17KB takes just a few microsoeconds, but when the market gets busy recv for teh same number of bytes takes up to 100 or 200 milliseconds. There clearly is no way the application can work around this problem - IQConnect.exe will always hog all the cycles when the market is busy and you are watching a lot of tickers. So the only solution I can think of is for the application to detect this condition and cut back in the number of tickers watched when it happens. IQConnect.exe does not appear to discard older data in the socket pipeline or merge prints or do other things to avoid this overflow condition. (Perhaps it is best to leave this to the app - he may not want to lose old data, but the app developer needs to understand the problem in order to deal with it). The way I detect this vicious cycle of data backing up is when I need new data, I look at how many bytes I can get from the socket with a loop of recv calls before recv returns 0 indicating I have all the bytes. If this number of bytes is more than 1,000,000 then I send r<ticker>\r\n 's to reduce the number of tickers being watched. When things are running normally, the number stays under 250,000 bytes. The processor I use has a big effect as you would expect. For example on a 1.3 MHz laptop the problem occurs pretty much regardless of how busy the market is. But in a HP dual core 3 MHz Xeon, the problem only occurs on busy days and it runs OK most of the time. Larry L Miller
IQFeed Developer Support »
Manual entry of username and password Feb 20, 2007 03:41 PM
(Total replies: 1)
I want to use the IQFeed sockets API with C++ VS2005. It looks like when my app calls RegisterClientApp, a window pops up where the user enters name and password. Is there a way for my app to automatically enter the name and password so manual interaction is not necessary? Larry L Miller
IQFeed Developer Support »
Historical Data via TCP/IP - Minute History (HM) - data return order change May 15, 2005 11:17 AM
(Total replies: 10)
Jay, Thank you. I will follow your guidelines. I am glad that your daily, weekly and monthly data are split adjusted. I also will eagerly await your new server code. Being able to get snapshots efficiently will make it possible for for me to provide a capability that allows users to do custom screens for stocks that rise more than a specified percentage from the market open to the present time. Larry Miller L Miller
IQFeed Developer Support »
Historical Data via TCP/IP - Minute History (HM) - data return order change May 15, 2005 12:28 AM
(Total replies: 10)
Split Adjustment Are the stock prices (in particular the close prices) split adjusted in the history data? If not, does IQfeed provide any information about when splits (or stock dividentd) occur? Thanks, Larry L Miller
IQFeed Developer Support »
Historical Data via TCP/IP - Minute History (HM) - data return order change May 14, 2005 07:02 PM
(Total replies: 10)
One more equestion: Would the HD,MSFT,1; command given during market hours get me the previous day's open and close or the present day's open and current price? Larry L Miller
IQFeed Developer Support »
Historical Data via TCP/IP - Minute History (HM) - data return order change May 14, 2005 06:32 PM
(Total replies: 10)
Jay, Thanks for your reply. Lt me see if I can clarify exactly the best approach to use. I need to get the current price of about 50 stocks every 10 minutes between 11:00 AM and 11:59 AM. I also need to do this for a different set of stocks between 3:00 PM and 4:00 PM. Also, once every 15 minutes between 10:00 and 11:00 AM market time, I need to get the current price of about 500 stocks. The set of 500 stocks that I am interested in will change each 15 minutes. Thus 2000 different stocks wil be checked during the one-hour period. I also need to do this 500 stock check at 15 minute intervals between 2:00 PM and 3:00 PM market time. If I do all of the above using the streaming data, then I will be ignoring the great majority of the streaming data. Is there some way to slow down the rate of delivery of the streaming data? or specify that I only want it once? I can certainly just ignore it but it seems wasteful of bandwidth. Also, I would worry that the data might overflow somewhere in the pipeline. I can certainly do socket receives continually and efficiently dump the data at the receive end, but I don't have control of intermediate steps such as the flow control between IQConnect and the server. Alternatively, is there a command that I can give to just get the current price of a stock? or a set of stocks? Would it be more efficient to issue this command for each of the 2000 stocks that I need data for between 10:00 and 11:00 AM? (and do a similar thing between 2 and 3 PM). I know that I can issue a lot of w<ticker> and r<ticker> commands to the streaming feed system to do what I need to do but as I said, it seems wasteful of internet bandwidth. In fact I could issue w<ticker> for a set of tickers and as I get a data back beck for each ticker, I can send r<ticker> for that ticker. Then the next 10 or 15 minute interval interval I can do this again - send a bunch of <w<ticker> commands (up to 500 I understand) and then send r<ticker> commands aftern I get a packet for each ticker. Is this tie best way to do it? Thanks. Larry Miller L Miller
IQFeed Developer Support »
When will 2.3.0.2 be out of beta? May 13, 2005 04:06 PM
(Total replies: 3)
All, Is 2.3.0.2 more stable than 2.3.0.1 at this time? Thanks, Larry Miller L Miller
IQFeed Developer Support »
Historical Data via TCP/IP - Minute History (HM) - data return order change May 13, 2005 03:48 PM
(Total replies: 10)
I just subscribed to IQfeed and I am writing code to use the TCP/IP interface with Visual C++ to get Minute History data for US stocks. From the postings on the forums, it looks like there are a lot of occurrances of bad data and long delays, interleaved data from multiple requests, missing !ENDMSG! codes, and disconnects that cannot be automatically reconnected (the system crashes or hangs when trying to reconnect). A lot of these problems sound very difficult or impossible to work around particularly because the API does not seem to use the usual flags that you would expect in an asynchronous client server system running over the internet, like the data not being tagged with the request even though data from multiple requests may be interleaved. This sounds awful and scary to me. Would it really be possible to run for 24 hours (or 16 hours) getting Minute History without getting into a lot of these problems. I would really like my program to be able to stay connected overnight and run for more than one day. If not, does anyone know of another service that is more useable for getting minute hisroties on US stocks? Thanks, Larry Miller miller@scientificinvestor.com L Miller |
|||||||||||||||||||||||