|
|
Home
Search
Register
Login
Recent Posts
| Viewing User Profile for: DTN_Tim_Russell | |||||||||||||||||||||||||
| About | Contact | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Post Statistics | |||||||||||||||||||||||||
|
DTN_Tim_Russell has contributed to 41 posts out of 21184 total posts
(0.19%) in 7,291 days (0.01 posts per day).
20 Most recent posts:
Data and Content Support »
Time stamps on regional quotes May 11, 2005 08:37 AM
(Total replies: 4)
I'm looking into this. AFAIK, nothing should be sent in PST - if it is, we'll get it fixed soonest. I'll post more when I know more. Tim Russell Software Engineer DTN IQ & FinWin
Data and Content Support »
Numtrades for globex products is fubar Apr 1, 2005 09:44 AM
(Total replies: 8)
We're looking into this problem and should have a solution for this problem soon. I'll post more when I know more about what's actually causing this to not get reset any more. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
What does ".LNCR" mean, where I expect a number in the "Q" message on port 5009 ? Jan 27, 2005 09:57 PM
(Total replies: 10)
Dierk, We agree 100% on this one - data should never get corrupted. It's one of the bigger fixes I've got planned for going to QA testing in about 2 weeks. No promises yet, but that's the plan. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
Getting Regional quotes Dec 3, 2004 09:54 AM
(Total replies: 2)
Shouldn't those sizeof()'s be strlen()'s? Just wondering, not sure how IQFeed will respond to all those extraneous nulls. Tim Russell Software Engineer DTN IQ & FinWin This message was posted in a secure forum. Click here to access the topic where message was posted.
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 21, 2004 09:26 AM
(Total replies: 16)
The trade counters were originally for internal use, and used by our FinWin product - since it's a web service, it doesn't do streaming data, and this information is very useful to those customers. The reason we made this available in the streaming feed was because there seemed to be a bit of interest in it. However, it is NOT designed to be a method of letting you know data has been dropped in the feed. As for the corrupt messages - let me know what you see. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 21, 2004 08:32 AM
(Total replies: 16)
Ah - trades today. Okay. That value's maintained by the server - it's returned to you when you request it. However, it's not transmitted on a tick-by-tick basis, since it's easily calculable on the remote end. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 20, 2004 02:08 PM
(Total replies: 16)
Quote: Tim, Some of your comments concern me. I am monitoring the bandwidth between your servers and my PC and it is very high. After the initial burst (625KBytes/sec) I am getting around 300KBytes/sec sustained for the first hour. Last week I had an almost identical watch list and was receiving around 1/8th of this bandwidth which is closer to the figures you mentioned. The ratio of data downloaded to that output by IQFeed is around 1.2:1 which is much worse than last week. Clearly something is wrong. Even now (10:38am your time) I am getting ~180KB/sec from your servers. Is there any way you can determine what is being sent to me? (ID=200643) Dennis PS I'm still getting the "U,F|" Dennis - I've reposted your email here so that everybody can benefit. I was under a slight delusion as to how IQFeed handled outgoing data - basically I looked at code that has NOT been completed yet, that fixed a problem that is most likely causing your symptoms. Basically (and I hope you understand the basics of TCP/IP communication): - IQFeed sets TCP send / recv buffers to 256 Kbytes. - When incoming data is processed, a message is created for the client. - This message is sent to the client. No checking is (currently) being done to ensure that this message is sent in its entirety. This is fine so long as the 256k TCP buffer doesn't get full. - After the TCP buffer gets full, attempts to write to the socket will fail. - You will get corrupted data / lose data. The obvious problem here is that your code may not be processing messages from IQFeed fast enough. A couple of solutions: - Make your TCP recv buffer bigger. In java.net.Socket, I believe this is accomplished via the setReceiveBufferSize() method. - Read data from the incoming socket in one thread, and pass this data to another thread - via a queue or whatever - to be processed. Sorry for failing to notice this earlier. The recv buffer size fix is often offered, but I wasn't aware of why until I got one of our client guys to help me find the problem in the code. If this helps you, let me know. Now, for the amount of data you're seeing - are you getting this via the IQConnectionManager? If not, where are you getting these values? Only curious, as what I'm seeing going to your computer doesn't account for that much data. Thanks, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 20, 2004 09:15 AM
(Total replies: 16)
- 7MB sustained should be more than enough - at 9am EST, I'm seeing output at just under .2Mb on your watchlist. - Nice machine =) As I'm sure you guessed, no problems there. - I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output) Btw, as far as compression goes: I'm seeing 21KBytes/second on our server output, vs 160Kbytes/second (again, with your watchlist). I'm actually suprised about the "U,F" you're seeing in some fields - this is something you should NEVER see - it's part of our server feed before it's converted to the text feed that IQFeed puts out. I'll pass this on to the IQFeed guys to look at. Could be this message corruption is the root of your problem, though. Thanks for the input, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 19, 2004 10:28 AM
(Total replies: 16)
Dennis, Looking at our stored historical data vs what you sent, I see that those trades received were regular trades. Moreover, the volume on these trades (which is part of the historcal retrieval) is higher than the volume you have. The obvious reason for this is that data is being dropped passing between our servers and you. The "why" of the dropped data is what we need to pinpoint now; I'll need a bit more information from you. - What type of connection are you using to the Internet? - What type of machine is your application running on? - If possible, can you send me a private message with the list of symbols you're watching? This will give me a better idea of the bandwidth involved. Now, some background on why these things can impact your data reception. Our quote servers, after upgrades last month, have proven to be more than capable of keeping up with our data feed and transmitting it to customers in a timely manner. However, memory on these machines is not unlimited - we cannot queue data on our end waiting for the remote clients to be ready to receive it. Doing this would put our servers at the mercy of any client who wanted to break our servers just by not receiving data for an extended period of time. Timely (and complete) delivery of data is entirely dependant on the remote end's ability to read the data fast enough. While I'm not certain what symbols you are watching, with 1300 symbols being watched, I'm pretty sure this is generating a significant amount of data. Get back to me with your information, and we'll try to work from there. Thanx, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 19, 2004 08:34 AM
(Total replies: 16)
Dennis, If you could give me an example of a symbol you see this happening on, I can look into this. Thanks, Tim Russell Software Engineer DTN IQ & FinWin This message was posted in a secure forum. Click here to access the topic where message was posted. This message was posted in a secure forum. Click here to access the topic where message was posted. This message was posted in a secure forum. Click here to access the topic where message was posted. This message was posted in a secure forum. Click here to access the topic where message was posted. This message was posted in a secure forum. Click here to access the topic where message was posted. This message was posted in a secure forum. Click here to access the topic where message was posted.
IQFeed Datafeed Wish List »
Add Seconds & Milliseconds to Watchlist Update message Oct 8, 2004 02:00 PM
(Total replies: 22)
Quote: Information transmitted via TCP is guaranteed to be provided in order to the application, as it is a reliable protocol. UDP, on the other hand, is unreliable and data can arrive out of order, or not at all. I stand by my first post in this thread.IQFeed utilizes TCP for all transmission, so this isn't an issue. We're working on adding seconds to the data feed currently. I don't foresee adding milliseconds - as far as I know, the exchanges only send times with a precision of seconds. --- Original message by DTN_Tim_Russell on Sep 17, 2004 11:00 AM We ARE adding seconds into the data feed. Tim Russell Software Engineer DTN IQ & FinWin This message was posted in a secure forum. Click here to access the topic where message was posted.
IQFeed Datafeed Wish List »
How about sending the Time message once per second? Oct 7, 2004 12:28 PM
(Total replies: 4)
Well, the 75 seconds is about the maximum it's supposed to take, based on the fact that the feed should get timestamps once a minute. A failure in getting the timestamps is usually a result of network failure between the customer and our servers, not necessarily a failure in the TCP connection. I've tried to avoid too much detail in this area up to now, but it looks like I'm not going to be able to do that this time, so, here goes  1. Detecting a connection loss because your internet connection to your computer is disconnected - this is very simple and, if IQFeed doesn't currently have this functionality I'm sure it will in the future. 2. Detecting a connection loss caused by software failure on our end is also easily detected, as the operating system will handle closing out the sockets, causing IQFeed to do whatever it is it does to notify clients and attempt to reconnect to the server way before 75 seconds has past. 3. Now, the most difficult. When a route between an IQFeed instance and our servers goes down temporarily - or becomes congested, or whatever - data may temporarily be unable to be transmitted between client and server. This is the nature of TCP, and it's designed to take these things into account, waiting a set (sometimes very long) amount of time before declaring that the connection is no longer valid. IQFeed attempts to get around this by saying, "After 75 seconds, if I still haven't received a timestamp, attempt a reconnect". The need to wait a certain amount of time before giving up and attempting to reconnect is necessary, especially if the connection failed due to hardware problems somewhere along the line - attempts to reconnect every second are very likely going to be unsuccessful until the link is no longer congested, or the down hardware is back up, etc. Also, having arbitrarily short timeouts tends to cause a bunch of false negatives - i.e., "No data received for 1.5 seconds, attempt to reconnect!". You can see where this would be a problem. Let me recap a bit here: If a connection fails (server kicks you off, you unplug your cable, software crashes), the remote TCP end should be notified immediately. When a socket error doesn't occur, it's a very subjective thing to determine whether the TCP connection has 'failed' or is just stalled. In a nutshell, the 75-seconds is for 'stalled' connections, not necessarily lost connections. Tim Russell Software Engineer DTN IQ & FinWin |
|||||||||||||||||||||||||
