Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"If you are serious about your trading I would not rely on IB data for serious daytrading. Took me a while to justify the cost of IQ Feed and in the end, it's just a 2 point stop on ES. Better safe than sorry" - Comment from Public Forum
"I cannot believe what a difference it makes trading with ProphetX!" - Comment from Bruce in Los Angeles
"As a past ******* customer(and not a happy one), IQ Feed by DTN is a much better and cheaper product with great customer support. I have had no problems at all since switching over." - Comment from Public Forum
"It’s so nice to be working with real professionals!" - Comment from Len
"This is an excellent value, the system is generous (allowing for 500 stocks) and stable (and really is tick-by-tick), and the support is fantastic." - Comment from Shirin via Email
"I use IQ Feed, Great stuff as far as data analysis information, storage and retrieval is concerned." - Comment from Public Forum
"For anyone considering using DTN.IQ for a data feed, my experience with the quality of data and the tech support has been very positive." - Comment from Public Forum
"Version 4.0.0.2 has been working well for me and I appreciate that it is now a much tighter client to work with. I feel I can go to press with my own application and rely on a stable platform" - Comment from David in IA.
"I used to have *******, but they are way more money for the same thing. I have had no probs with data from DTN since switching over." - Comment from Public Forum Post
"I've been using Neoticker RT with IQFeed for two months, and I'm very happy with both of the products (I've had IQFeed for two years with very few complaints). The service from both companies is exceptional." - Comment from Public Forum
Home  Search  Register  Login  Blogs Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTN_IQFeed on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
Viewing User Profile for: jwelch
About Contact
Joined: Mar 15, 2018 10:13 PM
Last Post: Mar 15, 2018 10:38 PM
Last Visit: Mar 18, 2018 06:35 PM
Website:  
Location:
Occupation:
Interests:
AIM:
ICQ:
MSN IM:
Yahoo IM:
Post Statistics
jwelch has contributed to 1 posts out of 18926 total posts (0.01%) in 339 days (0.00 posts per day).

20 Most recent posts:
IQFeed Developer Support » Duplicate Q Messages Mar 15, 2018 10:38 PM (Total replies: 0)

TLDR; - Is it normal to expect the occasional duplicate Q message in the feed? Or are these meant to represent two genuine ticks that somehow end up with exactly the same data (including TickID?)

Details:
After doing some analysis on the feed data I receive, there are occasional duplicate tick messages (as in they have exactly the same crc64 value)

For Example, here is a summary of the duplicate lines found in a sample feed from 3/7/18.


Writing CRC Collisions to: IQFeed (3-7-2018).crc
Writing duplicate entries to: IQFeed (3-7-2018).dup
Processed 11734028 ticks
Found 2045 duplicate lines in the file
T: 0
Q: 1983
P: 2
S: 0
F: 60
CRC: 0 - No collisions

Ignoring the F and P duplicates as those might expected... here is a sample of some Q messages which had the same CRC64 value.


Duplicate Line Found on Hash [f612fb7a7fa49e97]
Line 1 [864989]: Q,HPQ,7,23.8700,-0.0800,-0.003340,366424,23.9506,23.7100,23.8600,23.8700,1100,1300,,0.2406,,23.7451,23.9500,0.0100,,,N,,,,-0.0800,0.0000,10.33,-0.970115,-0.0100,-0.0100,0.1249,1,0.010080,39086313,14,4,,9878149,,,,,1897,23.8356,877019989,N,,7,11,,200,09:44:32.512085,7,23.8700,200,09:44:32.512085,01,7,23.8700,200,09:44:32.512085,7,ba,09:44:32.595686,09:44:32.595686,03/07/2018,03/07/2018,03/07/2018,
Line 2 [864991]: Q,HPQ,7,23.8700,-0.0800,-0.003340,366424,23.9506,23.7100,23.8600,23.8700,1100,1300,,0.2406,,23.7451,23.9500,0.0100,,,N,,,,-0.0800,0.0000,10.33,-0.970115,-0.0100,-0.0100,0.1249,1,0.010080,39086313,14,4,,9878149,,,,,1897,23.8356,877019989,N,,7,11,,200,09:44:32.512085,7,23.8700,200,09:44:32.512085,01,7,23.8700,200,09:44:32.512085,7,ba,09:44:32.595686,09:44:32.595686,03/07/2018,03/07/2018,03/07/2018,
Duplicate Line Found on Hash [cada7e85403bc991]
Line 1 [866805]: Q,CTL,7,18.1000,0.0400,0.002215,501587,18.1600,17.8802,18.0900,18.1000,900,800,,0.2798,,17.9100,18.0600,0.0100,,,N,,,,0.0400,0.0000,8.19,-0.964172,-0.0100,-0.0100,0.1900,1,0.015459,19364502,14,4,,12264762,,,,,2470,18.0065,878202946,N,,7,7,,900,09:44:34.129734,19,18.1000,900,09:44:34.129734,01,19,18.1000,900,09:44:34.129734,19,ba,09:44:34.138719,09:44:34.138719,03/07/2018,03/07/2018,03/07/2018,
Line 2 [866807]: Q,CTL,7,18.1000,0.0400,0.002215,501587,18.1600,17.8802,18.0900,18.1000,900,800,,0.2798,,17.9100,18.0600,0.0100,,,N,,,,0.0400,0.0000,8.19,-0.964172,-0.0100,-0.0100,0.1900,1,0.015459,19364502,14,4,,12264762,,,,,2470,18.0065,878202946,N,,7,7,,900,09:44:34.129734,19,18.1000,900,09:44:34.129734,01,19,18.1000,900,09:44:34.129734,19,ba,09:44:34.138719,09:44:34.138719,03/07/2018,03/07/2018,03/07/2018,
Duplicate Line Found on Hash [3ee933e4adea4f37]
Line 1 [869514]: Q,T,7,36.7350,-0.1350,-0.003662,1214701,36.9800,36.5600,36.7400,36.7500,1000,1300,,0.4200,,36.5900,36.8700,0.0100,,,N,,,,-0.1350,0.0000,7.70,-0.952953,0.0100,0.0100,0.1450,1,0.011433,225609215,14,4,,17462057,,,,,5300,36.7725,878205011,N,,11,5,,100,09:44:35.459813,19,36.7350,100,09:44:35.459813,01,19,36.7350,100,09:44:35.459813,19,ba,09:44:37.923864,09:44:37.923864,03/07/2018,03/07/2018,03/07/2018,
Line 2 [869516]: Q,T,7,36.7350,-0.1350,-0.003662,1214701,36.9800,36.5600,36.7400,36.7500,1000,1300,,0.4200,,36.5900,36.8700,0.0100,,,N,,,,-0.1350,0.0000,7.70,-0.952953,0.0100,0.0100,0.1450,1,0.011433,225609215,14,4,,17462057,,,,,5300,36.7725,878205011,N,,11,5,,100,09:44:35.459813,19,36.7350,100,09:44:35.459813,01,19,36.7350,100,09:44:35.459813,19,ba,09:44:37.923864,09:44:37.923864,03/07/2018,03/07/2018,03/07/2018,


The number in square brackets e.g. 'Line 1 [#]' is a local message identifier. Clearly the messages are received very close to one another in time.

This is with the field parameters as follows:

S,CURRENT UPDATE FIELDNAMES,Symbol,Exchange ID,Last,Change,Percent Change,Total Volume,High,Low,Bid,Ask,Bid Size,Ask Size,Tick,Range,Open Interest,Open,Close,Spread,Settle,Delay,Restricted Code,Net Asset Value,Average Maturity,7 Day Yield,Extended Trading Change,Extended Trading Difference,Price-Earnings Ratio,Percent Off Average Volume,Bid Change,Ask Change,Change From Open,Market Open,Volatility,Market Capitalization,Fraction Display Code,Decimal Precision,Days to Expiration,Previous Day Volume,Open Range 1,Close Range 1,Open Range 2,Close Range 2,Number of Trades Today,VWAP,TickID,Financial Status Indicator,Settlement Date,Bid Market Center,Ask Market Center,Available Regions,Last Size,Last Time,Last Market Center,Most Recent Trade,Most Recent Trade Size,Most Recent Trade Time,Most Recent Trade Conditions,Most Recent Trade Market Center,Extended Trade,Extended Trade Size,Extended Trade Time,Extended Trade Market Center,Message Contents,Ask Time,Bid Time,Last Date,Extended Trade Date,Most Recent Trade Date


Note that TickID is included in the message, and is identical on lines which are found to have the same hash.

So, given the rather low occurrence of these duplicates (< 0.02%) this is probably not much of a concern for most analyses, but I was curious if this is expected behavior in the feed either way.

Thanks!


Time: Sat February 16, 2019 1:00 AM CFBB v1.2.0 16 ms.
© AderSoftware 2002-2003