||Jun 13, 2005 12:02 PM
||Mar 21, 2017 03:30 AM
||Mar 21, 2017 03:46 AM
Right Here & Now
stargrazer has contributed to 237 posts out of 17947 total posts
(1.32%) in 4,398 days (0.05 posts per day).
20 Most recent posts:
There are some differences in handling between IQFeed and IB.
Symbols in IQFeed are completely textual and obtained through a 'mktsymbols' text file.
For IB, to generate trades and request data through their API, contract numbers are typically required. To request contract numbers for equities traded on the primary exchanges, IQFeed symbols can typically be used directly for the lookup in the IB API. For options and futures, you have to perform lookups through the IB API based upon underlying symbol, currency, expiry dates, strikes, etc... Then use that contract number to make the request through the other portions of the IB API.
If the IQFeed people have these lookup tables as another service, cool.
But for us do-it-yourselfers, code needs to be written. I have written code to parse the IQFeed market symbols text file, pull out the options/futures info, and use that to look up contract details in the IB API.
If you want to wade through a bunch of stuff I've done:
An IQFeed provider:
An IB provider:
Send me a message at email@example.com if you'd like more instructions.
All this is based upon the fact that I start with the mktsymbols file, decode it, and use it to request data from IQFeed, and to generate trades with IB. To go the other way around, if you have IB contract info, I have some other code in the same repository which will build IQFeed symbols from contract info and underlying symbol names. There are some futures and options underlying symbol which require a manually composed lookup table, such as GC in InteractiveBrokers and QGC in IQFeed.
Edited by stargrazer on Mar 21, 2017 at 03:41 AM
A market replay would be best. At minimum, one or more futures (maybe the ES as mentioned, as they run almost 24 hours, and could rotate on 24 hours). Bonus would be one or more equities, maybe a few OPRA options to go along with the chosen equities.
IB runs what appears to be pseudo random values on their edemo account, but at least there is a time varying feed to run basic operational tests.
The following line in the symbols file has an extra \t before the exchange.
CS.17.CB\tCREDIT SUISSE NEW YORK 1.375% 05/26/17\t\tNYSE\tNYSE\tBONDS\t\t\t
Looping through the string looking for commas is the way I do it. I pull the delimited character strings out and process as required.
For converting characters to native values, you can try boost::lexical_cast, which uses in place iterators, so should be robust and fast.
For extra pizzaz you could try boost::spirit, something with which you would design a parser, and parse the whole thing and give you the appropriate converted values. Possibly a bit overkill as it will parse and convert more than just the few fields you might need. Or mix dynamic field sets with boost::spirit.
Two questions regarding options in the Market Symbols file:
1) In the following line:
AGII11418A36.36 AGII JAN 2014 C 36.360 OPRA OPRA IEOPTION
The option base symbol is AGII1. But in the descripion, the underlying shows as AGII. What is the 1 after the symbol, but before the four digit date, represent? There are a number of symbols
referencing the underlying this way.
2) The question is in the third set of symbols:
These I know are regular options:
SPY1304J145 SPY OCT 2013 C 145.000 OPRA OPRA IEOPTION
SPY1319J145 SPY OCT 2013 C 145.000 OPRA OPRA IEOPTION
These I know are mini options:
SPY71319J145 SPY OCT 2013 C 145.000 OPRA OPRA IEOPTION
But what are these options? With symbol SPYJ? The underlying in the description indicates SPY.
SPYJ1304J145 SPY OCT 2013 C 145.000 OPRA OPRA IEOPTION
SPYJ1319J145 SPY OCT 2013 C 145.000 OPRA OPRA IEOPTION
When calculating options parameters, one of the parameters is 'risk free rate of interest'.
Does any one have any suggestions for which symbols to use to obtain that rate for US based symbols? For some non US symbols, I use LIBOR which are symbols like ONLIB.X, 1WLIB.X, 2WLIB.X ....
What would be the USD equiv?
The symbol GLD has options available to it. Option naming doesn't appear to be consistent. For example, for a 2012/02/10 expiry for a GLD call is 'GLD1210B167'. 120210 is a Friday. On the other hand, for GLD option dated 120518, the option actually has a Saturday date: 'GLD1219E86'. Is there a method to this madness? I see IB has similar dating, so I believe it is official, but why?
At the time I wrote the parser for parsing the file, inertia led me to write a text parser for the description. I have subsequently found the decoding ring for decoding the symbol. I'll try that.
Has the XAUUSD.FXCM symbol been changed? I do not see it in the mktsymbols_v2.txt file.
In the mktsymbols_v2.txt file, quite a number of futures symbols have their descriptions truncated. I had been using the description to determine month/year of the future.
One random example:
XNZ13, SOCAL PIPE SYNTHETIC FUTURE DECEMBER 20
Do you know if the description field will be lengthened to accomodate a full description?
You definitely need to register for the developer docs. What you are seeing is the regular time stamp from the client. So, what you are seeing is a good thing... you have a good connection.
Based upon what I've encountered in terms of real time quote, market depth, and book walking, I think trying to make use of any quote information at the time of trade is an exercise in futility.
If you are trying to back test strategies based upon quote level information (which implies book level information), you are probably best off recording quote/trade changes through a trading day, and then back testing based upon what you've recorded as being important.
Bid/Ask values and sizes are very nebulous/transient in nature. You also have to be aware of the your round trips: the time to get the order to market, and when it gets executed (ie slippage), which is highly variable. And are you simulating market orders or limit orders? Simulations can have a hard time simulating 'touch' scenarios.
In the end, for my simulations, I record quotes during the day, and simulate off that stream, pretty much ignoring trades but for their trending and volume indications.
From an IQFeed perspective, the only way to backtest option based strategies, or, for that matter, any equity based strategy using primarily quotes, is to design your program to collect and save live data during trading hours. Then use the data you've saved to run through your backtesting application.
I use IQFeed's historical daily data to look for interesting trading candidates, then use the above mechanism to capture intraday details for testing.
Because options are sparsely traded, ticks (trades) don't arrive very often, and therefore it is difficult to build bars. The only alternative is to collect quotes and use them for your own bar building. Using the mid-point between bid and ask is common. This is meant for collecting data during live sessions.
If you are trying to build bars from history, the bars you get are typically sparse, and of zero height, basically due to the fact that trade data is sparse. If you want continuous range bars, you are pretty much SOL when using the history files, as the files only provide quotes during a trade.
Edited by stargrazer on Aug 10, 2010 at 04:07 PM
I agree with taa_dtn: "For back-testing real-time trading systems you need the historical data to match exactly what was sent on the real-time feed. That includes bad ticks, ticks that weren't properly split-adjusted, and split-adjusted trades that took place before the announcement of the split was transmitted on the feed."
The historical values need to reflect what was traded in real time in order ascertain that trading algorithms are doing what they are supposed to be doing.
It seems that many (more than usual) news stories are coming up empty of content. A couple of examples:
The first as seen in my program:
N,FLY,21077737284,:BTU:,20100509 213915,Peabody submits A$15 per share cash offer for controlling stake in ...
The second as seen from telneting to localhost 9100:
Is this unusual?
It isn't easy. I think the seminal paper on that is "Inferring Trade Direction from IntraDay Data" by Lee and Ready. But there is another hitch. I read in some recent papers that the 'short on an uptick' rule biases historical data.
There is a registry entry to turn logging off. With their new version, they may be logging more detail, thus creating larger log files.
The open is the value of the first trade of the interval, the close is the value of the last trade of the interval, the high is the max(all trades of interval), the low is the min(all trades of interval).