First, I want to say that I think this is a great feature. Thanks for making it available.

1. What methodology is used for closing a bar? Is it based on IQFeed's clock (e.g. send a BC message when IQFeed server time >= bar time) or does it wait to get a trade with a timestamp > bar time?

2. In most circumstances, the last BH bar received will be an incomplete (but up-to-date) bar. However, other than checking the bar time against my system clock, there doesn't seem to be a way to know that this is the last BH bar in the stream. Comparing times from two separate clocks can be problematic for obvious reasons - would you consider identifying this last incomplete bar with something like "BI" instead of "BH" ?

3. A major limitation of using streaming bars is that one never really knows where the market is when the bar arrives (this is especially true is less liquid names). For example, assume someone is working with 5min bars and they get a BC message showing a closing price of 53.55 - that price could actually be from a minute ago (or more). If the bid/ask since that print has moved up to 53.95 / 53.96, a trading model may no longer consider a signal valid that was triggered by the 53.55 bar closing price. That is just one example and the implications of this can be many, but they can all be addressed rather easily by adding a simple bid/ask snapshot to streaming bar messages. Would you consider implementing this?

4. Have you measured the processing overhead in constructing bars before disseminating messages?

1. A time bar is closed when it receives a trade with a trade time outside of the bar time (for time bars only; volume and tick intervals are closed when they reach the appropriate volume or number of ticks).

2,3 I have forwarded those requests to management for consideration.

4. I have looked at the processing overhead in constructing interval bars and haven't seen any obvious problems. Are you experiencing an issue with this particular feature of the feed?

---

Nathan W. Bartsch - Senior Software Engineer - Telvent DTN

1. If I had my druthers, all bars would come out synchronized at the end of each interval. That makes for a consistent trading clock of sorts and, more importantly, the synchronization opens up the possibility of multi-asset strategies (e.g. pairs trading...although this would also require the bid/ask addition discussed in #3 above). That said, if I understand the docs correctly, one could still gain these advantages by using the 'Update Interval' parameter. I've tried testing this with the following request that (I think) should send a BU message 1 second prior to the bar's DateTime. However, you'll notice that BU messages aren't sent for each bar...shouldn't they be given the request I sent?

REQUEST:
sout.write("BW,SPY,300,,1500,,093000,160000,SPY_5min,s,,299\r\n");

MESSAGES:
SPY_5min,BH,SPY,2013-12-02 10:55:00,180.9780,180.9800,180.8700,180.9400,26427069,586758,0,
SPY_5min,BC,SPY,2013-12-02 10:55:00,180.9780,181.0050,180.8700,181.0000,26477986,637675,,
SPY_5min,BC,SPY,2013-12-02 11:00:00,180.9900,181.0501,180.9700,181.0300,27138971,652885,,
SPY_5min,BC,SPY,2013-12-02 11:05:00,181.0400,181.0900,181.0300,181.0610,27655585,513314,,
SPY_5min,BU,SPY,2013-12-02 11:10:00,181.0700,181.1200,181.0500,181.1000,28131279,461294,,
SPY_5min,BC,SPY,2013-12-02 11:10:00,181.0700,181.1200,181.0500,181.0900,28131379,461394,,
SPY_5min,BU,SPY,2013-12-02 11:15:00,181.0900,181.1200,181.0600,181.1100,28549298,409919,,
SPY_5min,BC,SPY,2013-12-02 11:15:00,181.0900,181.1200,181.0600,181.1100,28549298,409919,,
SPY_5min,BU,SPY,2013-12-02 11:20:00,181.1100,181.1600,181.0050,181.0400,29227467,651458,,
SPY_5min,BC,SPY,2013-12-02 11:20:00,181.1100,181.1600,181.0050,181.0400,29227467,651458,,
SPY_5min,BU,SPY,2013-12-02 11:25:00,181.0500,181.1300,181.0300,181.1000,29470302,233735,,
SPY_5min,BC,SPY,2013-12-02 11:25:00,181.0500,181.1300,181.0300,181.1000,29470302,233735,,
SPY_5min,BC,SPY,2013-12-02 11:30:00,181.1000,181.1200,181.0100,181.0100,30288069,814167,,
SPY_5min,BU,SPY,2013-12-02 11:35:00,181.0200,181.1300,180.9999,180.9999,31797173,1007987,,
SPY_5min,BC,SPY,2013-12-02 11:35:00,181.0200,181.1300,180.9999,180.9999,31797173,1007987

4. I didn't mean to insinuate that I've found a problem but I would like an answer to the question if possible.

The updates are sent based on the number of the timestamps received from the server after the first 'new' update to that bar arrived. So for a 'BU' message to arrive you would need to get a trade that updated the next bar and receive 299 timestamps before a trade for the next interval arrives.

If a trade for the next interval arrives before the 299th timestamp, the previous interval is completed and a 'BC' message is sent without sending a 'BU' message. The counter then starts back up tracking timestamps since the first 'new' update to the next bar arrived.

Sorry if that's a little confusing; the short answer is 'timing is everything'.

---

Nathan W. Bartsch - Senior Software Engineer - Telvent DTN

So if I understand you correctly, the Update Interval is the number of occurrences a timestamp is received that crosses the interval after the initial time that the interval is crossed within the bar period. Respectfully, that is so convoluted and unpredictable that it's hard to imagine how it could possibly be useful to anyone. As I said earlier, I think it's great that you guys have added this functionality but the Update Interval mechanics feel like something contrived by a programmer with the ease of database maintenance in mind rather than usefulness to a trader/end-user.

Anyhow, the three suggestions of BI marker, bid/ask snapshot, and clock synchronization are what I would need in order to make use of these (BI marker not a deal breaker). As someone who subscribes to the 1000 most active equity symbols each day, using these would easily decrease my bandwidth consumption by over 95%. Seems like that would be a win win for both of us, so I hope my comments are taken as constructive suggestions and not simply criticism (which is not at all my intention).

Nathan-

I realize that DTN can't cater to the whims of a single user on design issues, so I'll understand if not all (or any) of the suggestions above are implemented. However, would you mind doing me the favor of letting me know what you decide? Also, in the meantime, an approximate time frame for that decision would be appreciated as well. I ask this just so that I can manage my own workflow - I have some things on hold that will be impacted by my ability/inability to use the streaming bars.

Thanks.

There are a lot of issues to address here so if I miss something feel free to ask for clarification.

BI marker
-------------
You are correct that in most circumstances the last BH bar received will be incomplete. And unfortunately, we have the same restrictions as your app would in determining the close of a time based bar. We have no way of knowing if a bar is complete until a trade occurs with a timestamp that would be in the next bar. Even if the trade occurs with a timestamp that is at the end of a bar, there isn't anyway for us to know that another trade won't occur with the exact same timestamp. This means that for time based bars, the only way we could implement a BI marker would be to simply assume that the last bar retrieved from history is BI and mark it as such until the next trade occurs.


Adding the bid/ask
---------------------
Lets start with a clarification. Are you asking for the bid/ask at the time that the last trade was recorded for the bar or are you asking for the most current bid/ask when the bar is sent out?

Either way, if we were to implement this, the BU/BC updates would deviate from the BH updates (since we don't store bid/ask data in minute history and we only store bid/ask at the time of trade in tick history). As a result, this would probably be implemented as another message type or even a different request all together since (as you pointed out), its likely that another customer would want the data delivered in a different way.


Timeframe of changes
-------------------------
Any changes made would be included in a future version of IQFeed (5.2 or beyond). We haven't set release dates for these yet so the best answer we can give at this time would be "not short term".

Posible workarounds/solutions
----------------------------------
For the bid/ask data, my best recommendation would be to connect to the level 1 port and issue a watch with the fieldset containing just bid/ask fields and merging this data with your streaming bars data.

For the timing of updates in streaming bars, the easiest answer I can give would be to simply shorten your update interval to 1s. This means that at most you will be getting 2 times the number of symbols you are watching updates in any given second (if there was an update and a close for each symbol in a single second). It also means that any data you have received is at most 1s old.

BI Marker
------------
Yes, my interest in the BI marker was just to know that it was the last history bar being sent (whether it was actually incomplete or not). I think it would serve that purpose without causing issues as long as a BC message will always be sent for that BI bar once a next bar timestamp arrives (regardless of whether the BI bar was actually 'complete' or not).

Bid/Ask
---------
Ah, I see how that could cause issues. To clarify, I was asking for the most recent bid/ask when the message was sent (this was to address the possibility of the bar close price being very stale by the time the message arrived). Perhaps, adding bid/ask streaming bars could work. (Incidentally, I think bid/ask bars would be a great feature to offer in your historical database. It allows for much more granular backtesting (e.g. was the low ask price over the interval <= my buy limit price) with only a fraction of the storage requirements of a full tick database. Just a thought, sorry for the digression.)