We made a BW request on 02:52:00 for a 3-minute interval and update interval of 30 seconds. We received BH messages (based on historical data) uptill 02:52:00 and all prior history bars were 3-minutes apart. The problem is that after 02:52:00 marks, the next streaming bar comes at 02:54:00 which is two minutes apart from the last bar. After the bar at 02:54:00, all the streaming bars are once again 3-minutes apart. The stream is given below

B-@ES#-00180-s,BH,@ES#,2023-03-29 02:40:00,4026.75,4026.75,4026.25,4026.75,73084,391,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:43:00,4026.50,4027.50,4026.50,4027.00,73447,363,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:46:00,4027.00,4027.25,4026.25,4026.75,73754,307,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 02:49:00,4027.00,4027.25,4026.75,4026.75,74208,454,0,

B-@ES#-00180-s,BH,@ES#,2023-03-29 02:52:00,4026.75,4027.25,4026.50,4027.00,74509,301,0,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:54:00,4026.75,4027.25,4026.50,4027.25,74546,338,,

B-@ES#-00180-s,BU,@ES#,2023-03-29 02:54:00,4026.75,4027.50,4026.50,4027.25,74621,413,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 02:54:00,4026.75,4027.50,4026.50,4027.50,74651,443,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4027.25,4027.25,74747,96,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4026.75,74966,315,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4027.00,75039,388,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4027.75,4026.50,4027.75,75217,566,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 02:57:00,4027.75,4028.00,4026.50,4028.00,75381,730,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 02:57:00,4027.75,4028.25,4026.50,4028.25,75452,801,,


My question is if there is a way we can make sure that all history, update and closed bars are 3 minutes apart?

In another occurrence, I receive the following stream:


B-@ES#-00180-s,BH,@ES#,2023-03-29 04:26:00,4032.75,4034.50,4032.75,4034.50,113520,792,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:29:00,4034.25,4035.00,4034.25,4034.50,114138,618,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:32:00,4034.50,4036.75,4034.25,4036.75,115551,1413,0,
B-@ES#-00180-s,BH,@ES#,2023-03-29 04:35:00,4036.75,4036.75,4035.50,4035.75,115989,438,0,

B-@ES#-00180-s,BC,@ES#,2023-03-29 04:36:00,4036.75,4036.75,4035.50,4035.75,115989,438,,

B-@ES#-00180-s,BU,@ES#,2023-03-29 04:33:00,4036.00,4036.75,4036.00,4036.50,116292,303,,
B-@ES#-00180-s,BC,@ES#,2023-03-29 04:33:00,4036.00,4037.25,4036.00,4037.25,116458,469,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4037.00,4037.25,116978,520,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.75,4036.75,117217,759,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4036.75,117426,968,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4036.50,117630,1172,,
B-@ES#-00180-s,BU,@ES#,2023-03-29 04:36:00,4037.25,4037.75,4036.50,4037.00,117808,1350,,


The request was made on 04:32:00 for a 3-minute interval and update interval of 30 seconds. We received BH messages (based on historical data) uptill 04:35:00 and all prior history bars were 3-minutes apart. There are multiple problems with the stream:

• How can IQFEED calculate the 04:35:00 history bar at 04:32:00 mark. I understand if it starts sending BU messages for the 04:35:00 bar, but a history bar means that the bar has been closed based on historical data. So, how the 04:35:00 history bar can be closed at 04:32:00 mark.


• We received a BC close bar message for 04:36:00 (highlighted green) and later received BU (update) messages for the same bar.


• The previously discussed problem of misaligned interval bars like 04:32, 04:33, 04:35 and 04:36 still remains. The difference between these bars should be 3 minutes.

Can anyone please clarify how can I make sure that all historical and current bars are received in order as well as the same time interval apart?

That looks like an error. I will escalate this to the dev team. BU messages can arrive at any time (if you're using UpdateInterval), but IQFeed should clearly tell you which bar the belong to. If you set a 180-second bar, there should only be messages for every 3 minutes, not 2 minutes apart like that.

Your other two questions are interwoven with that problem, so I can't really answer them until we solve that one. There should never be BU updates after a BC message is sent, because because the BC message can only be sent after the interval is complete. Which in turn only happens when a tick *after* the interval has happened. @ES# trades so frequently this never takes long, but the behind-the-scenes working of BW may be having a timing issue.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Also, what is the exact BW command you're sending? I can infer some of it from your post, but our investigation would be helped if you can post it in its entirety. Thank you.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Thanks Gary for the answer.

I am using the following command (requested at 12:39:xx, where I don't know the seconds for the requested time):

BW,@ES#,180,20230404 093900,,,,,B-@ES#-00180-s,s,'',30

Once again, I got the following output:

B-@ES#-00180-s,BH,@ES#,2023-04-04 12:24:00,4116.25,4120.75,4116.25,4120.75,943647,11233,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:27:00,4120.75,4125.25,4120.25,4124.75,955352,11679,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:30:00,4125.00,4126.00,4124.25,4126.00,961787,6435,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:33:00,4126.00,4128.00,4125.50,4126.25,969806,8019,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:36:00,4126.25,4126.75,4123.75,4124.00,977011,7005,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:39:00,4124.00,4125.00,4121.50,4123.00,986272,8861,0,
B-@ES#-00180-s,BH,@ES#,2023-04-04 12:42:00,4123.00,4124.00,4122.75,4123.50,987705,1433,0,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4125.00,4122.75,4125.00,989230,2958,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4126.50,992407,6135,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4125.00,994452,8180,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4125.75,996636,10364,,
B-@ES#-00180-s,BC,@ES#,2023-04-04 12:42:00,4123.00,4127.50,4122.75,4126.75,997380,11108,,
B-@ES#-00180-s,BU,@ES#,2023-04-04 12:45:00,4126.50,4126.75,4125.75,4126.00,998656,1276,,

As you can see, I received a history bar for 2023-04-04 12:42:00 on 2023-04-04 12-39-xx mark and later received BU messages corresponding to the same bar.

The command for the above two examples in my first post are similar with the exception of the request time for the command and BeginDateTime parameter.

It is possible for a BH command to be followed by BU commands for the same bar. This is because a BH message doesn't have to be the final message for that bar. This bar behaves in a way analogous to the IncludePartialDatapoint parameter in some historical commands like HDX: it will send a message for an interval even if it's not complete yet.

This is really only likely to happen when the BW command is building the first BC message of the session. Your BW command included some history, which it provided in the form of BH messages up to the current one. Then you received 4 BU update messages for the same bar, because you set the UpdateInterval parameter of the BW command to send an update every 30 seconds. Then you get the BC message for that bar, which is the final message, because BC is only sent when it the bar is complete (with one caveat, see next paragraph). Then you began getting BU messages for the next bar.

Even BC message bars aren't 100% final, because they can still be changed by a subsequent BU message for the same bar. This happens when the exchange issues as a Trade Correction, which changes a tick in a way that the bar must now be recalculated.

Your other example was a BC message followed by BU messages from earlier, which is a different matter still being investigated.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

Thanks Gary for your insight.

Just to give you some more information about our requirement: we are trying to write data to a csv file which is supposed to be current all the time. Based on your above explanation, I assume that we should take the last bar of the first BW response as 'not closed' and if we receive any subsequent bar with the same timestamp, we should update the last bar. It seems all fine and we can easily implement this logic. However, Example 2 in the original post has some more issues that we cannot resolve with the above logic. The biggest issue is that the bars are not 3 minutes apart uniformly -- some are 2 minutes apart. Furthermore, the update message are not received for the last bar only but comes for a few bars further back as well. Please let us know how the investigation goes.

Let me make a counter-suggestion: What if you set the bar length to 30 seconds instead of 3 minutes, set UpdateInterval to 0 instead of 30 seconds, then assemble your 3-minute bars from those 30-second bars.

BW,@ES#,30,20230404 093900,,,,,B-@ES#-00180-s,s,',0

This gives you the immediacy of the every-30-seconds update, while also knowing that a BH message represents a complete 30-second bar is complete. The tradeoff is that the BH message will not be sent until a tick has occurred after that 30-second bar. So if something doesn't trade for a few seconds, you might not get the 9:17:30 bar until 9:17:38. If you can live with that, then this approach might work better for you, depending on how your program otherwise works.

I know I still owe you an answer on the 2-minute / 3-minute business, which I will post soon.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist
Edited by DTN_Gary_Stephen on Apr 11, 2023 at 03:54 PM

I was able to reproduce the issue with the misaligned bar start times. Will post a solution when we have one.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

I can provide a workaround: if you're using a Start Date/Start Time, make sure it's evenly divisible by the bar length you specified. You didn't give the command for your initial post, but your follow-up was:

BW,@ES#,180,20230404 093900,,,,,B-@ES#-00180-s,s,'',30

This works, because if you start at :39 after the hour, the remainder (21) divides evenly into 3-minute bars. I suspect your command from the OP started at something like 094000, where it doesn't.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist

This issue will be resolved as part of a larger update later on. in the meantime, use the workaround I listed above: if you use a start time, make sure the remainder of the hour is evenly divisible by the bar length you specified.

---

Sincerely,
Gary Stephen
DTN IQFeed Implementation Support Specialist