Joined: |
Apr 2, 2013 08:04 PM |
Last Post: |
Sep 20, 2013 10:09 AM |
Last Visit: |
Sep 20, 2013 10:09 AM |
Website: |
|
Location: |
Virginia |
Occupation: |
|
Interests: |
Optimization |
|
AIM: |
|
ICQ: |
|
MSN IM: |
|
Yahoo IM: |
|
|
gamozo has contributed to 4 posts out of 21251 total posts
(0.02%) in 4,267 days (0.00 posts per day).
20 Most recent posts:
Just tick-level on pretty much all the top 20 most traded futures. I think @ES# has about 55 million datapoints at tick level.
I pull at ticklevel as I postprocess all the data and frequently use tick data for simulated fills rather than n-length bars.
It's not a huge issue, I usually just pull once and then have the files around forever. I'm just requesting the feature if it's easy to add as it would be nice here and there.
-Brandon
Yep. That's what I currently do. However that only works with multiple symbols. It would be nice to have one symbol at full speed. And the only way I could think of doing this is if I knew how many ticks there were historically, with that I could start multiple queries at different starting points. I guess I could do it in chunks of 100k trades or so and just guess for now, but it'd be nice to know how many trades to expect.
Is there any chance we could get multi-threaded decompression in IQConnect? I'm bottlenecking on CPU rather than download bandwidth.
Currently I just pull all my symbols at once, which means I bottleneck on bandwidth which makes me happy. However I would like to pull a single symbol at full network bandwidth.
Another thing that could allow me to implement this on my end is if you add a 'get number of ticks', so I could query how many ticks of history there is for a symbol. Once I get that number I could start at trades/threads offsets and do a multi-threaded pull.
Any chance this could happen soon?
-Brandon Edited by gamozo on Sep 19, 2013 at 05:22 PM
I really do not like working with text-based protocols, as they're clumsy, expensive to parse, and take longer to write parsers for.
I'm wondering if there could be a mode added where data will be sent in binary fixed-width structures rather than CSVs. It pains my love for optimization to know that you guys take in binary data, convert it to CSV, and I convert it right back the second I read it.
I know it's probably something that most people do not ask for, but I can't imagine it would take too long for you guys to implement it.
-Brandon Edited by gamozo on Apr 2, 2013 at 08:20 PM
|
|