ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JohnnyS <jtk...@gmail.com>
Subject Re: Local Node Query Optimization
Date Mon, 17 Aug 2015 19:06:30 GMT

The C++ implementation is almost completely custom. I used zip_iterators
from Cuda Thrust to create a "virtual container" in the host memory from a
bunch of STL vectors. I then sort this by key and use upper and lower to
find the limits for the records in a tile. I then run a simple scan over
these to count the number of distinct users. I'm fairly sure that this could
be further optimized. The reading of the STL vectors sequentially should be
as fast as possible. If I insert more data, the container needs to be sorted
again which is not optimal.

I could use global queries to count the number of distinct users. I do also
a map-reduce like reduction to aggregate the gps tracks in a tile. To do
this, I follow
and do a query broadcast, reduce locally and this reduced result is returned
to the caller. If it is possible to create complex custom aggregate SQL
functions I could probably use these.

I tried profiling using virtualvm and the bottleneck might be the
marshalling, but the results are somewhat conflicting and I'm not sure of
this. Hazelcast has binary and object formats for the values
http://docs.hazelcast.org/docs/latest/manual/html/map-inmemoryformat.html. I
could try this if something similar is available in Ignite.


View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Local-Node-Query-Optimization-tp996p1007.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message