ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Welly Tambunan <if05...@gmail.com>
Subject Re: Storing Time Series data efficiently on Ignite
Date Thu, 03 Dec 2015 12:33:44 GMT
Hi Denis,

Thanks for your clear explanation.

Our data structure is something like this one <sensorid: UUID, time: Long,
value: Double>

When i put a data point with that composite key. <sensorid, time>, is there
any guarantee that it will be store close together in a same node ?

In case of update, however we need to be able to update in range, like
replace the range <startId, endIdx, List[DataPoint]>, (where DataPoint =>
<time, value> )
so it will clear that range and insert the new data point into that range.

So we need to do two step process to do that ? Select the key and then
delete the key ?


Thanks





On Thu, Dec 3, 2015 at 7:00 PM, Denis Magda <dmagda@gridgain.com> wrote:

> Hi Welly,
>
> Ignite perfectly fits for your task.
>
> First, if I understand you properly there are will be many time series for
> a give sensor ID.
> If so then I would use a compound key like (sensorId, time) for all cache
> related operations.
>
> As an example you may want to use classes like this.
>
> SampleKey {
> int sernsorId;
> long sampleTime;
> }
>
> Sample {
> int sensorId;
> long sampleTime;
> byte[] data1;
> byte[] data2;
> etc.
> }
>
> And use them this way
>
> cache.put(new SampleKey(1, time), sample);
> cache.get(new SampleKey(2, time));
>
> Second, to retrieve samples depending on sensor ID, time (time range) or
> other parameters you can leverage Ignite SQL engine that is designed
> exactly for the use cases you have. [1]
> However, if you're going to use an object field in 'SELECT' or 'WHERE'
> clause you have to annotate it properly or specify using CacheTypeMetadata
> [2]
>
> Third, when you need to update data series you can remove the old one and
> insert the new one that should have new time.
> To perform a remove I would suggest doing the following:
> - select sensor ID and sampleTime of all the entries to delete;
> - use cache.removeAll by passing SampleKeys that are created using the
> data retrieved with SQL above.
>
> Moreover, you can use an eviction or expire policy that is used in cases
> when old data must be removed from cache automatically.
> Just refer to these articles for more info  - [3]
>
> Finally, Ignite has bunch of cache and SQL related examples. They are
> located in "datagrid" folder of "examples" module.
> Have a look at them and probably you'll come up with better solution based
> on Ignite that suggested by me above cause definitely you know all the
> details of your case better ;)
>
> [1] https://apacheignite.readme.io/docs/sql-queries
> [2]
> https://apacheignite.readme.io/docs/sql-queries#configuring-sql-indexes-by-annotations
> [3] https://apacheignite.readme.io/docs/evictions
> [4] https://apacheignite.readme.io/docs/expiry-policies
>
>
> Regards,
> Denis
>
>
> On 12/3/2015 12:50 PM, Welly Tambunan wrote:
>
> Hi Igniters,
>
> Currently we are trying to asses the possibility of using Ignite on our
> Architecture.
>
> We have a case where we want to store time series data in memory.  We will
> have a lots of sensor data. So i think i will use sensor id as a key to
> retrieve the time series from cache.
>
> However i can't find any sorted list structure in ignite to store our time
> series. The index can be long ( for time ). So it will need to be sorted by
> index.
>
>
> We also have a query for getting range of index, ex: Give me all series
> from start idx to end idx.
> For updating we also need to be able to update a range with new data
> series.
>
> We just don't want to re uploaded the data again over and over again to
> cache everytime there's an update on the range. We want to be able to
> update the cache partially based on the range.
>
> Is there any way we can achieve this on Ignite ?
>
> Any suggestion or reference would be really appreciated.
>
> Cheers
>
> --
> Welly Tambunan
> Triplelands
>
> http://weltam.wordpress.com
> http://www.triplelands.com <http://www.triplelands.com/blog/>
>
>
>


-- 
Welly Tambunan
Triplelands

http://weltam.wordpress.com
http://www.triplelands.com <http://www.triplelands.com/blog/>

Mime
View raw message