accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Vines <vi...@apache.org>
Subject Re: AgeOffFilter + moving data to a different table
Date Wed, 04 Sep 2013 17:21:57 GMT
No, the iterator structures, which filters are built on, do not provide a
mechanism for writing to another table. While you could put a writer in
there, it is HIGHLY not recommended because it can potentially cause
deadlocking. You will need an external process to manage writing of old
records to a new table.


On Wed, Sep 4, 2013 at 1:11 PM, Slater, David M. <David.Slater@jhuapl.edu>wrote:

> Hey,****
>
> ** **
>
> I was wondering if it was possible (and a good idea) to modify an
> AgeOffFilter so that when a record is aged off, it gets sent to a different
> table. ****
>
> ** **
>
> Basically, I am storing netflow data and aggregating that information
> (through a Combiner) into a single entry. Updates to the aggregated record
> come regularly when the flow is active, but when the no updates have come
> in a certain amount of time (e.g. 30 seconds), I assume that the flow is
> done and want to migrate that to a different table (with further indexing).
> It is important that I migrate it from the current table so that the
> combiner doesn’t combine two different flows (separated sufficiently in
> time) with the same Key (ips, ports, and protocol). ****
>
> ** **
>
> Is there a way to do this with AgeOffFilters? Would I need an external
> program running that would periodically scan the table with an ageofffilter
> and then re-ingest those netflows into a new table? ****
>
> ** **
>
> Regards,
> David****
>

Mime
View raw message