Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1253D1058F for ; Wed, 4 Sep 2013 17:34:20 +0000 (UTC) Received: (qmail 59837 invoked by uid 500); 4 Sep 2013 17:34:19 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 59674 invoked by uid 500); 4 Sep 2013 17:34:19 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 59616 invoked by uid 99); 4 Sep 2013 17:34:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Sep 2013 17:34:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of david.medinets@gmail.com designates 74.125.82.176 as permitted sender) Received: from [74.125.82.176] (HELO mail-we0-f176.google.com) (74.125.82.176) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Sep 2013 17:34:04 +0000 Received: by mail-we0-f176.google.com with SMTP id u56so684160wes.7 for ; Wed, 04 Sep 2013 10:33:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=CgHxwitR5EeNYS7X7ET9AS/4GpvWgLaPU9tVKpuL3os=; b=rcfY10YZOXtw6YVcquhqwrY5qe3uqrO3raN7rMKDzgExikkpZNdSvkGNGwsLHjZY+U /g4LOqFjRKQfX43Cvgztoh8ZbpQ2dQRF6PEmyCaaSDLcaspA7s5XddZlHmI1vUE7XrjI PAc308yazmf/J2xVlXeGohtlAapZq/e0OXOBZ9n9uJAYzQneNjrPfQfI1Oo8nKcsiH+n vQnx4ls0duFszyIxzT946VIF+LO17o6cdCiGOrF1MfURCHewgoduDEBZgDcoQjk4bnlC yMGwF/jbxb/zuiNv2pd90U48dpzWf6/IGzaCz+nojyTlI9VBatJ573zBbpdrvY4MfTLO lwfA== MIME-Version: 1.0 X-Received: by 10.194.89.38 with SMTP id bl6mr2283263wjb.50.1378316024659; Wed, 04 Sep 2013 10:33:44 -0700 (PDT) Received: by 10.194.83.137 with HTTP; Wed, 4 Sep 2013 10:33:44 -0700 (PDT) In-Reply-To: References: Date: Wed, 4 Sep 2013 13:33:44 -0400 Message-ID: Subject: Re: AgeOffFilter + moving data to a different table From: David Medinets To: accumulo-user , vines@apache.org Content-Type: multipart/alternative; boundary=089e010d8a8009c98004e592357d X-Virus-Checked: Checked by ClamAV on apache.org --089e010d8a8009c98004e592357d Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Can you change the storage process so that each new set of data goes to a different table? Then you could sweep information to a aggregation table without any conflicts. On Wed, Sep 4, 2013 at 1:21 PM, John Vines wrote: > No, the iterator structures, which filters are built on, do not provide a > mechanism for writing to another table. While you could put a writer in > there, it is HIGHLY not recommended because it can potentially cause > deadlocking. You will need an external process to manage writing of old > records to a new table. > > > On Wed, Sep 4, 2013 at 1:11 PM, Slater, David M. wrote: > >> Hey,**** >> >> ** ** >> >> I was wondering if it was possible (and a good idea) to modify an >> AgeOffFilter so that when a record is aged off, it gets sent to a differ= ent >> table. **** >> >> ** ** >> >> Basically, I am storing netflow data and aggregating that information >> (through a Combiner) into a single entry. Updates to the aggregated reco= rd >> come regularly when the flow is active, but when the no updates have com= e >> in a certain amount of time (e.g. 30 seconds), I assume that the flow is >> done and want to migrate that to a different table (with further indexin= g). >> It is important that I migrate it from the current table so that the >> combiner doesn=92t combine two different flows (separated sufficiently i= n >> time) with the same Key (ips, ports, and protocol). **** >> >> ** ** >> >> Is there a way to do this with AgeOffFilters? Would I need an external >> program running that would periodically scan the table with an ageofffil= ter >> and then re-ingest those netflows into a new table? **** >> >> ** ** >> >> Regards, >> David**** >> > > --089e010d8a8009c98004e592357d Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Can you change the storage process so that each new set of= data goes to a different table? Then you could sweep information to a aggr= egation table without any conflicts.

On Wed, Sep 4, 2013 at 1:21 PM, John Vines <= span dir=3D"ltr"><= vines@apache.org> wrote:
No, the iterator structures, which filters are built on, d= o not provide a mechanism for writing to another table. While you could put= a writer in there, it is HIGHLY not recommended because it can potentially= cause deadlocking. You will need an external process to manage writing of = old records to a new table.


On Wed, Sep 4= , 2013 at 1:11 PM, Slater, David M. <David.Slater@jhuapl.edu>= wrote:

Hey,

=A0

I was wondering if it was possible (and a good idea)= to modify an AgeOffFilter so that when a record is aged off, it gets sent = to a different table.

=A0

Basically, I am storing netflow data and aggregating= that information (through a Combiner) into a single entry. Updates to the = aggregated record come regularly when the flow is active, but when the no u= pdates have come in a certain amount of time (e.g. 30 seconds), I assume th= at the flow is done and want to migrate that to a different table (with fur= ther indexing). It is important that I migrate it from the current table so= that the combiner doesn=92t combine two different flows (separated suffici= ently in time) with the same Key (ips, ports, and protocol). =

=A0

Is there= a way to do this with AgeOffFilters? Would I need an external program runn= ing that would periodically scan the table with an ageofffilter and then re= -ingest those netflows into a new table?

=A0

Regards,=
David



--089e010d8a8009c98004e592357d--