Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 958F3187DE for ; Tue, 8 Dec 2015 21:51:23 +0000 (UTC) Received: (qmail 32491 invoked by uid 500); 8 Dec 2015 21:51:10 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 32438 invoked by uid 500); 8 Dec 2015 21:51:10 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 32428 invoked by uid 99); 8 Dec 2015 21:51:10 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2015 21:51:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4D2FD180999 for ; Tue, 8 Dec 2015 21:51:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=deenlo-com.20150623.gappssmtp.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id veluuNhKXC3y for ; Tue, 8 Dec 2015 21:51:03 +0000 (UTC) Received: from mail-vk0-f48.google.com (mail-vk0-f48.google.com [209.85.213.48]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 6D77842B96 for ; Tue, 8 Dec 2015 21:51:03 +0000 (UTC) Received: by vkbs1 with SMTP id s1so30739235vkb.1 for ; Tue, 08 Dec 2015 13:50:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=deenlo-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hLOJYRmdMVkG4VbN/aaEITItr0zR7wFmv72d2TeEcvk=; b=myZqM5xDLFbA55SkuZ6ydhLyKL69NLULRu2sRhLIJRqiSJLhr++RoQoWxOqTyC8dbK vac3rjAzrjskoBDqHFzpYbWKo4kM00QfbNRG+1EVAPlvfH8aBBziRG1oKKLXtdvHnVZq NElJ5tj0qPHCisb9Try8bPN++s3uVKusD//7ESs9Sphj66rBPXzsWA+/UGR919BuNMKb 1jN8gVVLYE/kB6BGmeCYX11sm3GQm9CUvybYtiPDBFucPhb7WmzJBabCtKnNLrkDRbK8 su2w79PS8qZoA3+OdAcLb4Di7J2q38Tnz60VcHewE6Fmkc70HPoiPmCdBkIkimiWgJSg GpKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=hLOJYRmdMVkG4VbN/aaEITItr0zR7wFmv72d2TeEcvk=; b=WzlMqA3ePf/du6HrRaAYp5/zf/EMfXcQ0g2Ss5Uo5iqTtItJdArJjZpdHJos+lvWXQ dyLnBuXRM2/mAM1KqyJ4Ve6RWvv4NCHdTcc87AnX/b08XtMCCfWw7JijB3UWapdE5kd4 QgUgdyDRa5cteW7Hdrkli1BdKIRJpEZIHYlNZhX23VgnbeqCk1QQQWzaQvgydhbv/NdE m73j6CnyziNao3EZ029YOfHlL+meHPtnTARsEJscCSzPSV8pr3v/1VcVkpwS9mBkR6jn cIbz0/u8inrIeaJb0nHnNzyykUTwYLNsbljJEOpt7ugDURfh9hPIKo3F8NlYV2GyZXqP uNfg== X-Gm-Message-State: ALoCoQlnFHkzpnOIYTqZKiXek9I1LBUSlUWoF/wRKhKsMGd3khQKjnELxs1rKBK+4EVvmqzVucz+94XI4msMi4UJp3xAAsdX3Q== MIME-Version: 1.0 X-Received: by 10.129.80.138 with SMTP id e132mr3567288ywb.90.1449611417566; Tue, 08 Dec 2015 13:50:17 -0800 (PST) Received: by 10.37.78.133 with HTTP; Tue, 8 Dec 2015 13:50:17 -0800 (PST) In-Reply-To: References: <565FBC11.30802@gmail.com> <565FBCE0.8050904@gmail.com> Date: Tue, 8 Dec 2015 16:50:17 -0500 Message-ID: Subject: Re: Trigger for Accumulo table From: Keith Turner To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary=001a1147fa529b8a04052669f5cb --001a1147fa529b8a04052669f5cb Content-Type: text/plain; charset=UTF-8 Constraints are checked before data is written. In the case of failures a constraint may see data thats never successfully written. On Tue, Dec 8, 2015 at 4:18 PM, Christopher wrote: > Look at org.apache.accumulo.core.constraints.Constraint for a description > and org.apache.accumulo.core.constraints.DefaultKeySizeConstraint as an > example. > > In short, Mutations which are live-ingested into a tablet server are > validated against constraints you specify on the table. That means that all > Mutations written to a table go through this bit of user-provided code at > least once. You could use that fact to your advantage. However, this would > be highly experimental and might have some caveats to consider. > > You can configure a constraint on a table with > connector.tableOperations().addConstraint(...) > > > On Sun, Dec 6, 2015 at 10:49 PM Thai Ngo wrote: > >> Christopher, >> >> This is interesting! Could you please give me more details about this? >> >> Thanks, >> Thai >> >> On Thu, Dec 3, 2015 at 12:17 PM, Christopher wrote: >> >>> You could also implement a constraint to notify an external system when >>> a row is updated. >>> >>> On Wed, Dec 2, 2015, 22:54 Josh Elser wrote: >>> >>>> oops :) >>>> >>>> [1] http://fluo.io/ >>>> >>>> Josh Elser wrote: >>>> > Hi Thai, >>>> > >>>> > There is no out-of-the-box feature provided with Accumulo that does >>>> what >>>> > you're asking for. Accumulo doesn't provide any functionality to push >>>> > notifications to other systems. You could potentially maintain other >>>> > tables/columns in which you maintain the last time a row was updated, >>>> > but the onus is on your "other services" to read the table to find out >>>> > when a change occurred (which is probably not scalable at "real >>>> time"). >>>> > >>>> > There are other systems you could likely leverage to solve this, >>>> > depending on the durability and scalability that your application >>>> needs. >>>> > >>>> > For a system "close" to Accumulo, you could take a look at Fluo [1] >>>> > which is an implementation of Google's "Percolator" system. This is a >>>> > system based on throughput rather than low-latency, so it may not be a >>>> > good fit for your needs. There are probably other systems in the >>>> Apache >>>> > ecosystem (Kafka, Storm, Flink or Spark Streaming maybe?) that are be >>>> > helpful to your problem. I'm not an expert on these to recommend on >>>> (nor >>>> > do I think I understand your entire architecture well enough). >>>> > >>>> > Thai Ngo wrote: >>>> >> Hi list, >>>> >> >>>> >> I have a use-case when existing rows in a table will be updated by an >>>> >> internal service. Data in a row of this table is composed of 2 parts: >>>> >> 1st part - immutable and the 2nd one - will be updated (filled in) a >>>> >> little later. >>>> >> >>>> >> Currently, I have a need of knowing when and which rows will be >>>> updated >>>> >> in the table so that other services will be wisely start consuming >>>> the >>>> >> data. It will make more sense when I need to consume the data in near >>>> >> realtime. So developing a notification function or simpler - a >>>> trigger >>>> >> is what I really want to do now. >>>> >> >>>> >> I am curious to know if someone has done similar job or there are >>>> >> features or APIs or best practices available for Accumulo so far. I'm >>>> >> thinking of letting the internal service which updates the data >>>> notify >>>> >> us whenever it updates the data. >>>> >> >>>> >> What do you think? >>>> >> >>>> >> Thanks, >>>> >> Thai >>>> >>> >> --001a1147fa529b8a04052669f5cb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Constraints are checked before data is written.=C2=A0 In t= he case of failures a constraint may see data thats never successfully writ= ten.

On = Tue, Dec 8, 2015 at 4:18 PM, Christopher <ctubbsii@apache.org> wrote:
Lo= ok at org.apache.accumulo.core.constraints.Constraint for a description and= org.apache.accumulo.core.constraints.DefaultKeySizeConstraint as an exampl= e.

In short, Mutations which are live-ingested into a tablet s= erver are validated against constraints you specify on the table. That mean= s that all Mutations written to a table go through this bit of user-provide= d code at least once. You could use that fact to your advantage. However, t= his would be highly experimental and might have some caveats to consider.
You can configure a constraint on a table with connector.tableO= perations().addConstraint(...)


On Sun, Dec 6, 2015 at 10:49 P= M Thai Ngo <ba= othaingo@gmail.com> wrote:
<= div dir=3D"ltr">Christopher,

This is interesting! Could = you please give me more details about this?

Thanks= ,
Thai
On Thu, Dec 3, 2015 at 12:17 PM, Christopher <ctubbsii@apache.org> wrote:

You could also implement a constraint to notify an extern= al system when a row is updated.


On Wed, Dec 2, 2015, 22:54= =C2=A0Josh Elser <josh.elser@gmail.com> wrote:
oops :)

[1] http:/= /fluo.io/

Josh Elser wrote:
> Hi Thai,
>
> There is no out-of-the-box feature provided with Accumulo that does wh= at
> you're asking for. Accumulo doesn't provide any functionality = to push
> notifications to other systems. You could potentially maintain other > tables/columns in which you maintain the last time a row was updated,<= br> > but the onus is on your "other services" to read the table t= o find out
> when a change occurred (which is probably not scalable at "real t= ime").
>
> There are other systems you could likely leverage to solve this,
> depending on the durability and scalability that your application need= s.
>
> For a system "close" to Accumulo, you could take a look at F= luo [1]
> which is an implementation of Google's "Percolator" syst= em. This is a
> system based on throughput rather than low-latency, so it may not be a=
> good fit for your needs. There are probably other systems in the Apach= e
> ecosystem (Kafka, Storm, Flink or Spark Streaming maybe?) that are be<= br> > helpful to your problem. I'm not an expert on these to recommend o= n (nor
> do I think I understand your entire architecture well enough).
>
> Thai Ngo wrote:
>> Hi list,
>>
>> I have a use-case when existing rows in a table will be updated by= an
>> internal service. Data in a row of this table is composed of 2 par= ts:
>> 1st part - immutable and the 2nd one - will be updated (filled in)= a
>> little later.
>>
>> Currently, I have a need of knowing when and which rows will be up= dated
>> in the table so that other services will be wisely start consuming= the
>> data. It will make more sense when I need to consume the data in n= ear
>> realtime. So developing a notification function or simpler - a tri= gger
>> is what I really want to do now.
>>
>> I am curious to know if someone has done similar job or there are<= br> >> features or APIs or best practices available for Accumulo so far. = I'm
>> thinking of letting the internal service which updates the data no= tify
>> us whenever it updates the data.
>>
>> What do you think?
>>
>> Thanks,
>> Thai


--001a1147fa529b8a04052669f5cb--