accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Fuchs <afu...@apache.org>
Subject Re: Watching for Changes with Write Ahead Log?
Date Thu, 01 Oct 2015 20:02:03 GMT
I would stay away from ThreadLocal -- the threads that run Constraints can
be dynamically generated in a resizable thread pool, and cleaning up after
them could be challenging. Static might work better if you can make it
thread safe, maybe with a resource pool.

Adam


On Thu, Oct 1, 2015 at 2:39 PM, John Vines <vines@apache.org> wrote:

> As dirty as it is, that sounds like a case for a static, or maybe thread
> local, object
>
> On Thu, Oct 1, 2015, 7:19 PM Parise, Jonathan <Jonathan.Parise@gd-ms.com>
> wrote:
>
>> I have a few follow up questions in regard to constraints.
>>
>>
>>
>> What is the lifecycle of a constraint? What I mean by this is are the
>> constraints somehow tied to Accumulo’s lifecycle or are they just
>> instantiated each time a mutation occurs and then disposed?
>>
>>
>>
>> Also, are there multiple instances of the same constraint class at any
>> time or do all mutation on a table go through the exact same constraint?
>>
>>
>>
>> My guess is that  when a mutation comes in a new constraint is made
>> through reflection. Then check() is called, the violation codes are parsed
>> and the object is disposed/finalized.
>>
>>
>>
>> The reason I ask is that what I want to do is update my ElasticSearch
>> index each time I see a mutation on the table. However, I don’t want to
>> have to make a connection, send the data and then tear down the connection
>> each time. That’s a lot of unnecessary overhead and with all that overhead
>> happening on every mutation performance could be badly impacted.
>>
>>
>>
>> Is there some way to cache something like a connection and reuse it
>> between calls to the Constraint’s check() method? How would such a thing be
>> cleaned up if Accumulo is shut down?
>>
>>
>>
>>
>>
>> Thanks again,
>>
>>
>>
>> Jon
>>
>> *From:* Parise, Jonathan [mailto:Jonathan.Parise@gd-ms.com
>> <Jonathan.Parise@gd-ms.com>]
>> *Sent:* Wednesday, September 30, 2015 9:21 AM
>> *To:* user@accumulo.apache.org
>> *Subject:* RE: Watching for Changes with Write Ahead Log?
>>
>>
>>
>> In this particular case, I need to update some of my application state
>> when changes made by another system occur.
>>
>>
>>
>> I would need to do a few things to accomplish my goal.
>>
>>
>>
>> 1)      Be notified or see that a table had changed
>>
>> 2)      Checked that against changes I know my system has made
>>
>> 3)      If my system is not the originator of the change, update
>> internal state to reflect the change.
>>
>>
>>
>> Examples of state I may need to update include an ElasticSearch index and
>> also an in memory cache.
>>
>>
>>
>> I’m going to read up on constraints again and see if I can use them for
>> this purpose.
>>
>>
>>
>> Thanks!
>>
>>
>>
>> Jon
>>
>>
>>
>>
>>
>>
>>
>> *From:* Adam Fuchs [mailto:afuchs@apache.org <afuchs@apache.org>]
>> *Sent:* Tuesday, September 29, 2015 5:46 PM
>> *To:* user@accumulo.apache.org
>> *Subject:* Re: Watching for Changes with Write Ahead Log?
>>
>>
>>
>> Jon,
>>
>>
>>
>> You might think about putting a constraint on your table. I think the API
>> for constraints is flexible enough for your purpose, but I'm not exactly
>> sure how you would want to manage the results / side effects of your
>> observations.
>>
>>
>>
>> Adam
>>
>>
>>
>>
>>
>> On Tue, Sep 29, 2015 at 5:41 PM, Parise, Jonathan <
>> Jonathan.Parise@gd-ms.com> wrote:
>>
>> Hi,
>>
>>
>>
>> I’m working on a system where generally changes to Accumulo will come
>> through that system. However, in some cases, another system may change data
>> without my system being aware of it.
>>
>>
>>
>> What I would like to do is somehow listen for changes to the tables my
>> system cares about. I know there is a write ahead log that I could
>> potentially listen to for changes, but I don’t know how to use it. I looked
>> around for some documentation about it, and I don’t see much. I get the
>> impression that it isn’t really intended for this type of use case.
>>
>>
>>
>> Does anyone have any suggestions on how to watch a table for changes and
>> then determine if those changes were made by a different system.
>>
>>
>>
>> Is there some documentation about how to use the write ahead log?
>>
>>
>>
>>
>>
>> Thanks,
>>
>>
>>
>> Jon Parise
>>
>>
>>
>

Mime
View raw message