accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Turner <ke...@deenlo.com>
Subject Re: Watching for Changes with Write Ahead Log?
Date Thu, 01 Oct 2015 19:16:30 GMT
You can follow constrainChecker atomic ref.  Its been awhile, but I think
its only reloaded when constraint config changes.  Also multiple threads
may use a constraint.

https://github.com/apache/accumulo/blob/1.6.2/server/tserver/src/main/java/org/apache/accumulo/tserver/Tablet.java#L2462

On Thu, Oct 1, 2015 at 3:03 PM, Parise, Jonathan <Jonathan.Parise@gd-ms.com>
wrote:

> Are the constraint threads reused? For example is there a thread pool or
> something the constraints use to run?
>
>
>
> I have the Accumulo source on my machine, but I am not sure where to look
> to figure this out. What class invokes the constraints?
>
>
>
> Thanks,
>
>
>
> Jon
>
>
>
> *From:* Keith Turner [mailto:keith@deenlo.com]
> *Sent:* Thursday, October 01, 2015 3:02 PM
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> Could possibly use a ThreadLocal containing a SoftReference
>
> Another place you could possibly put this code instead of in a constraint
> is in a minor compaction iterator.   Although you will still have the clean
> up problem since iterators do not have a close method.  There is an open
> ticket where Billie suggested that Accumulo can call close on any iterator
> that implements Closeable
>
>
>
> On Thu, Oct 1, 2015 at 2:48 PM, Parise, Jonathan <
> Jonathan.Parise@gd-ms.com> wrote:
>
> I think this is one of those things where there really aren’t great
> solutions.
>
>
>
> A static connection could work, but if multiple Constraint instances can
> exist at the same time, it would probably not work. Since all of them would
> be trying to use the same connection at the same time.
>
>
>
> ThreadLocal could possibly work better. The only question is how long
> lived is the thread that calls the constraints? For example that thread
> could be torn down as soon as the constraint is done. In that case the
> performance would be no better than creating and tearing down everything
> each time check() is called.
>
>
>
> This is why I am trying to understand the Constraint’s lifecycle, so I can
> come up with the least bad way of solving this problem.
>
>
>
> Thanks for the ideas! I am just not sure I know enough about the lifecycle
> of Constraints to understand if these suggestions would be helpful.
>
>
>
> Jon Parise
>
>
>
> *From:* John Vines [mailto:vines@apache.org]
> *Sent:* Thursday, October 01, 2015 2:40 PM
>
>
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> As dirty as it is, that sounds like a case for a static, or maybe thread
> local, object
>
>
>
> On Thu, Oct 1, 2015, 7:19 PM Parise, Jonathan <Jonathan.Parise@gd-ms.com>
> wrote:
>
> I have a few follow up questions in regard to constraints.
>
>
>
> What is the lifecycle of a constraint? What I mean by this is are the
> constraints somehow tied to Accumulo’s lifecycle or are they just
> instantiated each time a mutation occurs and then disposed?
>
>
>
> Also, are there multiple instances of the same constraint class at any
> time or do all mutation on a table go through the exact same constraint?
>
>
>
> My guess is that  when a mutation comes in a new constraint is made
> through reflection. Then check() is called, the violation codes are parsed
> and the object is disposed/finalized.
>
>
>
> The reason I ask is that what I want to do is update my ElasticSearch
> index each time I see a mutation on the table. However, I don’t want to
> have to make a connection, send the data and then tear down the connection
> each time. That’s a lot of unnecessary overhead and with all that overhead
> happening on every mutation performance could be badly impacted.
>
>
>
> Is there some way to cache something like a connection and reuse it
> between calls to the Constraint’s check() method? How would such a thing be
> cleaned up if Accumulo is shut down?
>
>
>
>
>
> Thanks again,
>
>
>
> Jon
>
> *From:* Parise, Jonathan [mailto:Jonathan.Parise@gd-ms.com
> <Jonathan.Parise@gd-ms.com>]
> *Sent:* Wednesday, September 30, 2015 9:21 AM
> *To:* user@accumulo.apache.org
> *Subject:* RE: Watching for Changes with Write Ahead Log?
>
>
>
> In this particular case, I need to update some of my application state
> when changes made by another system occur.
>
>
>
> I would need to do a few things to accomplish my goal.
>
>
>
> 1)      Be notified or see that a table had changed
>
> 2)      Checked that against changes I know my system has made
>
> 3)      If my system is not the originator of the change, update internal
> state to reflect the change.
>
>
>
> Examples of state I may need to update include an ElasticSearch index and
> also an in memory cache.
>
>
>
> I’m going to read up on constraints again and see if I can use them for
> this purpose.
>
>
>
> Thanks!
>
>
>
> Jon
>
>
>
>
>
>
>
> *From:* Adam Fuchs [mailto:afuchs@apache.org <afuchs@apache.org>]
> *Sent:* Tuesday, September 29, 2015 5:46 PM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Watching for Changes with Write Ahead Log?
>
>
>
> Jon,
>
>
>
> You might think about putting a constraint on your table. I think the API
> for constraints is flexible enough for your purpose, but I'm not exactly
> sure how you would want to manage the results / side effects of your
> observations.
>
>
>
> Adam
>
>
>
>
>
> On Tue, Sep 29, 2015 at 5:41 PM, Parise, Jonathan <
> Jonathan.Parise@gd-ms.com> wrote:
>
> Hi,
>
>
>
> I’m working on a system where generally changes to Accumulo will come
> through that system. However, in some cases, another system may change data
> without my system being aware of it.
>
>
>
> What I would like to do is somehow listen for changes to the tables my
> system cares about. I know there is a write ahead log that I could
> potentially listen to for changes, but I don’t know how to use it. I looked
> around for some documentation about it, and I don’t see much. I get the
> impression that it isn’t really intended for this type of use case.
>
>
>
> Does anyone have any suggestions on how to watch a table for changes and
> then determine if those changes were made by a different system.
>
>
>
> Is there some documentation about how to use the write ahead log?
>
>
>
>
>
> Thanks,
>
>
>
> Jon Parise
>
>
>
>
>

Mime
View raw message