accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Parise, Jonathan" <Jonathan.Par...@gd-ms.com>
Subject RE: Watching for Changes with Write Ahead Log?
Date Thu, 01 Oct 2015 18:48:50 GMT
I think this is one of those things where there really aren’t great solutions.

A static connection could work, but if multiple Constraint instances can exist at the same
time, it would probably not work. Since all of them would be trying to use the same connection
at the same time.

ThreadLocal could possibly work better. The only question is how long lived is the thread
that calls the constraints? For example that thread could be torn down as soon as the constraint
is done. In that case the performance would be no better than creating and tearing down everything
each time check() is called.

This is why I am trying to understand the Constraint’s lifecycle, so I can come up with
the least bad way of solving this problem.

Thanks for the ideas! I am just not sure I know enough about the lifecycle of Constraints
to understand if these suggestions would be helpful.

Jon Parise

From: John Vines [mailto:vines@apache.org]
Sent: Thursday, October 01, 2015 2:40 PM
To: user@accumulo.apache.org
Subject: Re: Watching for Changes with Write Ahead Log?


As dirty as it is, that sounds like a case for a static, or maybe thread local, object

On Thu, Oct 1, 2015, 7:19 PM Parise, Jonathan <Jonathan.Parise@gd-ms.com<mailto:Jonathan.Parise@gd-ms.com>>
wrote:
I have a few follow up questions in regard to constraints.

What is the lifecycle of a constraint? What I mean by this is are the constraints somehow
tied to Accumulo’s lifecycle or are they just instantiated each time a mutation occurs and
then disposed?

Also, are there multiple instances of the same constraint class at any time or do all mutation
on a table go through the exact same constraint?

My guess is that  when a mutation comes in a new constraint is made through reflection. Then
check() is called, the violation codes are parsed and the object is disposed/finalized.

The reason I ask is that what I want to do is update my ElasticSearch index each time I see
a mutation on the table. However, I don’t want to have to make a connection, send the data
and then tear down the connection each time. That’s a lot of unnecessary overhead and with
all that overhead happening on every mutation performance could be badly impacted.

Is there some way to cache something like a connection and reuse it between calls to the Constraint’s
check() method? How would such a thing be cleaned up if Accumulo is shut down?


Thanks again,

Jon
From: Parise, Jonathan [mailto:Jonathan.Parise@gd-ms.com]
Sent: Wednesday, September 30, 2015 9:21 AM
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: RE: Watching for Changes with Write Ahead Log?

In this particular case, I need to update some of my application state when changes made by
another system occur.

I would need to do a few things to accomplish my goal.


1)      Be notified or see that a table had changed

2)      Checked that against changes I know my system has made

3)      If my system is not the originator of the change, update internal state to reflect
the change.

Examples of state I may need to update include an ElasticSearch index and also an in memory
cache.

I’m going to read up on constraints again and see if I can use them for this purpose.

Thanks!

Jon



From: Adam Fuchs [mailto:afuchs@apache.org]
Sent: Tuesday, September 29, 2015 5:46 PM
To: user@accumulo.apache.org<mailto:user@accumulo.apache.org>
Subject: Re: Watching for Changes with Write Ahead Log?

Jon,

You might think about putting a constraint on your table. I think the API for constraints
is flexible enough for your purpose, but I'm not exactly sure how you would want to manage
the results / side effects of your observations.

Adam


On Tue, Sep 29, 2015 at 5:41 PM, Parise, Jonathan <Jonathan.Parise@gd-ms.com<mailto:Jonathan.Parise@gd-ms.com>>
wrote:
Hi,

I’m working on a system where generally changes to Accumulo will come through that system.
However, in some cases, another system may change data without my system being aware of it.

What I would like to do is somehow listen for changes to the tables my system cares about.
I know there is a write ahead log that I could potentially listen to for changes, but I don’t
know how to use it. I looked around for some documentation about it, and I don’t see much.
I get the impression that it isn’t really intended for this type of use case.

Does anyone have any suggestions on how to watch a table for changes and then determine if
those changes were made by a different system.

Is there some documentation about how to use the write ahead log?


Thanks,

Jon Parise

Mime
View raw message