accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Povey <...@maana.io>
Subject Accumulo Iterator painful development because TS don't pick up changes to Jars
Date Fri, 30 Oct 2015 00:00:53 GMT
Caveat I’m still running 1.6.2 internally here, and things may have changed and I could simply
“be doing it wrong”, or have missed the solution in the docs. It’s also probably not
a typical use case.

This is not really an issue for most day to day development, but our internal testing process
makes this changing iterators a nightmare.
Before I start I am aware of general.dynamic.classpaths, and because it appears that wildcards
are only respected at the file level, which is insufficient for our use case as you’ll see
later.

I’ll try and explain our internal test environment to help understand the issue.
We run daily (or more frequent) drops of our codebase against two internal clusters across
a variety of data sources (most of them aren’t particularly large). 
To give some idea I count 462 tables on one of of the clusters and each instance of the application
is using 11 or so tables of which 4 or so have a variety of iterators we’ve written.
To resolve the conflicts since our application predates namespaces we prefix the tables and
the table contexts and upload the iterators to subdirectories with matching names. 
To complicate matters further many of the tables are dropped and new tables added at a pretty
frightening rate, so having to change the configuration, and restart servers to add a new
path to the dynamic.classpath property is something of a none starter.

It all works fine until a build has a change in an iterator and is targeted against an existing
table, the app correctly identifies and uploads the new jars, but accumulo obviously doesn’t
pick up the change. In many cases I could live with it if simply dropping the tables and reingesting
was sufficient, but short of ingesting into a new table name even that doesn’t always pick
up the new Iterators.
We have currently resorted to manually tracking every iterator change (the rate of which has
at least slowed down recently) and doing rolling restarts of tablet servers on off hours,
but we end up often not knowing if an bug is real or an issue in a TS having an old iterator
loaded.

Is there a way to get the TS to watch an entire subtree for Jar changes?

Assuming there isn’t, when I get a few days without a looming deliverable, I was going to
migrate to 1.7 and if that has the same issue look at making and submitting a fix.


Rob Povey






On 10/28/15, 2:25 PM, "Josh Elser" <josh.elser@gmail.com> wrote:

>Rob Povey wrote:
>> However I’m pretty reticent right now to add anymore iterators to our
>> project, they’ve been a test nightmare for us internally.
>
>Off-topic, I'd like to hear more about what is painful. Do you have the 
>time to fork off a thread and let us know how it hurts?
Mime
View raw message