Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Content-Type: text/plain; charset=UTF-8
From: "Sukant Hajra" <qn2b6c2b9w@snkmail.com>
To: accumulo-user@incubator.apache.org
Subject: Re: sanity checking application WALogs make sense
In-Reply-To: 
 <CAMz+DuvZxH2D+P5QLgePWGkAvC1dwXJrmDbVhHSkYYd-1yicvA@mail.gmail.com>
References: <14781-1347687856-274275@sneakemail.com>
 <CAMz+DuvZxH2D+P5QLgePWGkAvC1dwXJrmDbVhHSkYYd-1yicvA@mail.gmail.com>
Date: Sat, 15 Sep 2012 13:14:49 -0500
Message-ID: <3321-1347732891-972412@sneakemail.com>
User-Agent: Sup/git
Content-Transfer-Encoding: 8bit

Excerpts from William Slacum's message of 2012-09-15 08:46:17 -0500:
>
> I'm a bit confused as to what you mean "if an iterator goes down
> mid-processing." If it goes down at all, then whatever scope it's running in-
> minor compaction, major compaction and scan- will most likely go down as well
> (unless your iterator eats an exception and ignores errors). A WALog
> shouldn't be deleted if whatever you were trying to do failed.

I believe I've answered my own question after thinking about iterators more and
looking at the code for some of the implementations.

I was thinking about iterators "writing" changes to Accumulo using something
like a BatchWriter.  Now I'm coming to the conclusion that even if that were
possible, it is not how iterators were designed, and very likely bad for data
integrity.  I don't feel that iterators should have any side-effects beyond
scanning data through the source provided by the init() method.  In this way,
I'm beginning to think about iterators more purely functionally.  Does that
sound right?  Or have people come up with iterator implementations with more
side-effects?

For instance, in one of my algorithms, authors might write conflicting data to
a row that needs to be resolved.  I feel I could install iterators at scan,
minor compaction, and major compaction to perform this resolution (which
happens to be a very simple idempotent operation).

Sorry if none of this sounds like a concrete question.  Some of what I'm
looking for is conversation and validation in light of some limited local
Accumulo expertise on my team.

Has anyone thought about building up a small IRC community, say on #accumulo on
Freenode?  There's a nice #hbase channel there, but at this point, I think I'm
past the point of asking Bigtable-general questions.

-Sukant