accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cardon, Tejay E" <>
Subject RE: EXTERNAL: Re: Custom Iterators
Date Wed, 22 Aug 2012 22:22:27 GMT
Thanks for the quick response.  Let me start by stating what I understand about Iterators
(to be sure I'm not completely off my rocker).

1. An iterator receives, as its source, another iterator (by way of the init method), which
becomes it's source of data.
2. When seek is called on an iterator, the iterator should respond by moving the pointer to
the first key/value that applied to that iterator and is within the range
    a. Depending on the iterator, that may not be the first key in the range
    b. Only keys (and their corresponding values) which include one of the column families
listed in the family list should be available as topKey and topValue. (this restriction should
continue until seek is called again, meaning that subsequent calls to next will only proceed
to key/values that also match the list provided.
    c. Generally speaking, a seek will result in the iterator calling seek on its source iterator
(although the parameters passed in may be different)
3. If an iterator needs configuration beyond just the source obtained in the init call, it
can get that through the options and/or env.
4. Iterators do not necessarily return the same types of key/values as they consume.  ie,
a Combiner may call next() and getTopValue multiple times each time those methods are called
on it.  And the value it returns as topKey may be a key that doesn't actually exist in the
datastore itself.

So my questions:
Is it correct that once seek is called, only topKeys that conform to the columnFamilies collection
should be returned.  And that this behavior persists until seek is called again, even when
next has been called?
How do iterators like the OrIterator obtain multiple sources?  (I assume you were trying to
address that with #3 in your response, but I don't understand what you mean by clone()ing
the source.  That would give me copies of the one source, but not multiple sources)
Why do some iterators have so many constructors if the system will simply construct them from
the default constructor?
Some iterators (such as OrIterator) throw an exception if init is called.  How do these iterators
get constructed and initialized?

If OrIterator can do what I'm asking for, how do I get it the "terms" and what format do they
come in?  You mentioned JEXL expressions, but I haven't seen anything about them in the documentation.

As for my statement about the OrIterator and multiple rows, the comments on the compareTo
for OrIterator.TermSource state "If your implementation can have more than one row in a tablet,
you must compare row key here first, then column qualifier."  But the code does not do so.
 It may be that I'm just not fully understanding the code, however.

Finally, I'm actually trying to do something a little more complex than just what I described
below.  This reply is already too long and had too many questions in it, but I'll get more
detail out after I have a better handle on how the iterator framework works.


From: William Slacum []
Sent: Wednesday, August 22, 2012 3:00 PM
Subject: EXTERNAL: Re: Custom Iterators

An or clause should be able to handle an enumeration of values, as that's supported in a JEXL
expression. It would not, however, surprise me if those iterators could not handle multiple
rows in a tablet. If you can reproduce that, please file a ticket. There will be a large update
occurring to the Wiki example in the near future.

Do you have any specific questions about how you should structure your iterator or the contract?
Making a tutorial has been on my to do list, but we all know how to do lists end up...

The big things to remember are:

1) The call order: Your iterator will be created via the default constructor, init() will
be called, then seek(). After seek() is called, your iterator should have a top if there is
data available. A client then can call hasTop(), getTopKey() and getTopValue() to check and
retrieve data (similar to hasNext() and next()) and then next to advance the pointer.

2) Your iterator can be destroyed during a scan and then reconstructed, being passed in the
last key returned to the client as the start of the range.

3) You can have multiple sources feed into a single iterator in a tree like fashion by clone()'ing
the source passed in to init.
On Wed, Aug 22, 2012 at 1:41 PM, Cardon, Tejay E <<>>
I'm interested in writing a custom iterator, and I've been looking for documentation on how
to do so.  Thus far, I've not been able to find anything beyond the java docs in SortedKeyValueIterator
and a few other sub-classes.  A few of the examples use Iterators, but provide no real info
on how to properly implement one.  Is there anywhere to find general guidance on the iterator

(If you're interested)
Specifically, for those that are curious, I'm trying to implement something similar to the
wikisearch example, but with some key differences.  In my case, I've got a file with various
attributes that being indexed.  So for each file there are 5 attributes, and each attribute
has a fixed number of possible values.  For example (totally made up):
personID, gender, hair color, country, race, personRecord

Row:binID; ColFam:Attribute_AttributeValue; ColQ:PersonID; Val:blank
Row:binID; ColFam:"D"; ColQ:personID; value:personRecord

A typical query would be:
Give me the personRecord for all people with:
Gender: male &
Hair color: blond or brown &
Country: USA or England or china or korea &
Race: white or oriental

The existing Iterators used in the wikisearch example are unable to handle the "or" clauses
in each attribute.
The OrIterator doesn't appear to handle the possibility more than one row per tablet

Tejay Cardon

View raw message