accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <>
Subject Re: How to choose BinId for Document partitioned index
Date Mon, 08 Feb 2016 01:29:21 GMT
Yes! Very astute, Jamie :)

For the wikisearch schemas, the general idea is that the inverted index 
tables can prune your row space for some terms. This way, you can know 
the exact rows you have to search in the sharded table to get good 
parallelism without a full-table scan.

Jamie Johnson wrote:
> Thanks guys.  I was also looking at some of the examples and saw the
> event store, I like the idea of including time as a prefix to the
> binning to limit the number of servers that need to be hit for time
> bound queries.  Without something like this queries end up having to hit
> all tablets right?  It's not always a full table scan since the
> iterators can bail on a row part way through but still needs to hit
> every row to some extent right?
> I also was looking at the wiki example but wasn't able to find a good
> description of how all the tables are used, does anything more exist?
> On Feb 6, 2016 2:20 PM, "Josh Elser" <
> <>> wrote:
>     You can get *really* fancy if you have lots of ingesters and lots of
>     servers, include some attribute in the data you're hashing to
>     control how many servers a given client will need to write to for
>     some batch of documents. This is probably overkill for most setups
>     though.
>     Guava provides a decent murmur3 implementation which will be much
>     faster than your run-of-the-mill MD5 for generating the hash (which
>     you'll mod by the max number of bins).
>     William Slacum wrote:
>         Often it'll be a hash of the document mod the number of bins you're
>         using. The hash should be "good" in the sense that it uniquely
>         identifies the document. It can be as simple as some unique
>         field in the
>         document or just a hash (like murmur) of the whole document.
>         On Saturday, February 6, 2016, Jamie Johnson <
>         <>
>         < <>>> wrote:
>              Just found this excellent write up that explains a bit.
>              On Feb 6, 2016 8:52 AM, "Jamie Johnson" <
>         <>
>         <javascript:_e(%7B%7D,'cvml','
>         <>');>> wrote:
>                  Reading the examples for table design I've come across a
>                  question associated with the document partitioned index,
>                  specifically what is typically chosen as the BinId or
>         maybe more
>                  appropriately what factors should influence what is
>         chosen as
>                  the BinId and what impact do they have?

View raw message