Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CACAwpDPVBJmqtfFmw9o4Lgc5Gm+GPUgvP_qB-St_LDLRjThWoA@mail.gmail.com>
References: 
 <CACAwpDPVBJmqtfFmw9o4Lgc5Gm+GPUgvP_qB-St_LDLRjThWoA@mail.gmail.com>
Date: Wed, 30 Sep 2015 11:27:10 -0400
Message-ID: 
 <CAPMpPc5hcXuNnfhkcPX3TU0BQAT0He9oeYeuxPvT2ukk_WAmrA@mail.gmail.com>
Subject: Re: Document Partitioned Indexing
From: Adam Fuchs <afuchs@apache.org>
To: user@accumulo.apache.org
Content-Type: multipart/alternative; boundary=089e0117646d73f9290520f89052

--089e0117646d73f9290520f89052
Content-Type: text/plain; charset=UTF-8

Hi Tom,

Sqrrl uses a document-distributed indexing strategy extensively. On top of
the reasons you mentioned, we also like the ability to explicitly structure
our index entries in both information content and sort order. This gives us
the ability to do interesting things like build custom indexes and do joins
between graph indexes and term indexes.

Eventually, I'd like to see Accumulo build out explicit support for this
type of indexing in the core as an embedded secondary indexing capability.
That would solve several of the challenges around compatibility with other
Accumulo features and usage patterns.

Cheers,
Adam


On Wed, Sep 30, 2015 at 3:48 AM, Tom D <tomdata8@gmail.com> wrote:

> Hi,
>
> Have been doing a little reading about different distributed (text)
> indexing techniques and picked up on the Document Partitioned Index
> approach on Accumulo.
>
> I am interested in the use-cases people would have for indexing data in
> this way over using a distributed search service (Elastic or SolrCloud).
>
> I can think of a few reasons, but wondered if there's something more
> obvious that I'm missing?
>
> - cell (field level) access controls
>
> - scale - I understand Accumulo will scale to thousands of nodes. I
> believe there are some limitations in Elastic / Solr at about 100 nodes.
>
> - integration with an existing schema or index in Accumulo (not sure about
> this one and what benefits it would have over calling out to a search
> service)
>
> - you want to take advantage of other features in Accumulo, e.g. Combining
> iterators to perform some aggregation alongside your document partitioned
> index (again, can't imagine use cases here, but maybe there are some)
>
> - more control over 'messy data', e.g partial duplicates that need merging
> at ingest
>
> Are there others? Be interesting to hear if people use this indexing
> strategy.
>
> Many thanks.
>
>
>

--089e0117646d73f9290520f89052
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Tom,<input name=3D"virtru-metadata" type=3D"hidden" val=
ue=3D"{&quot;email-policy&quot;:{&quot;state&quot;:&quot;closed&quot;,&quot=
;expirationUnit&quot;:&quot;days&quot;,&quot;disableCopyPaste&quot;:false,&=
quot;disablePrint&quot;:false,&quot;disableForwarding&quot;:false,&quot;exp=
ires&quot;:false},&quot;attachments&quot;:{}}"><div><br></div><div>Sqrrl us=
es a document-distributed indexing strategy extensively. On top of the reas=
ons you mentioned, we also like the ability to explicitly structure our ind=
ex entries in both information content and sort order. This gives us the ab=
ility to do interesting things like build custom indexes and do joins betwe=
en graph indexes and term indexes.</div><div><br></div><div>Eventually, I&#=
39;d like to see Accumulo build out explicit support for this type of index=
ing in the core as an embedded secondary indexing capability. That would so=
lve several of the challenges around compatibility with other Accumulo feat=
ures and usage patterns.</div><div><br></div><div>Cheers,</div><div>Adam</d=
iv><div><br></div></div><div class=3D"gmail_extra"><br><div class=3D"gmail_=
quote">On Wed, Sep 30, 2015 at 3:48 AM, Tom D <span dir=3D"ltr">&lt;<a href=
=3D"mailto:tomdata8@gmail.com" target=3D"_blank">tomdata8@gmail.com</a>&gt;=
</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex"><div style=3D"white-space:=
pre-wrap">Hi,<br><br>Have been doing a little reading about different distr=
ibuted (text) indexing techniques and picked up on the Document Partitioned=
 Index approach on Accumulo. <br><br>I am interested in the use-cases peopl=
e would have for indexing data in this way over using a distributed search =
service (Elastic or SolrCloud).<br><br>I can think of a few reasons, but wo=
ndered if there&#39;s something more obvious that I&#39;m missing?<br><br>-=
 cell (field level) access controls<br><br>- scale - I understand Accumulo =
will scale to thousands of nodes. I believe there are some limitations in E=
lastic / Solr at about 100 nodes. <br><br>- integration with an existing sc=
hema or index in Accumulo (not sure about this one and what benefits it wou=
ld have over calling out to a search service)<br><br>- you want to take adv=
antage of other features in Accumulo, e.g. Combining iterators to perform s=
ome aggregation alongside your document partitioned index (again, can&#39;t=
 imagine use cases here, but maybe there are some)<br><br>- more control ov=
er &#39;messy data&#39;, e.g partial duplicates that need merging at ingest=
<br><br>Are there others? Be interesting to hear if people use this indexin=
g strategy. <br><br>Many thanks. <br><br><br></div>
</blockquote></div><br></div>

--089e0117646d73f9290520f89052--