Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@accumulo.apache.org
Received-SPF: pass (athena.apache.org: domain of david.medinets@gmail.com
 designates 74.125.82.51 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPx=JkaEf18dwrHmW=V53oOTcXyx5H7BKsuRFUSjD0_TUDgiuw@mail.gmail.com>
References: 
 <CAPx=JkaEf18dwrHmW=V53oOTcXyx5H7BKsuRFUSjD0_TUDgiuw@mail.gmail.com>
Date: Sat, 4 Apr 2015 16:38:30 -0400
Message-ID: 
 <CAOiJXP6aDZ+5eRN+Cp2AcTrzZZZCeXQ73_riBNR882VwvSagFA@mail.gmail.com>
Subject: Re: Local Combiners to pre-sum at BatchWriter
From: David Medinets <david.medinets@gmail.com>
To: accumulo-user <user@accumulo.apache.org>
Content-Type: multipart/alternative; boundary=001a1134cfd842d0a10512ec0ccf

--001a1134cfd842d0a10512ec0ccf
Content-Type: text/plain; charset=UTF-8

Aren't you  essentially adding another of accumulo node?
On Apr 4, 2015 3:59 PM, "Dylan Hutchison" <dhutchis@mit.edu> wrote:

> I've been thinking about a scenario that seems common among high-ingest
> Accumulo users. Suppose we have a "combiner"-type iterator on a table on
> all scopes.  One technique to increase ingest performance is "pre-summing":
> run the combiner on local entries before they are sent through a
> BatchWriter, in order to reduce the number of entries sent to the tablet
> server.
>
> One way to do pre-summing is to create a Map<Key,Value> of entries to send
> to the server on the local client. This equates to the following client
> code, run for each entry to send to Accumulo:
>
>   Key k = nextKeyToSend();
>   Value v = nextValueToSend();
>   Value vPrev = map.get(k);
>   if (vPrev != null)
>     v = combiner.combine(vPrev, v);
>   map.put(k, v);
>
> Each time our map size exceeds a threshold (don't want to run out of
> memory on the client),
>
>   BatchWriter bw; // setup previously from connector
>   for (Map.Entry<Key,Value> entry : map.entrySet()) {
>     Key k = entry.getKey();
>     Mutation m = new Mutation(k.getRow());
>     m.put(k.getColumnFamily(), k.getColumnQualifier(), entry.getValue());
>     bw.addMutation(m);
>   }
>
> (side note: using one entry change per mutation.  I've never investigated
> whether it would be more efficient to put all the updates to a single row
> [i.e. chaining multiple columns in the same row] in one mutation instead.)
>
> This solution works, but it duplicates the purpose of the BatchWriter and
> adds complexity to the client.  If we have to create a separate "cache"
> collection, track its size and dump to a BatchWriter once it gets too big,
> then it seems like we're reimplementing the behavior of the BatchWriter
> that provides an internal cache of size set by
> BatchWriterConfig.setMaxMemory() (that starts flushing once half the
> maximum memory is used), and we're using two caches (user-created map + the
> BatchWriter) where one should be sufficient.
>
> I'm wondering whether there is a way to pre-sum mutations added to a
> BatchWriter automatically, so that we can add entries to a BatchWriter and
> trust that it will apply a combiner function to them before transmitting to
> the tablet server. Something to the effect of:
>
>   BatchWriter bw; // setup previously from connector
>   Combiner combiner = new SummingCombiner();
>   Map<String, String> combinerOptions = new HashMap<>();
>   combinerOptions.put("all", "true"); // or some other column subset option
>   bw.addCombiner(combiner);
>   // or perhaps more generally/ambitiously: bw.addWriteIterator(combiner);
>
>   // effect: combiner will be applied right before flushing data to server
>   // if the combiner throws an exception, then throw a
> MutationsRejectedException
>
> Is there a better way to accomplish this, without duplicating
> BatchWriter's buffer?  Or would this make a nice addition to the API?  If I
> understand the BatchWriter correctly, it already sorts entries before
> sending to the tablet server, because the tablet server can process them
> more efficiently that way.  If so, the overhead cost seems small to add a
> combining step after the sorting phase and before the network transmit
> phase, especially if it reduces network traffic anyway.
>
> Regards,
> Dylan Hutchison
>
>

--001a1134cfd842d0a10512ec0ccf
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<p dir=3D"ltr">Aren&#39;t you=C2=A0 essentially adding another of accumulo =
node?</p>
<div class=3D"gmail_quote">On Apr 4, 2015 3:59 PM, &quot;Dylan Hutchison&qu=
ot; &lt;<a href=3D"mailto:dhutchis@mit.edu">dhutchis@mit.edu</a>&gt; wrote:=
<br type=3D"attribution"><blockquote class=3D"gmail_quote" style=3D"margin:=
0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"><d=
iv>I&#39;ve been thinking about a scenario that seems common among high-ing=
est Accumulo users. Suppose we have a &quot;combiner&quot;-type iterator on=
 a table on all scopes.=C2=A0 One technique to increase ingest performance =
is &quot;pre-summing&quot;: run the combiner on local entries before they a=
re sent through a BatchWriter, in order to reduce the number of entries sen=
t to the tablet server.</div><div><br></div><div>One way to do pre-summing =
is to create a Map&lt;Key,Value&gt; of entries to send to the server on the=
 local client. This equates to the following client code, run for each entr=
y to send to Accumulo:</div><div><br></div><div>=C2=A0 Key k =3D nextKeyToS=
end();</div><div>=C2=A0 Value v =3D nextValueToSend();</div><div>=C2=A0 Val=
ue vPrev =3D map.get(k);</div><div>=C2=A0 if (vPrev !=3D null)</div><div>=
=C2=A0 =C2=A0 v =3D combiner.combine(vPrev, v);</div><div>=C2=A0 map.put(k,=
 v);</div><div><br></div><div>Each time our map size exceeds a threshold (d=
on&#39;t want to run out of memory on the client),</div><div><br></div><div=
>=C2=A0 BatchWriter bw; // setup previously from connector</div><div>=C2=A0=
 for (Map.Entry&lt;Key,Value&gt; entry : map.entrySet()) {</div><div>=C2=A0=
 =C2=A0 Key k =3D entry.getKey();</div><div>=C2=A0 =C2=A0 Mutation m =3D ne=
w Mutation(k.getRow());</div><div>=C2=A0 =C2=A0 m.put(k.getColumnFamily(), =
k.getColumnQualifier(), entry.getValue());</div><div>=C2=A0 =C2=A0 bw.addMu=
tation(m);</div><div>=C2=A0 }</div><div><br></div><div>(side note: using on=
e entry change per mutation.=C2=A0 I&#39;ve never investigated whether it w=
ould be more efficient to put all the updates to a single row [i.e. chainin=
g multiple columns in the same row] in one mutation instead.)</div><div><br=
></div><div>This solution works, but it duplicates the purpose of the Batch=
Writer and adds complexity to the client.=C2=A0 If we have to create a sepa=
rate &quot;cache&quot; collection, track its size and dump to a BatchWriter=
 once it gets too big, then it seems like we&#39;re reimplementing the beha=
vior of the BatchWriter that provides an internal cache of size set by Batc=
hWriterConfig.setMaxMemory() (that starts flushing once half the maximum me=
mory is used), and we&#39;re using two caches (user-created map + the Batch=
Writer) where one should be sufficient.</div><div><br></div><div>I&#39;m wo=
ndering whether there is a way to pre-sum mutations added to a BatchWriter =
automatically, so that we can add entries to a BatchWriter and trust that i=
t will apply a combiner function to them before transmitting to the tablet =
server. Something to the effect of:</div><div><br></div><div>=C2=A0 BatchWr=
iter bw; // setup previously from connector</div><div>=C2=A0 Combiner combi=
ner =3D new SummingCombiner();</div><div>=C2=A0 Map&lt;String, String&gt; c=
ombinerOptions =3D new HashMap&lt;&gt;();</div><div>=C2=A0 combinerOptions.=
put(&quot;all&quot;, &quot;true&quot;); // or some other column subset opti=
on</div><div>=C2=A0 bw.addCombiner(combiner);</div><div>=C2=A0 // or perhap=
s more generally/ambitiously: bw.addWriteIterator(combiner);</div><div>=C2=
=A0=C2=A0</div><div>=C2=A0 // effect: combiner will be applied right before=
 flushing data to server</div><div>=C2=A0 // if the combiner throws an exce=
ption, then throw a MutationsRejectedException</div><div><br></div><div>Is =
there a better way to accomplish this, without duplicating BatchWriter&#39;=
s buffer?=C2=A0 Or would this make a nice addition to the API?=C2=A0 If I u=
nderstand the BatchWriter correctly, it already sorts entries before sendin=
g to the tablet server, because the tablet server can process them more eff=
iciently that way.=C2=A0 If so, the overhead cost seems small to add a comb=
ining step after the sorting phase and before the network transmit phase, e=
specially if it reduces network traffic anyway.</div><div><br></div><div>Re=
gards,</div><div>Dylan Hutchison</div><div><br></div></div>
</blockquote></div>

--001a1134cfd842d0a10512ec0ccf--