Return-Path: X-Original-To: apmail-accumulo-user-archive@www.apache.org Delivered-To: apmail-accumulo-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AAE1F10D55 for ; Fri, 20 Sep 2013 16:48:29 +0000 (UTC) Received: (qmail 88434 invoked by uid 500); 20 Sep 2013 16:48:27 -0000 Delivered-To: apmail-accumulo-user-archive@accumulo.apache.org Received: (qmail 88085 invoked by uid 500); 20 Sep 2013 16:48:21 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 88077 invoked by uid 99); 20 Sep 2013 16:48:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Sep 2013 16:48:19 +0000 X-ASF-Spam-Status: No, hits=-0.1 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_MED,SPF_HELO_PASS,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of David.Slater@jhuapl.edu designates 128.244.251.37 as permitted sender) Received: from [128.244.251.37] (HELO piper.jhuapl.edu) (128.244.251.37) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Sep 2013 16:48:11 +0000 Received: from aplexcas1.dom1.jhuapl.edu (unknown [128.244.198.90]) by piper.jhuapl.edu with smtp (TLS: TLSv1/SSLv3,128bits,RC4-MD5) id 686a_521f_7c6d0126_909f_4e7c_83f9_c93cbc56c29b; Fri, 20 Sep 2013 12:47:48 -0400 Received: from aplesstripe.dom1.jhuapl.edu ([128.244.198.211]) by aplexcas1.dom1.jhuapl.edu ([128.244.198.90]) with mapi; Fri, 20 Sep 2013 12:47:48 -0400 From: "Slater, David M." To: "user@accumulo.apache.org" Date: Fri, 20 Sep 2013 12:47:47 -0400 Subject: RE: BatchWriter performance on 1.4 Thread-Topic: BatchWriter performance on 1.4 Thread-Index: Ac61jEhOwJxUVssUQSiKRVl79bXd2AAlB5BA Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_AC78983C72177B4D9D1C14F7F4AEBA214478B6D86Caplesstripedo_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_AC78983C72177B4D9D1C14F7F4AEBA214478B6D86Caplesstripedo_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I was using flush() after sending a bunch of mutations to the batchwriters = to limit their latency. I thought it would normally flush the buffer to ens= ure that the maxLatency is not violated. If the maxLatency is quite large, = how do I ensure that it doesn't wait a long time before writing? If the returned batchscanners are all thread safe, then I'm still going to = have the bottleneck of their synchronized addMutations method, correct? I'm looking for "org.apache.accumulo.client.impl" in the log4j.properties, = generic_logger.xml the and other config files, but can't locate it. Do I ne= ed to create a new entry for it there? Thanks, David From: Keith Turner [mailto:keith@deenlo.com] Sent: Thursday, September 19, 2013 7:01 PM To: user@accumulo.apache.org Subject: Re: BatchWriter performance on 1.4 On Thu, Sep 19, 2013 at 5:08 PM, Slater, David M. > wrote: Thanks Keith, I'm looking at it now. It appears like what I would want. As = for the proper usage... Would I create one using the Connector, then .getBatchWriter() for each of the tables I'm interested in, add data to each of BatchWriters returned, yes. and then hit flush() when I want to write all of that to get written? Why are you calling flush() ? Doing this frequently will increase rpc ove= rhead and lower throughput. Would the individual batch writers spawned by the multiTableBatchWriter sti= ll have synchronized addMutations() methods so I would have to worry about = blocking still, or would that all happen at the flush() method? The returned batch writers are thread safe. They all add to the same queue/= buffer in a synchronized manner. Calling flush() on any of the batch writ= ers returned from getBatchWriter() will block the others. If you enable set the log4j log level to TRACE for org.apache.accumulo.clie= nt.impl you can see output like the following. Binning is the process of t= aking each mutation and deciding which tablet and tablet server it goes to. 2013-09-19 18:43:37,261 [impl.ThriftTransportPool] TRACE: Using existing = connection to 127.0.0.1:9997 2013-09-19 18:43:37,393 [impl.TabletLocatorImpl] TRACE: tid=3D12 oid=3D13= Binning 80909 mutations for table 3 2013-09-19 18:43:37,402 [impl.TabletLocatorImpl] TRACE: tid=3D12 oid=3D13= Binned 80909 mutations for table 3 to 1 tservers in 0.009 secs 2013-09-19 18:43:37,402 [impl.TabletServerBatchWriter] TRACE: Started sen= ding 80,909 mutations to 1 tablet servers 2013-09-19 18:43:37,656 [impl.ThriftTransportPool] TRACE: Returned connec= tion 127.0.0.1:9997 (120000) ioCount : 1459116 2013-09-19 18:43:37,657 [impl.TabletServerBatchWriter] TRACE: sent 80,909= mutations to 127.0.0.1:9997 in 0.40 secs (204,832.9= 1 mutations/sec) with 0 failures When you close the batch writer, it will log some summary stats like the fo= llowing. 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: TABLET SERV= ER BATCH WRITER STATISTICS 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: Added = : 1,000,000 mutations 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: Sent = : 1,000,000 mutations 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: Resent perc= entage : 0.00% 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: Overall tim= e : 5.94 secs 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: Overall sen= d rate : 168,406.87 mutations/sec 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: Send effici= ency : 86.91% 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: BACKGROUND = WRITER PROCESS STATISTICS 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: Total send = time : 5.16 secs 86.91% 2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: Average sen= d rate : 193,760.90 mutations/sec 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: Total bin t= ime : 0.46 secs 7.81% 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: Average bin= rate : 2,155,172.41 mutations/sec 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: tservers pe= r batch : 1.00 avg 1 min 1 max 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: tablets per= batch : 1.00 avg 1 min 1 max 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: SYSTEM STAT= ISTICS 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: JVM GC Time= : 0.53 secs 2013-09-19 18:43:39,152 [impl.TabletServerBatchWriter] TRACE: JVM Compile= Time : 1.60 secs 2013-09-19 18:43:39,152 [impl.TabletServerBatchWriter] TRACE: System load= average : initial=3D 0.22 final=3D 0.20 What do these numbers look like for you? Keith From: Keith Turner [mailto:keith@deenlo.com] Sent: Thursday, September 19, 2013 12:39 PM To: user@accumulo.apache.org Subject: Re: BatchWriter performance on 1.4 Are you aware of the multi table batch writer? I am not sure if it would b= e useful, but wanted to make sure you knew about it. It will use the same= thread pool to process mutations for multiple tables. Also it will batch = mutations for multiple tablets into the same rpc calls. On Wed, Sep 18, 2013 at 5:07 PM, Slater, David M. > wrote: Hi, I'm running a single-threaded ingestion program that takes data from an= input source, parses it into mutations, and then writes those mutations (s= equentially) to four different BatchWriters (all on different tables). Most= of the time (95%) taken is on adding mutations, e.g. batchWriter.addMutati= ons(mutations); I am wondering how to reduce the time taken by these method= s. 1) For the method batchWriter.addMutations(Iterable), does it mat= ter for performance whether the mutations returned by the iterator are sort= ed in lexicographic order? 2) If the Iterable that I pass to the BatchWriter is very large, = will I need to wait for a number of Batches to be written and flushed befor= e it will finish iterating, or does it transfer the elements of the Iterabl= e to a different intermediate list? 3) If that is the case, would it then make sense to spawn off short threads= for each time I make use of addMutations? At a high level, my code looks like this: BatchWriter bw1 =3D connector.createBatchWriter(...) BatchWriter bw2 =3D ... ... while(true) { String[] data =3D input.getData(); List mutations1 =3D parseData1(data); List mutations2 =3D parseData2(data); ... bw1.addMutations(mutations1); bw2.addMutations(mutations2); ... } Thanks, David --_000_AC78983C72177B4D9D1C14F7F4AEBA214478B6D86Caplesstripedo_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I was usi= ng flush() after sending a bunch of mutations to the batchwriters to limit = their latency. I thought it would normally flush the buffer to ensure that = the maxLatency is not violated. If the maxLatency is quite large, how do I = ensure that it doesn’t wait a long time before writing?

 

If the returned batchscanners are all thread safe, then= I’m still going to have the bottleneck of their synchronized addMuta= tions method, correct?

 

I’m looking for = “org.apache.accumulo.client.impl” in the log4j.pr= operties, generic_logger.xml the and other config files, but can’t lo= cate it. Do I need to create a new entry for it there?

 

Thanks,
David

 

From: Keith Turner [mailto:= keith@deenlo.com]
Sent: Thursday, September 19, 2013 7:01 PM
= To: user@accumulo.apache.org
Subject: Re: BatchWriter perf= ormance on 1.4

 <= /p>

On Thu, Sep 19, 2013 at 5:08 PM, Sla= ter, David M. <David.Slater@jhuapl.edu> wrote:

<= div>

<= span style=3D'font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F= 497D'>Thanks Keith, I’m looking at it now. It appears like what I wou= ld want. As for the proper usage…

=  

Would I create one using the Conn= ector,

then .getBatchWriter() for each = of the tables I’m interested in,

a= dd data to each of BatchWriters returned,

=

 

yes.

 <= o:p>

and then hit flush() when I want to write all of that = to get written?

 

Why are y= ou calling flush() ?   Doing this frequently will increase rpc overhea= d and lower throughput.

 = ;

<= div>

 

Would the = individual batch writers spawned by the multiTableBatchWriter still have sy= nchronized addMutations() methods so I would have to worry about blocking s= till, or would that all happen at the flush() method?

=

 

The returned batch writers are thread safe. Th= ey all add to the same queue/buffer in a synchronized manner.   C= alling flush() on any of the batch writers returned from getBatchWriter() w= ill block the others.   

 

If you= enable set the log4j log level to TRACE for org.apache.accumulo.client.imp= l you can see output like the following.  Binning is the process = of taking each mutation and deciding which tablet and tablet server it goes= to.

 

  2013-09-19 18:43:37,261 [impl.ThriftTra= nsportPool] TRACE: Using existing connection to 127.0.0.1:9997

&nb= sp; 2013-09-19 18:43:37,393 [impl.TabletLocatorImpl] TRACE: tid=3D12 oid=3D= 13  Binning 80909 mutations for table 3

  2013-09-19 18:43:37,402 [impl.TabletLocatorImpl] TRA= CE: tid=3D12 oid=3D13  Binned 80909 mutations for table 3 to 1 tserver= s in 0.009 secs

  2013-0= 9-19 18:43:37,402 [impl.TabletServerBatchWriter] TRACE: Started sending 80,= 909 mutations to 1 tablet servers

  2013-09-19 18:43:37,656 [impl.ThriftTransportPool] TRACE: Retur= ned connection 127.0.0.1:9997 (120000= ) ioCount : 1459116

  20= 13-09-19 18:43:37,657 [impl.TabletServerBatchWriter] TRACE: sent 80,909 mut= ations to 127.0.0.1:9997 in 0.40 secs= (204,832.91 mutations/sec) with 0 failures

=

 

W= hen you close the batch writer, it will log some summary stats like the fol= lowing.   

&nb= sp;

 

  2013-09-19 18:43:39,149 [impl.TabletServerBat= chWriter] TRACE: 

 = 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: TABLET SERVE= R BATCH WRITER STATISTICS

&nb= sp; 2013-09-19 18:43:39,149 [impl.TabletServerBatchWriter] TRACE: Added &nb= sp;              :  1,000,000 mutat= ions

  2013-09-19 18:43:= 39,149 [impl.TabletServerBatchWriter] TRACE: Sent       &nbs= p;         :  1,000,000 mutations

  2013-09-19 18:43:39,149 [impl.TabletSe= rverBatchWriter] TRACE: Resent percentage   :       0.0= 0%

  2013-09-19 18:43:39= ,150 [impl.TabletServerBatchWriter] TRACE: Overall time      = ;   :       5.94 secs

  2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] = TRACE: Overall send rate    : 168,406.87 mutations/sec=

  2013-09-19 18:43:39,150 [impl.Ta= bletServerBatchWriter] TRACE: Send efficiency      :   =    86.91%

  20= 13-09-19 18:43:39,150 [impl.TabletServerBatchWriter] TRACE: 

  2013-09-19 18:43:39,150 [impl.T= abletServerBatchWriter] TRACE: BACKGROUND WRITER PROCESS STATISTICS

  2013-09-19 18:43:39,150 [impl= .TabletServerBatchWriter] TRACE: Total send time      : &nbs= p;     5.16 secs  86.91%

  2013-09-19 18:43:39,150 [impl.TabletServerBatchWriter] = TRACE: Average send rate    : 193,760.90 mutations/sec=

  2013-09-19 18:43:39,151 [impl.Ta= bletServerBatchWriter] TRACE: Total bin time       :   =     0.46 secs   7.81%

  2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE= : Average bin rate     : 2,155,172.41 mutations/sec

  2013-09-19 18:43:39,151 [impl.Table= tServerBatchWriter] TRACE: tservers per batch   :     1.00 a= vg       1 min      1 max

  2013-09-19 18:43:39,151 [impl.TabletServe= rBatchWriter] TRACE: tablets per batch    :     1.00 av= g       1 min      1 max

=

  2013-09-19 18:43:39,151 [impl.TabletServer= BatchWriter] TRACE: 

&nb= sp; 2013-09-19 18:43:39,151 [impl.TabletServerBatchWriter] TRACE: SYSTEM ST= ATISTICS

  2013-09-19 18= :43:39,151 [impl.TabletServerBatchWriter] TRACE: JVM GC Time     =      :       0.53 secs

  2013-09-19 18:43:39,152 [impl.TabletServerBa= tchWriter] TRACE: JVM Compile Time     :       1.6= 0 secs

  2013-09-19 18:4= 3:39,152 [impl.TabletServerBatchWriter] TRACE: System load average : initia= l=3D  0.22 final=3D  0.20

 

What do t= hese numbers look like for you?

 

Keith

 

 

From: Keith Turner [mailto:keith@deenlo.com]
Sent: Thursday,= September 19, 2013 12:39 PM
To: user@accumulo.apache.org


Subject: Re: BatchWriter performa= nce on 1.4

 

Are = you aware of the multi table batch writer?  I am not sure if it would = be useful, but wanted to make sure you knew about it.   It will use th= e same thread pool to process mutations for multiple tables.  Also it = will batch mutations for multiple tablets into the same rpc calls.

 

On Wed, Sep 18= , 2013 at 5:07 PM, Slater, David M. <David.Slater@jhuapl.edu> wrote:=

Hi, I’m running a single-threaded ingestion prog= ram that takes data from an input source, parses it into mutations, and the= n writes those mutations (sequentially) to four different BatchWriters (all= on different tables). Most of the time (95%) taken is on adding mutations,= e.g. batchWriter.addMutations(mutations); I am wondering how to reduce the= time taken by these methods.

 

1) For the method batchWriter.addMutations(Iterable<Mutation>), = does it matter for performance whether the mutations returned by the iterat= or are sorted in lexicographic order?

 

2) If the Iterable<Mutation> that I pass to the BatchWri= ter is very large, will I need to wait for a number of Batches to be writte= n and flushed before it will finish iterating, or does it transfer the elem= ents of the Iterable to a different intermediate list?

 

3) If that is the case, would it then make sen= se to spawn off short threads for each time I make use of addMutations?

 

At a high level, my code look= s like this:

 

BatchWriter b= w1 =3D connector.createBatchWriter(…)

BatchWrite= r bw2 =3D …

…

while(t= rue) {

String[] data =3D input.getDat= a();

List<Mutation> mutations1 = =3D parseData1(data);

     &n= bsp;          List<Mutation= > mutations2 =3D parseData2(data);

  &nbs= p;             = …

       &nbs= p;        bw1.addMutations(mutations1);<= o:p>

         =        bw2.addMutations(mutations2);

          &n= bsp;     …

}

<= p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:= auto'>Thanks,
David

 

 

= --_000_AC78983C72177B4D9D1C14F7F4AEBA214478B6D86Caplesstripedo_--