From user-return-7377-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org Wed Aug 29 22:03:46 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 84CA2180657 for ; Wed, 29 Aug 2018 22:03:45 +0200 (CEST) Received: (qmail 37946 invoked by uid 500); 29 Aug 2018 20:03:44 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 37936 invoked by uid 99); 29 Aug 2018 20:03:44 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 29 Aug 2018 20:03:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 194A4C05D2 for ; Wed, 29 Aug 2018 20:03:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.119 X-Spam-Level: ** X-Spam-Status: No, score=2.119 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 1iyugdKq2Exd for ; Wed, 29 Aug 2018 20:03:42 +0000 (UTC) Received: from mail-oi0-f67.google.com (mail-oi0-f67.google.com [209.85.218.67]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 08A0B5F382 for ; Wed, 29 Aug 2018 20:03:42 +0000 (UTC) Received: by mail-oi0-f67.google.com with SMTP id 8-v6so11438420oip.0 for ; Wed, 29 Aug 2018 13:03:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=dCFPjQdSff+k6c5AO3S9LYIApAm7phPSoIm3EwKma5U=; b=MGB4NlX7thmJggyMT1epMlDa66guAmFDxpBMr/UZvyykUhjiJosIQzGDWOpPZcezIO K6mPlcY22CHFPyPrBjd4fbaDRFLL34EvckcQGDHgCN4+JkMD6wxMcwwfyrPomQOvLmwX wZbhK4ypVr63EMlXyY5XEESCDp00+sb1/IRZ0V6G6WExXV5LfmImpZmCDZsmugQXxUNF FInyOXQr3WGhtwOnSQ1qAAD1X2O05VRD0vnvhc/rd58MsHTIn7vqRQmVW2JUG7KzGqcc OtdNJQ+IV2FLWs3yMJELFQ5+8a25hH6ObtepLi1g9mQWQhNU5kzZArI31P++5Ki/qQQ8 yRdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=dCFPjQdSff+k6c5AO3S9LYIApAm7phPSoIm3EwKma5U=; b=JbZr3l33RZzcjS8frGt25B7BIcleRNiMe5g8xDjpnRMq4d9oCWq8mb90ZPvh9mEm3j xKgZB3fnAxlk1hCkV6FThrq/QsO8+Qy7Ek29Ej14krlO7rhgEQqik01zaIs5jFbW82zi E7HLppJJtDcojEb/Dnnsve5uS+iw1GKkKHMtw44YzFBnW9LV6GLolY+ctqRqQXpbhEWh +682MfIlHp4nNt8uYfyFf9n3NxZa5us+4hf1Uu45/Uv4g8AF7zsGBYwDTIEMJf55d6pS K8EXIT6jQAjkBHE4wFv1O1z8OPN1GCkryIdn4VknHRbR8QTOj70HvPr10I+Fbife72kE XwlQ== X-Gm-Message-State: APzg51A/vN/MbqDvsLp3R95H8u+nkB9/gg+j5ZjTblBeuZ6ASsmNOEbi 89dKMoTBvqz3Dx36XALYEmiF9JDJ3dWJWoOuG0LobwAd X-Google-Smtp-Source: ANB0VdZJ1CiNzyuiHTl5KGZyHLiN/qX4ebH+m/PaPQxZ65eGVLHoYVYhNXohwvzPaJNSjJejGe6jRvv/wZATVmltMVo= X-Received: by 2002:aca:ad86:: with SMTP id w128-v6mr4401050oie.112.1535573021209; Wed, 29 Aug 2018 13:03:41 -0700 (PDT) MIME-Version: 1.0 References: <005d01d43fd0$dcaa9020$95ffb060$@comcast.net> In-Reply-To: <005d01d43fd0$dcaa9020$95ffb060$@comcast.net> From: guy sharon Date: Wed, 29 Aug 2018 23:03:29 +0300 Message-ID: Subject: Re: Accumulo performance on various hardware configurations To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary="00000000000074b9c1057498745a" --00000000000074b9c1057498745a Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable OK, good news at last! Performance has improved by one order of magnitude. The bad news is that I don't know why. As far as I can tell the only things I changed were table.scan.max.memory and the performance profile on Muchos. Both had no effect when I initially tested them and I doubt that there's a "settling in" period for both of these so I don't know what it is. But at least now it's clear that it's possible to get better performance. I'll try to investigate why it happened. Thanks everyone for helping! Regarding my use case: I'm trying to use Accumulo as a graph database. Traversing a graph, or several graphs at once, means getting a row (vertex) by ID, sending it to the client, deciding if it's relevant and then retrieving the neighboring vertices. So lots of reads by ID and back and forth between client and server. A full table scan is not exactly like that but it was the simplest use case I could think of that looked somewhat similar. On Wed, Aug 29, 2018 at 10:45 PM wrote: > This may suggest an issue with client, either getting the data to the > client or the client itself (although I think there are other performance > related changes you could make). I=E2=80=99m curious what the end goal is= here. Is > this a real world use case? If you are using this type of benchmark to > evaluate the speed of Accumulo, then you will likely not get the same > performance when you apply your data and your real use cases. > > > > *From:* guy sharon > *Sent:* Wednesday, August 29, 2018 3:13 PM > *To:* user@accumulo.apache.org > *Subject:* Re: Accumulo performance on various hardware configurations > > > > hi Mike, > > > > As per Mike Miller's suggestion I started using > org.apache.accumulo.examples.simple.helloworld.ReadData from Accumulo wit= h > debugging turned off and a BatchScanner with 10 threads. I redid all the > measurements and although this was 20% faster than using the shell there > was no difference once I started playing with the hardware configurations= . > > > > Guy. > > > > On Wed, Aug 29, 2018 at 10:06 PM Michael Wall wrote: > > Guy, > > > > Can you go into specifics about how you are measuring this? Are you stil= l > using "bin/accumulo shell -u root -p secret -e "scan -t hellotable -np" | > wc -l" as you mentioned earlier in the thread? As Mike Miller suggested, > serializing that back to the display and then counting 6M entries is goin= g > to take some time. Try using a Batch Scanner directly. > > > > Mike > > > > On Wed, Aug 29, 2018 at 2:56 PM guy sharon > wrote: > > Yes, I tried the high performance configuration which translates to 4G > heap size, but that didn't affect performance. Neither did setting > table.scan.max.memory to 4096k (default is 512k). Even if I accept that t= he > read performance here is reasonable I don't understand why none of the > hardware configuration changes (except going to 48 cores, which made thin= gs > worse) made any difference. > > > > On Wed, Aug 29, 2018 at 8:33 PM Mike Walch wrote: > > Muchos does not automatically change its Accumulo configuration to take > advantage of better hardware. However, it does have a performance profile > setting in its configuration (see link below) where you can select a > profile (or create your own) based on your the hardware you are using. > > > > > https://github.com/apache/fluo-muchos/blob/master/conf/muchos.props.examp= le#L94 > > On Wed, Aug 29, 2018 at 11:35 AM Josh Elser wrote: > > Does Muchos actually change the Accumulo configuration when you are > changing the underlying hardware? > > On 8/29/18 8:04 AM, guy sharon wrote: > > hi, > > > > Continuing my performance benchmarks, I'm still trying to figure out if > > the results I'm getting are reasonable and why throwing more hardware a= t > > the problem doesn't help. What I'm doing is a full table scan on a tabl= e > > with 6M entries. This is Accumulo 1.7.4 with Zookeeper 3.4.12 and Hadoo= p > > 2.8.4. The table is populated by > > org.apache.accumulo.examples.simple.helloworld.InsertWithBatchWriter > > modified to write 6M entries instead of 50k. Reads are performed by > > "bin/accumulo org.apache.accumulo.examples.simple.helloworld.ReadData -= i > > muchos -z localhost:2181 -u root -t hellotable -p secret". Here are the > > results I got: > > > > 1. 5 tserver cluster as configured by Muchos > > (https://github.com/apache/fluo-muchos), running on m5d.large AWS > > machines (2vCPU, 8GB RAM) running CentOS 7. Master is on a separate > > server. Scan took 12 seconds. > > 2. As above except with m5d.xlarge (4vCPU, 16GB RAM). Same results. > > 3. Splitting the table to 4 tablets causes the runtime to increase to 1= 6 > > seconds. > > 4. 7 tserver cluster running m5d.xlarge servers. 12 seconds. > > 5. Single node cluster on m5d.12xlarge (48 cores, 192GB RAM), running > > Amazon Linux. Configuration as provided by Uno > > (https://github.com/apache/fluo-uno). Total time was 26 seconds. > > > > Offhand I would say this is very slow. I'm guessing I'm making some sor= t > > of newbie (possibly configuration) mistake but I can't figure out what > > it is. Can anyone point me to something that might help me find out wha= t > > it is? > > > > thanks, > > Guy. > > > > > > --00000000000074b9c1057498745a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
OK, good news at last! Performance has improved by on= e order of magnitude. The bad news is that I don't know why. As far as = I can tell the only things I changed were table.scan.max.memory and the per= formance profile on Muchos. Both had no effect when I initially tested them= and I doubt that there's a "settling in" period for both of = these so I don't know what it is. But at least now it's clear that = it's possible to get better performance. I'll try to investigate wh= y it happened. Thanks everyone for helping!

Regard= ing my use case: I'm trying to use Accumulo as a graph database. Traver= sing a graph, or several graphs at once, means getting a row (vertex) by ID= , sending it to the client, deciding if it's relevant and then retrievi= ng the neighboring vertices. So lots of reads by ID and back and forth betw= een client and server. A full table scan is not exactly like that but it wa= s the simplest use case I could think of that looked somewhat similar.
<= /div>

On Wed, Aug 29, = 2018 at 10:45 PM <dlmarion@comca= st.net> wrote:

This may suggest an issue with client, = either getting the data to the client or the client itself (although I thin= k there are other performance related changes you could make). I=E2=80=99m = curious what the end goal is here. Is this a real world use case? If you ar= e using this type of benchmark to evaluate the speed of Accumulo, then you = will likely not get the same performance when you apply your data and your = real use cases.

=C2=A0

From: guy sharon <guy.sharon.1977@gmail.com&g= t;
Sent: Wednesday, August 29, 2018 3:13 PM
To: user@accumulo.apach= e.org
Subject: Re: Accumulo performance on various hardware c= onfigurations

=C2=A0<= /p>

hi Mike,

=C2=A0

As per Mike Miller's suggestion I started using org.apache.accumulo= .examples.simple.helloworld.ReadData from Accumulo with debugging turned of= f and a BatchScanner with 10 threads. I redid all the measurements and alth= ough this was 20% faster than using the shell there was no difference once = I started playing with the hardware configurations.

=

=C2=A0

Guy.

= =C2=A0

On Wed, Aug 29, 2018 at 1= 0:06 PM Michael Wall <mjwall@gmail.com> wrote:

Guy,=

=C2=A0

<= p class=3D"MsoNormal">Can you go into specifics about how you are measuring= this?=C2=A0 Are you still using "bin/accumulo shell -u root -p secret -e "scan -t hellotable= -np" | wc -l" as you mentioned earlier in the thread?=C2=A0 As M= ike Miller suggested, serializing that back to the display and then countin= g 6M entries is going to take some time.=C2=A0 Try using a Batch Scanner di= rectly.

= =C2=A0

Mike

=C2=A0

On Wed, Aug = 29, 2018 at 2:56 PM guy sharon <guy.sharon.1977@gmail.com> wrote:

Yes, I tried the high performance configuration which tran= slates to 4G heap size, but that didn't affect performance. Neither did= setting table.scan.max.memory to 4096k (default is 512k). Even if I accept= that the read performance here is reasonable I don't understand why no= ne of the hardware configuration changes (except going to 48 cores, which m= ade things worse) made any difference.

=C2=A0

On Wed, = Aug 29, 2018 at 8:33 PM Mike Walch <mwalch@apache.org> wrote:

=

Muchos does not automatically change its Accumulo configuration t= o take advantage of better hardware. However, it does have a performance=C2= =A0profile setting in its configuration (see link=C2=A0below) where you can= select a profile (or create your own) based on your the hardware you are u= sing.

=C2=A0

https://github.com/apache/fluo-muchos/blob/m= aster/conf/muchos.props.example#L94

On Wed, Aug 29, 2018 at 11:35 AM Josh Elser <elserj@apache.org> wrote:=

Does Muchos actually change the Accumulo configurat= ion when you are
changing the underlying hardware?

On 8/29/18 8:= 04 AM, guy sharon wrote:
> hi,
>
> Continuing my perform= ance benchmarks, I'm still trying to figure out if
> the results= I'm getting are reasonable and why throwing more hardware at
> = the problem doesn't help. What I'm doing is a full table scan on a = table
> with 6M entries. This is Accumulo 1.7.4 with Zookeeper 3.4.1= 2 and Hadoop
> 2.8.4. The table is populated by
> org.apache.= accumulo.examples.simple.helloworld.InsertWithBatchWriter
> modified= to write 6M entries instead of 50k. Reads are performed by
> "= bin/accumulo org.apache.accumulo.examples.simple.helloworld.ReadData -i > muchos -z localhost:2181 -u root -t hellotable -p secret". Here = are the
> results I got:
>
> 1. 5 tserver cluster as co= nfigured by Muchos
> (https://github.com/apache/fluo-muchos), running o= n m5d.large AWS
> machines (2vCPU, 8GB RAM) running CentOS 7. Master= is on a separate
> server. Scan took 12 seconds.
> 2. As abov= e except with m5d.xlarge (4vCPU, 16GB RAM). Same results.
> 3. Splitt= ing the table to 4 tablets causes the runtime to increase to 16
> se= conds.
> 4. 7 tserver cluster running m5d.xlarge servers. 12 seconds.=
> 5. Single node cluster on m5d.12xlarge (48 cores, 192GB RAM), runn= ing
> Amazon Linux. Configuration as provided by Uno
> (https://github.= com/apache/fluo-uno). Total time was 26 seconds.
>
> Offha= nd I would say this is very slow. I'm guessing I'm making some sort=
> of newbie (possibly configuration) mistake but I can't figure= out what
> it is. Can anyone point me to something that might help = me find out what
> it is?
>
> thanks,
> Guy.
&= gt;
>

=
=
--00000000000074b9c1057498745a--