From user-return-7351-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org Tue Aug 28 15:06:38 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C2CB9180621 for ; Tue, 28 Aug 2018 15:06:37 +0200 (CEST) Received: (qmail 59461 invoked by uid 500); 28 Aug 2018 13:06:36 -0000 Mailing-List: contact user-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@accumulo.apache.org Delivered-To: mailing list user@accumulo.apache.org Received: (qmail 59451 invoked by uid 99); 28 Aug 2018 13:06:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Aug 2018 13:06:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 6CFD3C06AD for ; Tue, 28 Aug 2018 13:06:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.889 X-Spam-Level: * X-Spam-Status: No, score=1.889 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id CV6oAdj_UbeF for ; Tue, 28 Aug 2018 13:06:35 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 1AA0C5F3B8 for ; Tue, 28 Aug 2018 13:06:35 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id c22-v6so1379826iob.1 for ; Tue, 28 Aug 2018 06:06:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=og26cTPYtpPZyo0NXbnpujEEjCjBSh0d/27dvCYlUxk=; b=Gq3vMifqQaWvTH/MB+hkG+Crxf1P8DD7vX8QY3C+4zJQwE1BKD96o70Ho6688jpyr3 YJatXTQsQEj4k4ufad/nhSkPBpaL6tIAj0C0XaWUdK1Ar4czEYLfdMzwrO62LowTGhtX 47okLJznDXqLmSNC//cdhY/lD28v7UFO/oDFmwyEMxd964VqJMfLGbRDFJy04z3aEIgJ GakCMMAIaG72bJAOyfv7Us0mYrsAewYFLKZS1PN0flEajMN0MQDiGc+xxQcK6K7DF2lr OIxsttNrhsVl9bWePTcUsho0LcpaN4sbxZFbijMuH8wlYGsEohlLIhAn5aFXc+9S+nny 687w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=og26cTPYtpPZyo0NXbnpujEEjCjBSh0d/27dvCYlUxk=; b=WvOygT5eztyI8n7g7nzUMs7Hst2KRVjHdgMQtt4UfPi6baNwMun18B2j71y/Z6mKBV FMoLYc6A3ny3nDbvWd5fjhKmrCIDzPd4pswVQ+3Pu3IruM2qHlTd0SqrmzpvCtb5vFby P+8sfKhOPA/KkgYX+QEisA6kpuf7sJkNcW94LT9VGfesmO/RJ2/Iibcpb0WjoWGgXm+G zyvWwOAmFMi4IPFsRlZ0yW3daomv+lb5iIxYrb2YNV6eQafzfak5plVXRqw0GpmNOyZB 0eH5P/8gjJNWx+kIiRhnrOKdzh/5NV0aM02htmOFS+ODnTzQb0Ffg4Lo913YBRzMEUP9 vaIg== X-Gm-Message-State: APzg51AQWJ9RbhxbNYmQgjSKc1d8hjiwxzJmuGqgJPfGQMZVoQ8QHJ3q p7q0y9U4AEARFauPQsyrHUK704l0jEIQwWIET0O1 X-Google-Smtp-Source: ANB0Vdbs9lfWmXqNvjoZnwCydpbWFSWlLnlgFiHXXwRROHhyXxN/ZcUENHzSYoGQw/DrDQg2sI1FbUVMPXP8PWBqQ08= X-Received: by 2002:a6b:3084:: with SMTP id w126-v6mr1143226iow.223.1535461588301; Tue, 28 Aug 2018 06:06:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Michael Wall Date: Tue, 28 Aug 2018 09:06:16 -0400 Message-ID: Subject: Re: benchmarking To: user@accumulo.apache.org Content-Type: multipart/alternative; boundary="000000000000897eb905747e82f0" --000000000000897eb905747e82f0 Content-Type: text/plain; charset="UTF-8" Hi Guy, I can't say if that is reasonable without more info. How are you running datanodes, namenodes and zookeepers? Also, what are the JVM options for each process? Can you share your dockerfiles? What OS are you on? How much of your OS can Docker take? What is the data in your benchmark_table? Like Sean mentioned, running multiple tservers will help to distribute the load. You may or may not have headroom. It is possible to run multiple tservers on the same host, even without docker. Like Jeremy mentioned, I have seem better performance than you are getting on a single node cluster but I usually use the standalone mini accumulo for that, not a full cluster setup with HDFS. Mike On Tue, Aug 28, 2018 at 2:59 AM guy sharon wrote: > hi Mike, > > Thanks for the links. > > My current setup is a 4 node cluster (tserver, master, gc, monitor) > running on Alpine Docker containers on a laptop with an i7 processor (8 > cores) with 16GB of RAM. As an example I'm running a count of all entries > for a table with 6.3M entries with "accumulo shell -u root -p secret -e > "scan -t benchmark_table -np" | wc -l" and it takes 43 seconds. Not sure if > this is reasonable or not. Seems a little slow to me. What do you think? > > BR, > Guy. > > > > > On Mon, Aug 27, 2018 at 4:43 PM Michael Wall wrote: > >> Hi Guy, >> >> Here are a couple links I found. Can you tell us more about your setup >> and what you are seeing? >> >> https://accumulo.apache.org/papers/accumulo-benchmarking-2.1.pdf >> https://www.youtube.com/watch?v=Ae9THpmpFpM >> >> Mike >> >> >> On Sat, Aug 25, 2018 at 5:09 PM guy sharon >> wrote: >> >>> hi, >>> >>> I've just started working with Accumulo and I think I'm experiencing >>> slow reads/writes. I'm aware of the recommended configuration. Does anyone >>> know of any standard benchmarks and benchmarking tools I can use to tell if >>> the performance I'm getting is reasonable? >>> >>> >>> --000000000000897eb905747e82f0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Guy,

I can't say if that is reas= onable without more info.=C2=A0 How are you running datanodes, namenodes an= d zookeepers?=C2=A0 Also, what are the JVM options for each process?=C2=A0 = Can you share your dockerfiles?=C2=A0 What OS are you on?=C2=A0 How much of= your OS can Docker take?=C2=A0 What is the data in your benchmark_table?
Like Sean mentioned, running multiple tservers will help = to distribute the load.=C2=A0 You may or may not have headroom.=C2=A0 It is= possible to run multiple tservers on the same host, even without docker.

Like Jeremy mentioned, I have seem better performan= ce than you are getting on a single node cluster but I usually use the stan= dalone mini accumulo for that, not a full cluster setup with HDFS.

Mike

On Tue, Aug 28, 2018 at 2:59 AM guy sharon <guy.sharon.1977@gmail.com> wrote:
hi Mike,

Thanks for the links.

My current set= up is a 4 node cluster (tserver, master, gc, monitor) running on Alpine Doc= ker containers on a laptop with an i7 processor (8 cores) with 16GB of RAM.= As an example I'm running a count of all entries for a table with 6.3M= entries with "accumulo shell -u root -p secret=C2=A0 -e "scan -t= benchmark_table -np" | wc -l" and it takes 43 seconds. Not sure = if this is reasonable or not. Seems a little slow to me. What do you think?=

BR,
Guy.


<= /div>


O= n Mon, Aug 27, 2018 at 4:43 PM Michael Wall <mjwall@apache.org> wrote:
Hi Guy,

Here = are a couple links I found.=C2=A0 Can you tell us more about your setup and= what you are seeing?


=
Mike


On Sat, Aug 25, 2018 at 5:09 PM guy sharon <guy.sharon.1977@gmail.com> wro= te:
hi,
=

I've just started working with Accumulo and I think= I'm experiencing slow reads/writes. I'm aware of the recommended c= onfiguration. Does anyone know of any standard benchmarks and benchmarking = tools I can use to tell if the performance I'm getting is reasonable?


--000000000000897eb905747e82f0--