Return-Path: <dev-return-16754-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id 381C4200BD1
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Mon, 28 Nov 2016 16:48:36 +0100 (CET)
Received: by cust-asf.ponee.io (Postfix)
	id 36B35160B0D; Mon, 28 Nov 2016 15:48:36 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id 7E679160B00
	for <archive-asf-public@cust-asf.ponee.io>; Mon, 28 Nov 2016 16:48:35 +0100 (CET)
Received: (qmail 39246 invoked by uid 500); 28 Nov 2016 15:48:34 -0000
Mailing-List: contact dev-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@accumulo.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@accumulo.apache.org>
List-Post: <mailto:dev@accumulo.apache.org>
List-Id: <dev.accumulo.apache.org>
Reply-To: dev@accumulo.apache.org
Delivered-To: mailing list dev@accumulo.apache.org
Delivered-To: moderator for dev@accumulo.apache.org
Received: (qmail 35042 invoked by uid 99); 28 Nov 2016 14:39:55 -0000
X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org
X-Spam-Flag: NO
X-Spam-Score: 1.564
X-Spam-Level: *
X-Spam-Status: No, score=1.564 tagged_above=-999 required=6.31
	tests=[FREEMAIL_ENVFROM_END_DIGIT=0.25, RCVD_IN_DNSWL_NONE=-0.0001,
	URIBL_BLOCKED=0.001, URI_HEX=1.313] autolearn=disabled
Date: Mon, 28 Nov 2016 07:39:46 -0700 (MST)
From: hujs <1465039901@qq.com>
To: dev@accumulo.apache.org
Message-ID: <1480343986594-19319.post@n5.nabble.com>
In-Reply-To: <582F2E74.3050509@gmail.com>
References: <tencent_3DF195E3670FC4CB614817AB@qq.com> <582F2D6A.1060209@gmail.com> <582F2E74.3050509@gmail.com>
Subject: Re: Write or Ingest bottleneck
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
archived-at: Mon, 28 Nov 2016 15:48:36 -0000

Sorry, the post was wrong, these days the network is not good, just to see,
figure in the "accumulo balance" this post can be seen. I'm tuning the
accumulo ingest rate now. I do not know what is the reason to limit the
accumulo ingest rate. I mainly used
Here are a few ways:
=C2=A0 1, per-splite, I through the custom j, n points will be divided into=
 three
tablets, each tserver one.
=C2=A0 2, adjust table.file.max, tserver.compaction.minor.concurrent.max an=
d
table.durability =3D flush
=C2=A0 3, by simulating multiple clients, with multiple Bathwriter send dat=
a.
=C2=A0 4, the opening of the native map
=C2=A0 5, configure the tserver's accumuo-site.xml and accumulo-env.sh memo=
ry
settings to 2g.
=C2=A0 However, ingest rate did not improve. My cluster is far from reachin=
g the
load: cpu utilization of less than 1/8, memory usage less than 1/5, IO 8-14=
%
wa, with start-ingest.sh test ingest rate, but the average ingest rate less
than 18 Million entry / s. My cluster each machine configuration: cpu 8,32 =
g
Me. I would like to know what the impact ingest rate,
How do I tune my cluster?
Using the accumuluo 1.7.1 version.The accumulo-site.xml configuration is as
follows:

<configuration>
 =20

  <property>
    <name>instance.volumes</name>
    <value>hdfs://master12:9000/accumulo</value>
    <description>comma separated list of URIs for volumes. example:
hdfs://localhost:9000/accumulo</description>
  </property>

  <property>
    <name>instance.zookeeper.host</name>
    <value>master12:2181,slave13:2181,slave10:2181,slave11:2181</value>
    <description>comma separated list of zookeeper servers</description>
  </property>

  <property>
    <name>logger.dir.walog</name>
    <value>walogs</value>
    <description>The property only needs to be set if upgrading from 1.4
which used to store write-ahead logs on the local
      filesystem. In 1.5 write-ahead logs are stored in DFS.  When 1.5 is
started for the first time it will copy any 1.4
      write ahead logs into DFS.  It is possible to specify a
comma-separated list of directories.
    </description>
  </property>

  <property>
    <name>instance.secret</name>
    <value>DEFAULT</value>
    <description>A secret unique to a given instance that all servers must
know in order to communicate with one another.
      Change it before initialization. To
      change it later use ./bin/accumulo
org.apache.accumulo.server.util.ChangeSecret --old [oldpasswd] --new
[newpasswd],
      and then update this file.
    </description>
  </property>

  <property>
    <name>tserver.memory.maps.max</name>
    <value>2G</value>
  </property>

  <property>
    <name>tserver.memory.maps.native.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>tserver.cache.data.size</name>
    <value>128M</value>
  </property>

  <property>
    <name>tserver.cache.index.size</name>
    <value>128M</value>
  </property>

  <property>
    <name>trace.token.property.password</name>
   =20
    <value>123456</value>
  </property>



  <property>
    <name>trace.user</name>
    <value>root</value>
  </property>

  <property>
    <name>tserver.sort.buffer.size</name>
    <value>500M</value>
  </property>

  <property>
    <name>tserver.walog.max.size</name>
    <value>2G</value>
  </property>

=20
 <property>
    <name>tserver.wal.blocksize</name>
    <value>2G</value>
  </property>
<property>
    <name>tserver.mutation.queue.max</name>
    <value>4M</value>
  </property>
<property>
    <name>tserver.compaction.major.concurrent.max</name>

    <value>8</value>
  </property>
<property>
    <name>tserver.compaction.minor.concurrent.max</name>

    <value>8</value>
  </property>

  <property>
    <name>general.classpaths</name>

    <value>
     =20
      $ACCUMULO_HOME/lib/accumulo-server.jar,
      $ACCUMULO_HOME/lib/accumulo-core.jar,
      $ACCUMULO_HOME/lib/accumulo-start.jar,
      $ACCUMULO_HOME/lib/accumulo-fate.jar,
      $ACCUMULO_HOME/lib/accumulo-proxy.jar,
      $ACCUMULO_HOME/lib/[^.].*.jar,
     =20
      $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
     =20
      $HADOOP_CONF_DIR,
     =20
      $HADOOP_PREFIX/share/hadoop/common/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/common/lib/(?!slf4j)[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/hdfs/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/mapreduce/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/yarn/[^.].*.jar,
      $HADOOP_PREFIX/share/hadoop/yarn/lib/jersey.*.jar,
  =20
    </value>
    <description>Classpaths that accumulo checks for updates and class
files.</description>
  </property>
</configuration>
                                                                           =
     =20
Think you dear josh Elser;




--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/=
Write-or-Ingest-bottleneck-tp19255p19319.html
Sent from the Developers mailing list archive at Nabble.com.