Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=XgFp3nMBjVtIW9DMXtHFztrsKFNDzeBSZx9epGDwF4O2fI07ecS6YQ1M0tDD+18AcXQUaxvSqr6Qc/XpR1fenHWYDGTK7Svf2eV6PYY9sO5tz2LIY4rx/o/1emvfXL7zS2Ku56DqzmNnVKM/OvZoUvM5L1Z23OrcIemUoyo7U1E=;
References: <loom.20121023T173704-953@post.gmane.org>
 <CAPcDmSvHU2CTOGZZbSs4NBjEnUAMD8+ggg5R_T=Lh9ewnq+JAw@mail.gmail.com>
 <loom.20121023T191245-129@post.gmane.org>
 <CAPcDmSu7p3HnU24+9xKVASNQj5UEZhw=DNwWKdhJrb2OCHUpcQ@mail.gmail.com>
 <CAGngS9eTXdMku+DSbk0GHXTHZuGWyAWiPgjshZMzoU09QhTBYA@mail.gmail.com>
Message-ID: <1351138230.41525.YahooMailNeo@web121703.mail.ne1.yahoo.com>
Date: Wed, 24 Oct 2012 21:10:30 -0700 (PDT)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: Hbase import Tsv performance (slow import)
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CAGngS9eTXdMku+DSbk0GHXTHZuGWyAWiPgjshZMzoU09QhTBYA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1001534069-418957583-1351138230=:41525"

--1001534069-418957583-1351138230=:41525
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

This is good advice Kevin we should add this to the HBase Reference Guide.=
=0A=0A=0A=0A________________________________=0A From: Kevin O'dell <kevin.o=
dell@cloudera.com>=0ATo: user@hbase.apache.org =0ASent: Tuesday, October 23=
, 2012 10:47 AM=0ASubject: Re: Hbase import Tsv performance (slow import)=
=0A =0AYou will want to make sure your table is pre-split.=A0 Also Import d=
oes=0Aputs, so you will want to make sure you are not flushing and blocking=
=0Aby raising your memstore, Hlog, and blocking count.=A0 This can greatly=
=0Aimprove your write speeds.=A0 I usually do a 256MB memstore(you can=0Alo=
wer it later if it is not a heavy writes table), 512MB Hlog(same=0Athing, y=
ou can lower back to default), and then raise the storefile=0Ablocking coun=
t to about 100.=0A=0AOn Tue, Oct 23, 2012 at 1:32 PM, Nicolas Liochon <nkey=
wal@gmail.com> wrote:=0A> Thanks, checking the schema itself is still inter=
esting (cf. the link sent)=0A> As well, with 3 machines and a replication f=
actor of 3, all the machines=0A> are used during a write. As HBase writes a=
ll entries into a write-ahead-log=0A> for safety, the number of writes is a=
lso doubled. So may be your machine is=0A> just dying under the load. Anywa=
y, here your cluster is going at the speed=0A> of the least powerful machin=
e, and this machine has a workload multiplied=0A> by 6 compared to a single=
 machine config (i.e. just writing a file locally).=0A>=0A> On Tue, Oct 23,=
 2012 at 7:13 PM, Nick maillard <=0A> nicolas.maillard@fifty-five.com> wrot=
e:=0A>=0A>> Thanks for the help!=0A>>=0A>> My conf files are : Hadoop:=0A>>=
 hdfs-site=0A>>=0A>> <configuration>=0A>>=A0 <property>=0A>>=A0  <name>dfs.=
replication</name>=0A>>=A0  <value>3</value=0A>>=A0  <description>Default b=
lock replication.=0A>>=A0  The actual number of replications can be specifi=
ed when the file is=0A>> created.=0A>>=A0  The default is used if replicati=
on is not specified in create time.=0A>>=A0  </description>=0A>> </property=
>=0A>> <property>=0A>>=A0  <name>dfs.data.dir</name>=0A>>=A0  <value>/home/=
runner/app/hadoop/dfs/data</value>=0A>>=A0  <description>Default block repl=
ication.=0A>>=A0  The actual number of replications can be specified when t=
he file is=0A>> created.=0A>>=A0  The default is used if replication is not=
 specified in create time.=0A>>=A0  </description>=0A>> </property>=0A>> <p=
roperty>=0A>>=A0 =A0 =A0 =A0  <name>dfs.datanode.max.xcievers</name>=0A>>=
=A0 =A0 =A0 =A0  <value>4096</value>=0A>>=A0 =A0 =A0  </property>=0A>> </co=
nfiguration>=0A>>=0A>>=0A>> Mapred-site.xml=0A>>=0A>> <configuration>=0A>>=
=A0 <property>=0A>>=A0  <name>mapred.job.tracker</name>=0A>>=A0  <value>mas=
ter:54311</value>=0A>>=A0  <description>The host and port that the MapReduc=
e job tracker runs=0A>>=A0  at.=A0 If "local", then jobs are run in-process=
 as a single map=0A>>=A0  and reduce task.=0A>>=A0  </description>=0A>> </p=
roperty>=0A>> <property>=0A>>=A0  <name>mapred.tasktracker.map.tasks.maximu=
m</name>=0A>>=A0  <value>14</value>=0A>>=A0  <description>The maximum numbe=
r of map tasks that will be run=0A>>=A0  simultaneously by a task tracker.=
=0A>>=A0  </description>=0A>> </property>=0A>>=0A>> <property>=0A>>=A0  <na=
me>mapred.tasktracker.reduce.tasks.maximum</name>=0A>>=A0  <value>14</value=
>=0A>>=A0  <description>The maximum number of reduce tasks that will be run=
=0A>>=A0  simultaneously by a task tracker.=0A>>=A0  </description>=0A>> </=
property>=0A>> <property>=0A>> <name>mapred.child.java.opts</name>=0A>>=A0 =
 <value>-Xmx400m</value>=0A>>=A0  <description>Java opts for the task track=
er child processes.=0A>>=A0  The following symbol, if present, will be inte=
rpolated: @taskid@ is=0A>> replaced=0A>>=A0  by current TaskID. Any other o=
ccurrences of '@' will go unchanged.=0A>>=A0  For example, to enable verbos=
e gc logging to a file named for the taskid=0A>> in=0A>>=A0  /tmp and to se=
t the heap maximum to be a gigabyte, pass a 'value' of:=0A>>=A0 =A0 =A0 =A0=
  -Xmx1024m -verbose:gc -Xloggc:/tmp/@taskid@.gc=0A>>=0A>>=A0  The configur=
ation variable mapred.child.ulimit can be used to control the=0A>>=A0  maxi=
mum virtual memory of the child processes.=0A>>=A0  </description>=0A>> </p=
roperty>=0A>> </configuration>=0A>>=0A>>=0A>> core-site.xml=0A>>=0A>> <conf=
iguration>=0A>>=A0 <property>=0A>>=A0  <name>hadoop.tmp.dir</name>=0A>>=A0 =
 <value>/home/runner/app/hadoop/tmp</value>=0A>>=A0  <description>A base fo=
r other temporary directories.</description>=0A>> </property>=0A>>=0A>> <pr=
operty>=0A>>=A0  <name>fs.default.name</name>=0A>>=A0  <value>hdfs://master=
:54310</value>=0A>>=A0  <description>The name of the default file system.=
=A0 A URI whose=0A>>=A0  scheme and authority determine the FileSystem impl=
ementation.=A0 The=0A>>=A0  uri's scheme determines the config property (fs=
.SCHEME.impl) naming=0A>>=A0  the FileSystem implementation class.=A0 The u=
ri's authority is used to=0A>>=A0  determine the host, port, etc. for a fil=
esystem.</description>=0A>> </property>=0A>>=0A>>=0A>> For Hbase:=0A>> hbas=
e-site:=0A>> <configuration>=0A>>=A0 <property>=0A>>=A0 =A0  <name>hbase.ro=
otdir</name>=0A>>=A0 =A0  <value>hdfs://master:54310/hbase</value>=0A>>=A0 =
</property>=0A>>=A0  <property>=0A>>=A0 =A0  <name>hbase.cluster.distribute=
d</name>=0A>>=A0 =A0  <value>true</value>=0A>>=A0 =A0  <description>The mod=
e the cluster will be in. Possible values are=0A>>=A0 =A0 =A0  false: stand=
alone and pseudo-distributed setups with managed=0A>> Zookeeper=0A>>=A0 =A0=
 =A0  true: fully-distributed with unmanaged Zookeeper Quorum (see=0A>> hba=
se-env.sh)=0A>>=A0 =A0  </description>=0A>>=A0  </property>=0A>> <property>=
=0A>>=A0 =A0 =A0 =A0  <name>hbase.zookeeper.property.clientPort</name>=0A>>=
=A0 =A0 =A0 =A0  <value>2222</value>=0A>>=A0 =A0  </property>=0A>>=A0 =A0  =
<property>=0A>>=A0 =A0 =A0 =A0  <name>hbase.zookeeper.quorum</name>=0A>>=A0=
 =A0 =A0 =A0  <value>ks25937.kimsufi.com</value>=0A>>=A0 =A0  </property>=
=0A>>=A0 =A0  <property>=0A>>=A0 =A0 =A0 =A0  <name>hbase.zookeeper.propert=
y.dataDir</name>=0A>>=A0 =A0 =A0 =A0  <value>/home/runner/hbase/hbase-0.94.=
2/tmp</value>=0A>>=A0 =A0  </property>=0A>> </configuration>=0A>>=0A>>=0A>>=
=0A>>=0A>> I am currently running import and looking at the logs to try and=
 understand=0A>> This seems definitely phishy:=0A>>=0A>> 2012-10-23 18:39:4=
9,107 INFO org.apache.hadoop.mapred.TaskTracker:=0A>> attempt_201210231145_=
0010_m_000041_0 0.21332978%=0A>> 2012-10-23 18:39:50,363 INFO org.apache.ha=
doop.mapred.TaskTracker:=0A>> attempt_201210231145_0010_m_000028_0 0.209368=
84%=0A>> 2012-10-23 18:49:38,098 INFO org.apache.hadoop.mapred.TaskTracker:=
=0A>> attempt_201210231145_0010_m_000030_0: Task=0A>> attempt_201210231145_=
0010_m_000030_0=0A>> failed to report status for 602 seconds. Killing!=0A>>=
 2012-10-23 18:49:38,116 INFO org.apache.hadoop.mapred.TaskTracker: Process=
=0A>> Thread Dump: lost task=0A>> 90 active threads=0A>> Thread 742 (proces=
s reaper):=0A>>=A0  State: RUNNABLE=0A>>=A0  Blocked count: 0=0A>>=A0  Wait=
ed count: 0=0A>>=A0  Stack:=0A>>=A0 =A0  java.lang.UNIXProcess.waitForProce=
ssExit(Native Method)=0A>>=A0 =A0  java.lang.UNIXProcess.access$200(UNIXPro=
cess.java:54)=0A>>=A0 =A0  java.lang.UNIXProcess$3.run(UNIXProcess.java:174=
)=0A>>=0A>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExe=
cutor.java:1110)=0A>>=0A>> java.util.concurrent.ThreadPoolExecutor$Worker.r=
un(ThreadPoolExecutor.java:603)=0A>>=A0 =A0  java.lang.Thread.run(Thread.ja=
va:722)=0A>> Thread 740 (process reaper):=0A>>=A0  State: RUNNABLE=0A>>=A0 =
 Blocked count: 0=0A>>=A0  Waited count: 0=0A>>=A0  Stack:=0A>>=A0 =A0  jav=
a.lang.UNIXProcess.waitForProcessExit(Native Method)=0A>>=A0 =A0  java.lang=
.UNIXProcess.access$200(UNIXProcess.java:54)=0A>>=A0 =A0  java.lang.UNIXPro=
cess$3.run(UNIXProcess.java:174)=0A>>=0A>>=0A=0A=0A=0A-- =0AKevin O'Dell=0A=
Customer Operations Engineer, Cloudera
--1001534069-418957583-1351138230=:41525--