Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of nanhengwu@gmail.com
 designates 209.85.210.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=oWEIo2eewWFfEuIOvVYnmmQX8IAEDaZZCmpASLNXS1/QEZE3JarLsqo2YfyJyBcnxK
         f//k/mGCpCFOZIev7LUr1Ir143qezcvz/YUwwsRFR2P1ex+tzZIMLzmcw6HotR8ECohJ
         wLUj3nbDcaNqsnff1RjgRM3goYzAJriwHNqyY=
MIME-Version: 1.0
In-Reply-To: <AANLkTi=CgNiOBVuE2dyiudu4BH-FCKjJOTv_CHZjXRxz@mail.gmail.com>
References: <AANLkTikYi_RA6UBt1JU40hypaEPAUdVwAxPW+sC5ai9=@mail.gmail.com>
	<AANLkTi=CgNiOBVuE2dyiudu4BH-FCKjJOTv_CHZjXRxz@mail.gmail.com>
Date: Mon, 27 Dec 2010 13:03:46 -0800
Message-ID: <AANLkTinLke_RfKz-4arpT3xZ_CaBAvWvGtha-AAtTTfo@mail.gmail.com>
Subject: Re: Bulk load questions
From: Nanheng Wu <nanhengwu@gmail.com>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for the answers. I will use these as my basis for
investigation. I am using a mapper only job, is it better to use the
HBase client to write to HBase or TableOutputFormat?

On Mon, Dec 27, 2010 at 8:38 AM, Stack <stack@duboce.net> wrote:
> On Mon, Dec 27, 2010 at 1:54 AM, Nanheng Wu <nanhengwu@gmail.com> wrote:
>> I am running some tests to load data from HDFS into HBase in a MR job.
>> I am pretty new to HBase and I have some questions regarding bulk load
>> performance: I have a small cluster with 4 nodes, I set up one node to
>> run Namenode/JobTracker/ZK, and the other three nodes all run
>> TaskTracker/DataNode/HRegion. During my test I am seeing about 1300
>> inserts per second total and it feels kind of slow.
>
> I don't know what your hardware is like but yeah, it sounds kinda slow.
>
>
> My rows are pretty
>> small ~250 bytes. I am wondering if it is a good idea to be running MR
>> on all nodes. Would it be better if I run MR load job on separate
>> nodes?
>
> Well, where do you think the time is being spent? =A0What is holding up
> the job do you think? =A0Is your MR job doing any massaging of the data.
> =A0Do you have many concurrent mappers run at same time on each node?
> Does your MR job do a map and reduce or just a map? =A0Is it the insert
> into hbase that is slow? =A0What do the hbase logs say? =A0Are they
> blocking because they are flushing memory?
>
> Also I observe that one task tracker's CPU usage was twice as
>> high as the other two.
>
> Maybe its the one that is doing the inserting? =A0How many regions in
> your hbase cluster? =A0When you look at hbase UI, is load being spread
> across the hbase cluster or you just hitting one node?
>
> St.Ack
>
> =A0I can't figure out why that is, does that
>> indicate some hot spots in the cluster? I'd really appreciate some
>> ideas, and please let me know if my description is not specific or
>> detailed enough and what other information I can provide to help
>> diagnose the problem. Thanks!
>>
>