Mailing-List: contact hbase-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of irfan.ma@gmail.com designates
 209.85.210.185 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=date:from:to:message-id:in-reply-to:subject:mime-version
         :content-type:content-transfer-encoding;
        b=KPOp1VFR95jMHwg3y8ccEypL/o/ev/6VWb9K6TxmvViq1rBlFXQl9vaKC4VIeEqrcR
         weltq2M5hI3B2i7f7amYrpTENwDMMzDR2sLqki2nZQZJFp5CLRFzY8K1NdKojrTpwTAU
         l/Pin1aybZSapI7/qIoRY3mnOmBcbTmEoyfME=
Date: Mon, 6 Jul 2009 14:24:43 -0400 (EDT)
From: Irfan Mohammed <irfan.ma@gmail.com>
To: hbase-dev@hadoop.apache.org
Message-ID: <31211275.2901246904679765.JavaMail.irfan@damascus>
In-Reply-To: <7c962aed0907061114s19ff8c29w8fb070f40455712f@mail.gmail.com>
Subject: Re: performance help
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Input is 1 file. 

These are 4 different tables "txn_m1", "txn_m2", "txn_m3", "txn_m4". To me, it looks like it is always doing 1 region per table and these tables are always on different regionservers. I never seen the same table on different regionservers. Does that sound right?

----- Original Message -----
From: "stack" <stack@duboce.net>
To: hbase-dev@hadoop.apache.org
Sent: Monday, July 6, 2009 2:14:43 PM GMT -05:00 US/Canada Eastern
Subject: Re: performance help

On Mon, Jul 6, 2009 at 11:06 AM, Irfan Mohammed <irfan.ma@gmail.com> wrote:

> I am working on writing to HDFS files. Will update you by end of day today.
>
> There are always 10 concurrent mappers running. I keep setting the
> setNumMaps(5) and also the following properties in mapred-site.xml to 3 but
> still end up running 10 concurrent maps.
>


Is your input ten files?


>
> There are 5 regionservers and the online regions are as follows :
>
> m1 : -ROOT-,,0
> m2 : txn_m1,,1245462904101
> m3 : txn_m4,,1245462942282
> m4 : txn_m2,,1245462890248
> m5 : .META.,,1
>     txn_m3,,1245460727203
>


So, that looks like 4 regions from table txn?

So thats about 1 region per regionserver?


> I have setAutoFlush(false) and also writeToWal(false) with the same
> behaviour.
>

If you did above and still takes 10 minutes, then that would seem to rule
out hbase (batching should have big impact on uploads and then setting
writeToWAL to false, should double throughput over whatever you were seeing
previous).

St.Ack