Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com
 designates 209.85.213.169 as permitted sender)
MIME-Version: 1.0
Sender: saint.ack@gmail.com
In-Reply-To: <DC472608-AB20-4909-B40D-24B790B16B9E@gbif.org>
References: <DC472608-AB20-4909-B40D-24B790B16B9E@gbif.org>
Date: Tue, 14 Feb 2012 08:14:19 -0800
Message-ID: 
 <CADcMMgEdQmsoFPsdpwcO41cAua77EFO8Yfc85jvt_wg-+303ww@mail.gmail.com>
Subject: Re: strange PerformanceEvaluation behaviour
From: Stack <stack@duboce.net>
To: user@hbase.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Tue, Feb 14, 2012 at 7:56 AM, Oliver Meyn (GBIF) <omeyn@gbif.org> wrote:
> 1) With a command line like 'hbase org.apache.hadoop.hbase.PerformanceEva=
luation randomWrite 10' I see 100 mappers spawned, rather than the expected=
 10. =A0I expect 10 because that's what the usage text implies, and what th=
e javadoc explicitly states - quoting from doMapReduce "Run as many maps as=
 asked-for clients." =A0The culprit appears to be the outer loop in writeIn=
putFile which sets up 10 splits for every "asked-for client" - at least, if=
 I'm reading it right. =A0Is this somehow expected, or is that code leftove=
r from some previous iteration/experiment?
>

Yeah.  I'd expect ten clients, each to its own map, each doing 1M items eac=
h.

Looking at writeInputFile, it seems to be dividing the namespace by
ten so, yeah x10 mappers.

> 2) With that same randomWrite command line above, I would expect a result=
ing table with 10 * (1024 * 1024) rows (so 10485700 =3D roughly 10M rows). =
=A0Instead what I'm seeing is that the randomWrite job reports writing that=
 many rows (exactly) but running rowcounter against the table reveals only =
6549899 rows. =A0A second attempt to build the table produces slightly diff=
erent results (e.g. 6627689). =A0I see a similar discrepancy when using 50 =
instead of 10 clients (~35% smaller than expected). =A0Key collision could =
explain it, but it seems pretty unlikely (given I only need e.g. 10M keys f=
rom a potential 2B).
>

Yeah, I'd think key overlap (print out the span for each mappers or
check that file written by writeInputFile).

Your clocks are all in sync?

St.Ack