Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of konstt2000@gmail.com
 designates 74.125.82.42 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1F39959E-164A-45A8-B621-BCBCC3126A8B@gmail.com>
References: 
 <CAE6kwsOF4km=ggC+Us_5qVj4RvM5E7_ed-5Twu2hzBTJX0A7MA@mail.gmail.com>
	<1F39959E-164A-45A8-B621-BCBCC3126A8B@gmail.com>
Date: Mon, 14 Apr 2014 14:41:38 +0200
Message-ID: 
 <CAE6kwsPm7tozWbL08U7UiedmsCad+wCaSAevZ8NzKpF7rxsfzA@mail.gmail.com>
Subject: Re: How to generate a large dataset quickly.
From: Guillermo Ortiz <konstt2000@gmail.com>
To: user@hbase.apache.org
Content-Type: multipart/alternative; boundary=f46d041825803108c604f7000139

--f46d041825803108c604f7000139
Content-Type: text/plain; charset=UTF-8

I'm using. 0.94.6-cdh4.4.0,

I use the bulkload:
FileInputFormat.addInputPath(job, new Path(INPUT_FOLDER));
FileOutputFormat.setOutputPath(job, hbasePath);
HTable table = new HTable(jConf, HBASE_TABLE);
HFileOutputFormat.configureIncrementalLoad(job, table);

It seems that it takes really long time when it starts to execute the Puts
to HBase in the reduce phase.


2014-04-14 14:35 GMT+02:00 Ted Yu <yuzhihong@gmail.com>:

> Which hbase release did you run mapreduce job ?
>
> Cheers
>
> On Apr 14, 2014, at 4:50 AM, Guillermo Ortiz <konstt2000@gmail.com> wrote:
>
> > I want to create a large dateset for HBase with different versions and
> > number of rows. It's about 10M rows and 100 versions to do some
> benchmarks.
> >
> > What's the fastest way to create it?? I'm generating the dataset with a
> > Mapreduce of 100.000rows and 10verions. It takes 17minutes and size
> around
> > 7Gb. I don't know if I could do it quickly. The bottleneck is when
> > MapReduces write the output and when transfer the output to the Reduces.
>

--f46d041825803108c604f7000139--