hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuja Rehman <shujamug...@gmail.com>
Subject Re: Bulk Load Sample Code
Date Mon, 15 Nov 2010 08:24:37 GMT
If HRegionPartitioner works correctly then what is the use of
configureIncrementalLoad() as discussed here

http://hbase.apache.org/docs/r0.89.20100621/bulk-loads.html

and why this link did not discuss about HRegionPartitioner? Where is the
documentaion of HRegionPartitioner and in which case we give preference to
HRegionPartitioner  over configureIncrementalLoad(). What is the difference
in both?


On Sat, Nov 13, 2010 at 5:12 AM, Todd Lipcon <todd@cloudera.com> wrote:

> I'm surprised that HRegionPartitioner works correctly for incremental load.
> It definitely won't work if the regions are also shifting during the MR
> job.
>
> Thanks
> -Todd
>
> On Fri, Nov 12, 2010 at 10:10 AM, Adam Phelps <amp@opendns.com> wrote:
>
> > On 11/10/10 11:57 AM, Stack wrote:
> >
> >> On Wed, Nov 10, 2010 at 11:53 AM, Shuja Rehman<shujamughal@gmail.com>
> >>  wrote:
> >>
> >>> oh! I think u have not read the full post. The essay has 3 paragraphs
>  :)
> >>>
> >>> *Should I need to add the following line also
> >>>
> >>>  job.setPartitionerClass(TotalOrderPartitioner.class);
> >>>
> >>>
> >> You need to specify other than default partitioner so yes, above seems
> >> necessary (Be aware that if only one reducer, all may appear to work
> >> though your partitioner is bad... its when you have multiple reducers
> >> that bad partitioner will show).
> >>
> >
> > I skimmed over this thread as we've been using LoadIncrementalHFiles to
> > load the output of our MR jobs, however it looks like we're using
> > HRegionPartitioner rather than TotalOrderPartitioner.  The current code
> is
> > definitely working, however the page regarding bulk loads that was posted
> > earlier implies that TotalOrderPartitioner is best for efficiency.  What
> is
> > the difference between the two?
> >
> > - Adam
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message