hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daisy Zhou <da...@wibidata.com>
Subject Re: Bulk-loading HFiles after table split (on ACL enabled cluster)
Date Fri, 19 Sep 2014 23:48:35 GMT
All right, thank you.  I've modified my client code to chmod while the
bulk-load is running instead, since even if I manually chmod beforehand,
the newly split HFiles need to be chmod'd before the bulk-load can continue.

On Wed, Sep 17, 2014 at 5:28 PM, Matteo Bertozzi <theo.bertozzi@gmail.com>
wrote:

> yeah, in a non secure cluster you have to manually the chmod.
> there was discussion to implement something like the SecureBulkLoadEndPoint
> even for the unsecure setup, but at the moment there is no jira/patch
> available.
> (the SecureBulkLoadEndPoint is basically doing a chmod 777 before starting
> the bulkload)
>
> Matteo
>
>
> On Wed, Sep 17, 2014 at 12:58 PM, Daisy Zhou <daisy@wibidata.com> wrote:
>
> > Thanks for the response, Matteo.
> >
> > My HBase is not a secure HBase, I only have ACL enabled on HDFS.  I did
> try
> > adding the SecureBulkLoadEndpoint coprocessor to my HBase cluster, but I
> > think it does something different, and it didn't help.
> >
> > I normally have to chmod -R a+rwx the hfile directory in order to
> bulk-load
> > them, because the hbase user and current user both need write access.
> Then
> > the newly created split HFiles do not have those same permissions,
> unless I
> > chmod them specifically.  Am I doing something wrong?
> >
> > Daisy
> >
> > On Tue, Sep 16, 2014 at 2:28 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > are you using the SecureBulkLoadEndpoint? that should take care of
> > > permissions
> > > http://hbase.apache.org/book/hbase.secure.bulkload.html
> > >
> > > Matteo
> > >
> > >
> > > On Tue, Sep 16, 2014 at 2:26 PM, Daisy Zhou <daisy@wibidata.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > I can't find mention of this issue on the Jira.  Is it known?  I
> think
> > > that
> > > > if a split of the HFiles is required, LoadIncrementalHFiles should
> > create
> > > > the new HFiles with the correct permissions to be bulk-loaded.
> > Currently
> > > it
> > > > just hangs because the permissions are wrong.
> > > >
> > > > Here is how I reproduce my issue:
> > > >
> > > > On a cluster with ACL enabled, I generate HFiles for a bulk-load,
> then
> > > > *force a table split*, and then attempt to bulk-load the HFiles.  The
> > > > bulk-load hangs (similar to when the hfiles' directory is not
> chown'ed
> > > > properly):
> > > >
> > > > 14/09/15 15:44:41 INFO
> > > > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles: Trying to
> > > > load
> > > >
> > >
> >
> hfile=hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/00000
> > > > first=\x00fs\xC0song-32\x00 last=\xFEI\x99~song-44\x0014/09/15
> > > > 15:44:41 INFO
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles:
> > > > HFile at
> > > >
> > >
> >
> hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/00000
> > > > no longer fits inside a single region. Splitting...14/09/15 15:44:42
> > > > INFO org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles:
> > > > Successfully split into new HFiles
> > > >
> > > >
> > >
> >
> hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/_tmp/kiji.kiji_music.table.songs,1.bottom
> > > > and
> > > >
> > >
> >
> hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/_tmp/kiji.kiji_music.table.songs,1.top14/09/15
> > > > 15:44:42 INFO
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles:
> > > > Split occured while grouping HFiles, retry attempt 1 with 2 files
> > > > remaining to group or split
> > > > 14/09/15 15:44:42 INFO
> > > > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles: Trying to
> > > > load
> > > >
> > >
> >
> hfile=hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/_tmp/kiji.kiji_music.table.songs,1.top
> > > > first=c\xA8\x0D\x81song-9\x00 last=\xFEI\x99~song-44\x0014/09/15
> > > > 15:44:42 INFO
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles:
> > > > Trying to load
> > > >
> > >
> >
> hfile=hdfs://bento:8020/user/daisy/kiji-mr-tutorial/hfile-output/part-r-00000.hfile/B/_tmp/kiji.kiji_music.table.songs,1.bottom
> > > > first=\x00fs\xC0song-32\x00 last=^49\xDEsong-13\x00
> > > >
> > > >
> > > > If I chmod -R 777 the directory and try again, the bulk load
> completes
> > > > successfully.
> > > >
> > > >
> > > > Daisy
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message