hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: modify prePut() behavior
Date Sat, 08 Feb 2014 03:51:05 GMT
In region server status screen, Coprocessors currently loaded by the
regionserver are shown.

FYI


On Fri, Feb 7, 2014 at 7:22 PM, ameet c <ameet.chaubal@gmail.com> wrote:

> Thanks Ted,
>
> finally figured what the problem was, my oversight.
> After poring over logs from individual region servers, it was much more
> straightforward.
> Due to permissions of 640 on the jar file that I was deploying to HDFS, the
> class was not getting loaded.
> Once I made the permissions 777, The region was able to load and run the
> class.
> and like a promise, bypass() worked fine.
>
> Thank you for persisting Ted...
>
>
>
>
> On Fri, Feb 7, 2014 at 10:12 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > If you can capture your scenario in a unit test, I will investigate.
> >
> > Thanks
> >
> >
> > On Fri, Feb 7, 2014 at 5:30 PM, ameet c <ameet.chaubal@gmail.com> wrote:
> >
> > > nope, at this point, I just have a basic test table and my prePut() has
> > > just one line e.bypass()
> > > I am just inserting a simple row with one column.
> > > I am expecting nothing to be inserted in the table when I do a scan.
> > > But that's not what I see.
> > > all puts are going through.
> > > Is this the correct behavior?
> > >
> > >
> > >
> > > On Fri, Feb 7, 2014 at 6:59 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > bq. I have disabled all processing in my observer
> > > >
> > > > Was it possible that there was other modification going on (such as
> > > Append)
> > > > ?
> > > >
> > > > Cheers
> > > >
> > > >
> > > > On Fri, Feb 7, 2014 at 11:26 AM, aa a <ameet.chaubal@gmail.com>
> wrote:
> > > >
> > > > > Here is my pre job run result,
> > > > >
> > > > > hbase(main):035:0> get 'subsrptn','8322910300|111021583220'
> > > > > COLUMN                               CELL
> > > > >
> > > > >  all_cf:custId                       timestamp=1391797558233,
> > > > > value=111021583220
> > > > >
> > > > >  all_cf:dataDt                       timestamp=1391797558233,
> > > > > value=2013-11-14
> > > > >
> > > > >  all_cf:mktCd                        timestamp=1391797558233,
> > value=hcl
> > > > >
> > > > >  all_cf:rplctLogSeqNbr               timestamp=1391797558233,
> > value=100
> > > > >
> > > > >  all_cf:rplctOprnType                timestamp=1391797558233,
> > > > value=UPDATE
> > > > >
> > > > >  all_cf:subsrptnNbrTxt               timestamp=1391797558233,
> > > > > value=8322910300
> > > > >
> > > > >  all_cf:subsrptnStsCd                timestamp=1391797558233,
> > > value=XXXX
> > > > >
> > > > >  all_cf:subsrptnStsRsnCd             timestamp=1391797558233,
> > > value=SOCHI
> > > > >
> > > > > 8 row(s) in 0.0570 seconds
> > > > >
> > > > > for now, I have disabled all processing in my observer: by simply
> > > having
> > > > > this,
> > > > > public void prePut(ObserverContext<RegionCoprocessorEnvironment>
e,
> > > > > Put put, WALEdit edit, boolean writeToWAL) throws IOException {
> > > > > e.bypass();
> > > > > }
> > > > > What this means is that when I try to do a put, it should fail,
> > right?
> > > > >
> > > > > my job is comprised of HDFS source and HBase sink via,
> > > > > TableMapReduceUtil.initTableReducerJob( Statics.TABLE_NAME, null,
> > job);
> > > > > ---
> > > > > so the table puts are taken care of by TableMapReduceUtil.
> > > > >
> > > > > When I run my job, the "get" returns this:
> > > > > hbase(main):036:0> get 'subsrptn','8322910300|111021583220' COLUMN
> > CELL
> > > > ...
> > > > > ..
> > > > > [ omitted for brevity ]
> > > > > all_cf:subsrptnStsRsnCd timestamp=1391801091387, value=SOCHI-1
> > > > >
> > > > > Notice that the value and timestamp have changed.
> > > > >
> > > > > Why do I see this?
> > > > >
> > > > > ameet
> > > > >
> > > > >
> > > > > On Fri, Feb 7, 2014 at 2:09 PM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >
> > > > > > bq. However, calling bypass() seems to have no effect.
> > > > > >
> > > > > > In HRegion :
> > > > > >
> > > > > >           if (coprocessorHost.prePut((Put) m, walEdit,
> > > > > m.getWriteToWAL()))
> > > > > > {
> > > > > >             // pre hook says skip this Put
> > > > > >             // mark as success and skip in doMiniBatchMutation
> > > > > >             batchOp.retCodeDetails[i] = OperationStatus.SUCCESS;
> > > > > >
> > > > > > Can you be a bit more specific about your observation ?
> > > > > >
> > > > > > Cheers
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 7, 2014 at 11:02 AM, aa a <ameet.chaubal@gmail.com>
> > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I'd like to perform a put but have it succeed only if certain
> > > > > conditions
> > > > > > > based on values of some columns are met.
> > > > > > > I am on 0.94.6
> > > > > > >
> > > > > > > my first option was:
> > > > > > > write a prePut() Observer, perform the checks in there
and then
> > > call
> > > > a
> > > > > > > bypass() if needed.
> > > > > > > However, calling bypass() seems to have no effect. I thought
> that
> > > > > > bypass()
> > > > > > > would cause HBase to not perform the put.
> > > > > > > Is that not correct?
> > > > > > >
> > > > > > > second option:
> > > > > > > write a preCheckAndPut() and perform the validations there.
> > problem
> > > > is
> > > > > > that
> > > > > > > I have been running the process via a map/reduce and using
> > > > > > > TableMapReduceUtil.initTableReducerJob(
> > > > > > >         TABLE_NAME,
> > > > > > >         null,
> > > > > > >         job);
> > > > > > > And there seems to be no way to tell this to use "checkAndPut"
> > > > instead
> > > > > of
> > > > > > > "put", is there?
> > > > > > >
> > > > > > >
> > > > > > > third option:
> > > > > > > I could try to do my own htable.checkAndPut from the mapper.
> > > However,
> > > > > > since
> > > > > > > I am on a kerberized cluster, I am not able to get an htable
in
> > the
> > > > > > setup()
> > > > > > > method of the mapper.
> > > > > > > GSS initiate failed [Caused by GSSException: No valid
> credentials
> > > > > > provided
> > > > > > > (Mechanism level: Failed to find any Kerberos tgt)]
> > > > > > >  trying to perform UserGroupInformation.loginUserFromKeytab()
> > fails
> > > > > too;
> > > > > > > I  get Caused by: javax.security.auth.login.LoginException:
> > Unable
> > > to
> > > > > > > obtain password from user
> > > > > > > So I am not able to do any htable call from the mapper.
> > > > > > >
> > > > > > > The only M/R that has worked for me to read from HDFS text
file
> > as
> > > > > source
> > > > > > > and write to HBase sink is
> > TableMapReduceUtil.initTableReducerJob()
> > > > > with
> > > > > > > null for reducer class.
> > > > > > >
> > > > > > > Any thoughts?
> > > > > > >
> > > > > > > Thank you much in advance
> > > > > > >
> > > > > > > ameet
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message