hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Veerraju Tadimeti <tvvr...@gmail.com>
Subject Re: scope of RegionCoprocessorEnvironment sharedData
Date Fri, 14 Jul 2017 18:31:33 GMT
yeah, you are right.  I am getting scanner in postScannerOpen().  I tested
with preScannerOpen() earlier ,which is null scanner.  Thanks alot.

Thanks,
Raju,
(972)273-0155.

On Thu, Jul 13, 2017 at 11:50 PM, Anoop John <anoop.hbase@gmail.com> wrote:

> No co processors can be configured either globally at cluster level
> (RS configuration) or at table level.
>
> You dont need preScannerOpen.  I checked the code and we are passing
> the actual scanner been created to postScannerOpen() method.  This
> being null in ur case been strange!  Pls make sure u r implementing
> post hook not pre.
>
> -Anoop-
>
> On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti <tvvraju@gmail.com>
> wrote:
> > Can I load coprocessor dynamically for a scan operation, it should not
> be loaded for another scan operation if not intended
> >  btw I invoke scan from hive
> >
> > Sent from my iPhone
> >
> >> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <tvvraju@gmail.com>
> wrote:
> >>
> >> hi,
> >>
> >> Hi John,
> >>
> >> Thanks for the reply.
> >>
> >> I implemented #2 (another way) in ur above post:
> >>
> >>
> >>
> >> i debug the logs  : in PostScannerOpen() , regionScanner method
> parameter object is null
> >>
> >> Also, in preScannerOpen() , i returned return super.preScannerOpen(e,
> scan, new DelegateRegionScanner(s));
> >> in postScannerNext() , internalScanner object is
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
> >>
> >> #1 way (Put scanner in local map) - may not be possible, cos if two
> different scan operation with and without attribute hits at the same time,
> how can we differentiate in postScannerNext.
> >>
> >>
> >> Thanks,
> >> Raju,
> >> (972)273-0155.
> >>
> >>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <anoop.hbase@gmail.com>
> wrote:
> >>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
> >>> the param type should have been RegionScanner  I guess. We made that
> >>> mistake!
> >>> -Anoop-
> >>>
> >>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>> > The tricky part is that postScannerNext() passes InternalScanner
> parameter
> >>> > instead of RegionScanner.
> >>> >
> >>> > FYI
> >>> >
> >>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <anoop.hbase@gmail.com>
> wrote:
> >>> >
> >>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
> >>> >> and so can not make use of the attribute in Scan within this hook.
> >>> >> Just setting the sharedData variable will cause issue with
> concurrent
> >>> >> scans. (As u imagine)
> >>> >>
> >>> >> So I can think of solving this in 2 possible ways. (May be more
ways
> >>> >> possible)
> >>> >>
> >>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
> >>> >> where u will get the ref to Scanner been created as well as the
> Scan.
> >>> >> If the Scan is having attribute, keep that scanner within ur Map.
> >>> >> During postScannerNext  check if the coming in scanner is there
in
> ur
> >>> >> Map. If so that means this is the one where u can do the action.
> >>> >> Also dont forget to implement postScannerClose and remove that
> scanner
> >>> >> from the Map.   Here u might have some perf penalty as u have to
add
> >>> >> and get from Map which has to be a concurrent map too.
> >>> >>
> >>> >> Another way
> >>> >>
> >>> >> 2. Create a custom scanner implementing RegionScanner.   The new
one
> >>> >> has to take an original Region Scanner and just delegate the calls.
> On
> >>> >> postScannerOpen, u will get the original scanner been created and
u
> >>> >> can just wrap it with ur new scanner object. ( If the Scan object
is
> >>> >> having required attribute)..  In postScannerNext() u can check
for
> ur
> >>> >> own RegionScanner type and if so u can do action.
> >>> >>
> >>> >>
> >>> >> -Anoop-
> >>> >>
> >>> >>
> >>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <yuzhihong@gmail.com>
wrote:
> >>> >> >             if (canUseGetOperation(e)) {
> >>> >> >                //logic goes here
> >>> >> >
> >>> >> > Does your Get target the same region being scanned ?
> >>> >> > If not, issuing the Get is not advised since the other region
may
> be
> >>> >> hosted
> >>> >> > on different region server.
> >>> >> >
> >>> >> > Cheers
> >>> >> >
> >>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <
> tvvraju@gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> >> hi,
> >>> >> >>
> >>> >> >> I have few questions regarding scope of
> *RegionCoprocessorEnvironment*
> >>> >> >>  sharedData.
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>    - *Is sharedData map is shared accross all instances
> simultaneously
> >>> >> ?*
> >>> >> >>       -  I am putting a variable in sharedData in
> preScannerOpen()
> >>> >> based on
> >>> >> >>       scan attribute,
> >>> >> >>       - check that variable exists in postScannerNext()
then
> apply
> >>> >> logic,
> >>> >> >>       - remove the variable postScannerClose().
> >>> >> >>       - If data is in multiple regions, when one coprocessor
> removes
> >>> >> >>       variable in postScannerClose(), will the variable
is NULL
> for
> >>> >> another
> >>> >> >>       region coprocessor in postScannerNext() ?
> >>> >> >>
> >>> >> >>
> >>> >> >>    - *    Is sharedData map is shared across all the client
> request
> >>> >> >>    operations ?*
> >>> >> >>
> >>> >> >> If a variable is set in sharedData for one client operation(say
> SCAN),
> >>> >> will
> >>> >> >> the variable is available for another client operation(new
SCAN)
> ?
> >>> >> >>
> >>> >> >>
> >>> >> >>    -  *Will the variables be garbage collected even if
we dont
> implement
> >>> >> >>    (removed variables in sharedData) postScannerClose()
method*
> >>> >> >>
> >>> >> >>
> >>> >> >> Please find below the logic that I am using currently
> >>> >> >> *CODE: *
> >>> >> >>
> >>> >> >>     public RegionScanner
> >>> >> >> *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment>
> >>> >> >> e, Scan scan, RegionScanner s) throws IOException {
> >>> >> >>         byte[] useGetInPostScannerNext =
> scan.getAttribute(USE_GET_
> >>> >> >> OPERATION_IN_POST_SCANNER_NEXT);
> >>> >> >>         String useGetInPostScannerNextStr = Bytes.toString(
> >>> >> >> useGetInPostScannerNext);
> >>> >> >>         if (Boolean.parseBoolean(useGetInPostScannerNextStr))
{
> >>> >> >>             e.getEnvironment().getSharedData().put(USE_GET_
> >>> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> >>> >> >>         }
> >>> >> >>         return super.preScannerOpen(e, scan, s);
> >>> >> >>     }
> >>> >> >>
> >>> >> >> @Override
> >>> >> >>     public boolean *postScannerNext*(final
> >>> >> >> ObserverContext<RegionCoprocessorEnvironment>
> >>> >> >> e,
> >>> >> >>             final InternalScanner s, final List<Result>
results,
> final
> >>> >> int
> >>> >> >> limit,
> >>> >> >>             final boolean hasMore) throws IOException
{
> >>> >> >>         try {
> >>> >> >>
> >>> >> >>             if (canUseGetOperation(e)) {
> >>> >> >>
> >>> >> >>                //logic goes here
> >>> >> >>             }
> >>> >> >>         } catch (Exception ex) {
> >>> >> >>             logger.error("Exception in postScannerNext
", ex);
> >>> >> >>             throw new IOException(ex);
> >>> >> >>         }
> >>> >> >>         return hasMore;
> >>> >> >>     }
> >>> >> >>
> >>> >> >>     @Override
> >>> >> >>     public void
> >>> >> >> *postScannerClose*(ObserverContext<RegionCoprocessorEnvironment>
> >>> >> >> e, InternalScanner s) throws IOException {
> >>> >> >>         if (canUseGetOperation(e)) {
> >>> >> >>             e.getEnvironment().getSharedData().remove(USE_
> >>> >> >> GET_OPERATION_IN_POST_SCANNER_NEXT);
> >>> >> >>         }
> >>> >> >>         super.postScannerClose(e, s);
> >>> >> >>     }
> >>> >> >>
> >>> >> >>     private boolean *canUseGetOperation*(final
> >>> >> >> ObserverContext<RegionCoprocessorEnvironment>
> >>> >> >> e) {
> >>> >> >>         String useGetOperationInPostScannerNext = (String)
> >>> >> >> e.getEnvironment().getSharedData().get(USE_GET_
> >>> >> OPERATION_IN_POST_SCANNER_
> >>> >> >> NEXT);
> >>> >> >>         return Boolean.parseBoolean(
> useGetOperationInPostScannerNext);
> >>> >> >>     }
> >>> >> >>
> >>> >> >> Thanks,
> >>> >> >> Raju,
> >>> >> >> (972)273-0155 <(972)%20273-0155>.
> >>> >> >>
> >>> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message