hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anoop John <anoop.hb...@gmail.com>
Subject Re: scope of RegionCoprocessorEnvironment sharedData
Date Fri, 14 Jul 2017 04:50:17 GMT
No co processors can be configured either globally at cluster level
(RS configuration) or at table level.

You dont need preScannerOpen.  I checked the code and we are passing
the actual scanner been created to postScannerOpen() method.  This
being null in ur case been strange!  Pls make sure u r implementing
post hook not pre.

-Anoop-

On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti <tvvraju@gmail.com> wrote:
> Can I load coprocessor dynamically for a scan operation, it should not be loaded for
another scan operation if not intended
>  btw I invoke scan from hive
>
> Sent from my iPhone
>
>> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <tvvraju@gmail.com> wrote:
>>
>> hi,
>>
>> Hi John,
>>
>> Thanks for the reply.
>>
>> I implemented #2 (another way) in ur above post:
>>
>>
>>
>> i debug the logs  : in PostScannerOpen() , regionScanner method parameter object
is null
>>
>> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, new DelegateRegionScanner(s));
>> in postScannerNext() , internalScanner object is org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
>>
>> #1 way (Put scanner in local map) - may not be possible, cos if two different scan
operation with and without attribute hits at the same time, how can we differentiate in postScannerNext.
>>
>>
>> Thanks,
>> Raju,
>> (972)273-0155.
>>
>>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <anoop.hbase@gmail.com> wrote:
>>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
>>> the param type should have been RegionScanner  I guess. We made that
>>> mistake!
>>> -Anoop-
>>>
>>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> > The tricky part is that postScannerNext() passes InternalScanner parameter
>>> > instead of RegionScanner.
>>> >
>>> > FYI
>>> >
>>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <anoop.hbase@gmail.com>
wrote:
>>> >
>>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
>>> >> and so can not make use of the attribute in Scan within this hook.
>>> >> Just setting the sharedData variable will cause issue with concurrent
>>> >> scans. (As u imagine)
>>> >>
>>> >> So I can think of solving this in 2 possible ways. (May be more ways
>>> >> possible)
>>> >>
>>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>>> >> where u will get the ref to Scanner been created as well as the Scan.
>>> >> If the Scan is having attribute, keep that scanner within ur Map.
>>> >> During postScannerNext  check if the coming in scanner is there in ur
>>> >> Map. If so that means this is the one where u can do the action.
>>> >> Also dont forget to implement postScannerClose and remove that scanner
>>> >> from the Map.   Here u might have some perf penalty as u have to add
>>> >> and get from Map which has to be a concurrent map too.
>>> >>
>>> >> Another way
>>> >>
>>> >> 2. Create a custom scanner implementing RegionScanner.   The new one
>>> >> has to take an original Region Scanner and just delegate the calls.
On
>>> >> postScannerOpen, u will get the original scanner been created and u
>>> >> can just wrap it with ur new scanner object. ( If the Scan object is
>>> >> having required attribute)..  In postScannerNext() u can check for ur
>>> >> own RegionScanner type and if so u can do action.
>>> >>
>>> >>
>>> >> -Anoop-
>>> >>
>>> >>
>>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> >> >             if (canUseGetOperation(e)) {
>>> >> >                //logic goes here
>>> >> >
>>> >> > Does your Get target the same region being scanned ?
>>> >> > If not, issuing the Get is not advised since the other region may
be
>>> >> hosted
>>> >> > on different region server.
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <tvvraju@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> >> hi,
>>> >> >>
>>> >> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>>> >> >>  sharedData.
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>    - *Is sharedData map is shared accross all instances simultaneously
>>> >> ?*
>>> >> >>       -  I am putting a variable in sharedData in preScannerOpen()
>>> >> based on
>>> >> >>       scan attribute,
>>> >> >>       - check that variable exists in postScannerNext() then
apply
>>> >> logic,
>>> >> >>       - remove the variable postScannerClose().
>>> >> >>       - If data is in multiple regions, when one coprocessor
removes
>>> >> >>       variable in postScannerClose(), will the variable is
NULL for
>>> >> another
>>> >> >>       region coprocessor in postScannerNext() ?
>>> >> >>
>>> >> >>
>>> >> >>    - *    Is sharedData map is shared across all the client
request
>>> >> >>    operations ?*
>>> >> >>
>>> >> >> If a variable is set in sharedData for one client operation(say
SCAN),
>>> >> will
>>> >> >> the variable is available for another client operation(new
SCAN) ?
>>> >> >>
>>> >> >>
>>> >> >>    -  *Will the variables be garbage collected even if we dont
implement
>>> >> >>    (removed variables in sharedData) postScannerClose() method*
>>> >> >>
>>> >> >>
>>> >> >> Please find below the logic that I am using currently
>>> >> >> *CODE: *
>>> >> >>
>>> >> >>     public RegionScanner
>>> >> >> *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment>
>>> >> >> e, Scan scan, RegionScanner s) throws IOException {
>>> >> >>         byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>>> >> >> OPERATION_IN_POST_SCANNER_NEXT);
>>> >> >>         String useGetInPostScannerNextStr = Bytes.toString(
>>> >> >> useGetInPostScannerNext);
>>> >> >>         if (Boolean.parseBoolean(useGetInPostScannerNextStr))
{
>>> >> >>             e.getEnvironment().getSharedData().put(USE_GET_
>>> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
>>> >> >>         }
>>> >> >>         return super.preScannerOpen(e, scan, s);
>>> >> >>     }
>>> >> >>
>>> >> >> @Override
>>> >> >>     public boolean *postScannerNext*(final
>>> >> >> ObserverContext<RegionCoprocessorEnvironment>
>>> >> >> e,
>>> >> >>             final InternalScanner s, final List<Result>
results, final
>>> >> int
>>> >> >> limit,
>>> >> >>             final boolean hasMore) throws IOException {
>>> >> >>         try {
>>> >> >>
>>> >> >>             if (canUseGetOperation(e)) {
>>> >> >>
>>> >> >>                //logic goes here
>>> >> >>             }
>>> >> >>         } catch (Exception ex) {
>>> >> >>             logger.error("Exception in postScannerNext ", ex);
>>> >> >>             throw new IOException(ex);
>>> >> >>         }
>>> >> >>         return hasMore;
>>> >> >>     }
>>> >> >>
>>> >> >>     @Override
>>> >> >>     public void
>>> >> >> *postScannerClose*(ObserverContext<RegionCoprocessorEnvironment>
>>> >> >> e, InternalScanner s) throws IOException {
>>> >> >>         if (canUseGetOperation(e)) {
>>> >> >>             e.getEnvironment().getSharedData().remove(USE_
>>> >> >> GET_OPERATION_IN_POST_SCANNER_NEXT);
>>> >> >>         }
>>> >> >>         super.postScannerClose(e, s);
>>> >> >>     }
>>> >> >>
>>> >> >>     private boolean *canUseGetOperation*(final
>>> >> >> ObserverContext<RegionCoprocessorEnvironment>
>>> >> >> e) {
>>> >> >>         String useGetOperationInPostScannerNext = (String)
>>> >> >> e.getEnvironment().getSharedData().get(USE_GET_
>>> >> OPERATION_IN_POST_SCANNER_
>>> >> >> NEXT);
>>> >> >>         return Boolean.parseBoolean(useGetOperationInPostScannerNext);
>>> >> >>     }
>>> >> >>
>>> >> >> Thanks,
>>> >> >> Raju,
>>> >> >> (972)273-0155 <(972)%20273-0155>.
>>> >> >>
>>> >>
>>

Mime
View raw message