hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: scope of RegionCoprocessorEnvironment sharedData
Date Fri, 07 Jul 2017 20:37:05 GMT
The tricky part about correlating preScannerOpen() with postScannerNext()
call is that:

Scan object is not passed to postScannerNext() (therefore the attribute
passed from client is absent)
The thread handling postScannerNext() may be different from the thread
handling preScannerOpen()

There is RegionScannerHolder class but it is private to RSRpcServices.

FYI

On Thu, Jul 6, 2017 at 1:59 PM, Veerraju Tadimeti <tvvraju@gmail.com> wrote:

> I would like to invoke postScannerNext() based on a scan attribute. When
> two different scans are invoked simultaneously, one with attribute and one
> without attribute ,  how can I invoke postScannerNext() based on the scan
> attribute.
>
> Currently i am doing like this
>
>    - checking the scan attribute in preScannerOpen(),
>       - if exists , set that attribute in sharedData,
>    - check that variable in postScannerNext()
>    - execute logic only if variable exists in sharedData
>
> If first scan operation with attribute is requested, we set that in
> sharedData in preScannerOpen(), meanwhile if second scan operation without
> attribute is requested, I think , according to your reply,  the
> postScannerNext() will get invoked for both the operations .
>
> is this correct? if Yes, what is the alternative.
>
> Thanks in Advance.
>
>
> Thanks,
> Raju,
> (972)273-0155.
>
> On Thu, Jul 6, 2017 at 1:39 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > If the two different scan operations are for the same region, same
> instance
> > of coprocessor is involved.
> >
> > Otherwise, different instances of coprocessor are involved.
> >
> > On Thu, Jul 6, 2017 at 11:22 AM, Veerraju Tadimeti <tvvraju@gmail.com>
> > wrote:
> >
> > > for two different scan operations , will same instance of coprocessor
> > > executes?
> > >
> > > Thanks,
> > > Raju,
> > > (972)273-0155.
> > >
> > > On Thu, Jul 6, 2017 at 11:53 AM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > sharedData is created when coprocessor is loaded (per region).
> > > >
> > > > So as long as the coprocessor instance doesn't change, the data you
> put
> > > in
> > > > should be there.
> > > >
> > > > On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti <tvvraju@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Is sharedData per Operation request ?
> > > > >
> > > > > I mean for a scan operation, if we put variables in sharedData in
> > > > > coprocessor, with a value,   will the same variables exists(with
> the
> > > same
> > > > > value) in sharedData in coprocessor for new scan operation ?
> > > > >
> > > > > Thanks,
> > > > > Raju,
> > > > > (972)273-0155.
> > > > >
> > > > > On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu <yuzhihong@gmail.com>
> wrote:
> > > > >
> > > > > > sharedData is in RegionEnvironment which is created
> > > > > > by RegionCoprocessorHost.
> > > > > > RegionCoprocessorHost is per region.
> > > > > >
> > > > > > When variable is removed in postScannerClose(), it wouldn't
> affect
> > > > > > sharedData
> > > > > > in another region.
> > > > > >
> > > > > > FYI
> > > > > >
> > > > > > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <
> > tvvraju@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > hi,
> > > > > > >
> > > > > > > I have few questions regarding scope of
> > > > *RegionCoprocessorEnvironment*
> > > > > > >  sharedData.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >    - *Is sharedData map is shared accross all instances
> > > > simultaneously
> > > > > ?*
> > > > > > >       -  I am putting a variable in sharedData in
> > preScannerOpen()
> > > > > based
> > > > > > on
> > > > > > >       scan attribute,
> > > > > > >       - check that variable exists in postScannerNext()
then
> > apply
> > > > > logic,
> > > > > > >       - remove the variable postScannerClose().
> > > > > > >       - If data is in multiple regions, when one coprocessor
> > > removes
> > > > > > >       variable in postScannerClose(), will the variable
is NULL
> > for
> > > > > > another
> > > > > > >       region coprocessor in postScannerNext() ?
> > > > > > >
> > > > > > >
> > > > > > >    - *    Is sharedData map is shared across all the client
> > request
> > > > > > >    operations ?*
> > > > > > >
> > > > > > > If a variable is set in sharedData for one client operation(say
> > > > SCAN),
> > > > > > will
> > > > > > > the variable is available for another client operation(new
> SCAN)
> > ?
> > > > > > >
> > > > > > >
> > > > > > >    -  *Will the variables be garbage collected even if
we dont
> > > > > implement
> > > > > > >    (removed variables in sharedData) postScannerClose()
method*
> > > > > > >
> > > > > > >
> > > > > > > Please find below the logic that I am using currently
> > > > > > > *CODE: *
> > > > > > >
> > > > > > >     public RegionScanner
> > > > > > > *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment>
> > > > > > > e, Scan scan, RegionScanner s) throws IOException {
> > > > > > >         byte[] useGetInPostScannerNext =
> > scan.getAttribute(USE_GET_
> > > > > > > OPERATION_IN_POST_SCANNER_NEXT);
> > > > > > >         String useGetInPostScannerNextStr = Bytes.toString(
> > > > > > > useGetInPostScannerNext);
> > > > > > >         if (Boolean.parseBoolean(useGetInPostScannerNextStr))
> {
> > > > > > >             e.getEnvironment().getSharedData().put(USE_GET_
> > > > > > > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > > > > > >         }
> > > > > > >         return super.preScannerOpen(e, scan, s);
> > > > > > >     }
> > > > > > >
> > > > > > > @Override
> > > > > > >     public boolean *postScannerNext*(final
> > > > > > > ObserverContext<RegionCoprocessorEnvironment>
> > > > > > > e,
> > > > > > >             final InternalScanner s, final List<Result>
> results,
> > > > final
> > > > > > int
> > > > > > > limit,
> > > > > > >             final boolean hasMore) throws IOException {
> > > > > > >         try {
> > > > > > >
> > > > > > >             if (canUseGetOperation(e)) {
> > > > > > >
> > > > > > >                //logic goes here
> > > > > > >             }
> > > > > > >         } catch (Exception ex) {
> > > > > > >             logger.error("Exception in postScannerNext
", ex);
> > > > > > >             throw new IOException(ex);
> > > > > > >         }
> > > > > > >         return hasMore;
> > > > > > >     }
> > > > > > >
> > > > > > >     @Override
> > > > > > >     public void
> > > > > > > *postScannerClose*(ObserverContext<
> RegionCoprocessorEnvironment>
> > > > > > > e, InternalScanner s) throws IOException {
> > > > > > >         if (canUseGetOperation(e)) {
> > > > > > >             e.getEnvironment().getSharedData().remove(USE_
> > > > > > > GET_OPERATION_IN_POST_SCANNER_NEXT);
> > > > > > >         }
> > > > > > >         super.postScannerClose(e, s);
> > > > > > >     }
> > > > > > >
> > > > > > >     private boolean *canUseGetOperation*(final
> > > > > > > ObserverContext<RegionCoprocessorEnvironment>
> > > > > > > e) {
> > > > > > >         String useGetOperationInPostScannerNext = (String)
> > > > > > > e.getEnvironment().getSharedData().get(USE_GET_
> > > > > > OPERATION_IN_POST_SCANNER_
> > > > > > > NEXT);
> > > > > > >         return Boolean.parseBoolean(
> > useGetOperationInPostScannerNe
> > > > xt);
> > > > > > >     }
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Raju,
> > > > > > > (972)273-0155 <(972)%20273-0155>.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message