hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Allow RegionCoprocessorEnvironment to register custom scanners?
Date Tue, 09 Aug 2011 16:50:25 GMT
Great! I'm glad you have everything you need Lars. If at some point you become stuck because
there is indeed some control or API surface missing, please write back!
 
Best regards,


   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)


----- Original Message -----
> From: lars hofhansl <lhofhansl@yahoo.com>
> To: "user@hbase.apache.org" <user@hbase.apache.org>; Andrew Purtell <apurtell@apache.org>
> Cc: 
> Sent: Monday, August 8, 2011 7:53 PM
> Subject: Re: Allow RegionCoprocessorEnvironment to register custom scanners?
> 
> I see.I just didn't see how you could communicate any information to the 
> server via a Scan, but now I see Scan.setAttribute(...).
> 
> Thanks Andy.
> 
> -- Lars
> 
> 
> 
> ________________________________
> From: Andrew Purtell <apurtell@apache.org>
> To: "user@hbase.apache.org" <user@hbase.apache.org>; lars 
> hofhansl <lhofhansl@yahoo.com>
> Sent: Monday, August 8, 2011 5:50 PM
> Subject: Re: Allow RegionCoprocessorEnvironment to register custom scanners?
> 
> The RegionObserver already wraps all of the scanner operations. 
> RegionObserver.preScannerOpen can create an InternalScanner and return it 
> exactly as you propose with "HRegionServer.addScanner(InternalScanner) 
> ". 
> 
> preScannerOpen takes a Scan object.
> 
> Only if preScannerOpen does not return an InternalScanner will the RegionServer 
> look for a "real" InternalScanner.
> 
> So I don't see what addScanner would buy you.
> 
> Best regards,
> 
> 
>    - Andy
> 
> Problems worthy of attack prove their worth by hitting back. - Piet Hein (via 
> Tom White)
> 
> 
>> ________________________________
>> From: lars hofhansl <lhofhansl@yahoo.com>
>> To: "user@hbase.apache.org" <user@hbase.apache.org>
>> Sent: Monday, August 8, 2011 5:38 PM
>> Subject: Allow RegionCoprocessorEnvironment to register custom scanners?
>> 
>> Currently coprocessors can't do any streaming operations.
>> 
>> I think that would be a necessary feature to perform long running operations 
> on the server (like scans) that in turn could produce a lot of data.
>> GroupBy type aggregates come to mind, but there are many more cases.
>> 
>> 
>> Somewhere I read about some approach for server side cursors (can't find 
> that discussion now).
>> I think a simpler approach would be allowing a coprocessor to register new 
> InternalScanners that it could implement,
>> and then have some way of accessing the scanner via the normal ClientScanner 
> mechanism.
>> Maybe by just exposing  long HRegionServer.addScanner(InternalScanner) 
> through RegionServerServices.
>> and adding  public ResultScanner getScanner(long scannerId) ... on HTable, 
> and similar on all other clients (I don't know anything about the client 
> beside the HTable Java client).
>> 
>> 
>> Or something similar (just making this up here).
>> 
>> 
>> That way all major parts are already in place (Client Scanners are good in 
> performing caching, the coprocessor could just wrap "real" internal 
> scanners, etc). The problem is just about how to wire up the parts.
>> 
>> 
>> Thoughts? Are questions like this better asked on the dev list?
>> 
>> Thanks.
>> 
>> -- Lars
>> 
>> 
>> 
>

Mime
View raw message