hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: AggregateProtocol Help
Date Wed, 04 Jan 2012 15:01:35 GMT
I can see room for improvement w.r.t. ColumnInterpreters
I logged two JIRAs:
https://issues.apache.org/jira/browse/HBASE-5122 is for loading
ColumnInterpreters dynamically

https://issues.apache.org/jira/browse/HBASE-5123 is for adding more
aggregation functions.

Royston:
Feel free to elaborate on 5213 and explain what Mult aggregate should do.

Cheers

On Wed, Jan 4, 2012 at 3:43 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:

> Ted, Himanshu and Gary,
>
> It works now! I re-created my HBase table to contain Bytes.toBytes(Long)
> values and that fixed it.
>
> For the time being we can convert everything to Longs and work with that,
> but we will probably write our own ColumnInterpreters soon for our data
> types, so thanks for the pointer to HBASE-4946. There are also Functions we
> need (e.g. Median, Weighted Median, Mult) which might best be placed in the
> Aggregations Protocol. We'll be sure to discuss this with you when we start.
>
> Meanwhile, thanks again for all your help!
>
> Royston
>
>
> On 3 Jan 2012, at 18:58, Ted Yu wrote:
>
> > I like long messages :-) because they provide more clues.
> >
> > For part 1, you don't have to call Bytes.toxxx as long as the interpreter
> > uses method consistent with the way you write values into HBase tables.
> >
> > For part 2, HBASE-4946 is related.
> > Basically you need to place the jar containing your coprocessor and
> > interpreter code on hdfs so that you can load it into your HBase table.
> > Look at this for details:
> > https://issues.apache.org/jira/browse/HBASE-4554
> >
> > Cheers
> >
> > On Tue, Jan 3, 2012 at 10:42 AM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Hi Ted,
> >>
> >> PART 1
> >> =====
> >> Thanks for the hint. I think maybe you have given me some inspiration!
> >>
> >> It looks like getValue will return null if the table value is not the
> >> length
> >> of a long. When we created our table (batch loading CSVs using the
> >> SampleUploader example) we simply have this as our put():
> >> put.add(family, Bytes.toBytes("advanceKWh"), advanceKWh);
> >> [note we do no Bytes.toxxx casts to the advanceKWh value. The values
> look
> >> OK
> >> from HBase shell though :-)]
> >>
> >> but I looked at TestAggregateProtocol.java again and I see there puts
> like:
> >> p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(l)),
> >> Bytes.toBytes(l * 10));
> >>
> >> So my hypothesis is that we need to do something like:
> >> Long l = new Long(1);
> >> put.add(family, Bytes.toBytes("advanceKWh"), Bytes.toBytes(l *
> >> advanceKWh));
> >> when we create the table.
> >>
> >> Do you think my hypothesis is correct? Did we build our table
> incorrectly
> >> for reading longs from it?
> >>
> >> PART 2
> >> =====
> >> Anyway we will obviously need to make our own interpreters. but we
> failed
> >> at
> >> this task so far:
> >> In order to implement our own ColumnInterpretter, we first attempted
> simply
> >> extending the LongColumnInterpreter and passing that as a parameter to
> >> aClient.sum().
> >> import org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter;
> >>
> >> public class LCI extends LongColumnInterpreter {
> >> public Long add(Long a, Long b) {
> >> System.out.print("LCI.add("+a+","+b+")");
> >> Long c = super.add(a,b);
> >> System.out.println(" -> "+c);
> >> return c;
> >> }
> >> };
> >> ....
> >> final ColumnInterpreter<Long, Long> ci = new LCI();
> >>  long sum=-1;
> >> try {
> >>  sum = aClient.sum(EDRP_TABLE, ci, scan);
> >> ...
> >> However, we received class not found errors in our regionserver log
> when we
> >> ran this code
> >> Clearly we are missing something
> >> We've started looking at modifying the HBase source and rebuilding that
> >> But I think this is not the way we should be working
> >> It seems that we should be implementing the ColumnInterpreter interface
> and
> >> passing an instance of that implementation class too as a parameter.  Is
> >> this correct?  Could you provide us with an example?  Any ideas why we
> >> might
> >> be getting class not found exception from the regionserver?
> >>
> >> Sorry for the long message!
> >>
> >> Royston (and Tom)
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Yu [mailto:yuzhihong@gmail.com]
> >> Sent: 03 January 2012 18:00
> >> To: user@hbase.apache.org
> >> Subject: Re: AggregateProtocol Help
> >>
> >> My previous email might not be hitting the root cause.
> >> I think the following method in LCI may be giving you the null:
> >>
> >> public Long getValue(byte[] colFamily, byte[] colQualifier, KeyValue kv)
> >>     throws IOException {
> >>   if (kv == null || kv.getValueLength() != Bytes.SIZEOF_LONG)
> >>     return null;
> >>   return Bytes.toLong(kv.getBuffer(), kv.getValueOffset());
> >>
> >> Look at the if statement above carefully.
> >> If it doesn't match how you store values in HBase, feel free to subclass
> >> LongColumnInterpreter and provide the correct interpretation.
> >>
> >> BTW you don't need to restart cluster just because you need to use your
> own
> >> interpreter :-)
> >>
> >> On Tue, Jan 3, 2012 at 9:48 AM, Royston Sellman <
> >> royston.sellman@googlemail.com> wrote:
> >>
> >>> Hi Ted,
> >>>
> >>> Here is the output. As you can see aClient is not nul:
> >>>
> >>> AggregationClient aClient = new AggregationClient(conf);
> >>>  System.err.println("aClient: "+aClient);
> >>>
> >>> <<<    aClient:
> >>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16
> >>>
> >>> It will take us a little while to add log code to LCI... we have to
> >>> edit the source, rebuild 0.92, redistribute round our cluster, restart
> >>> ;) We'll get back to you when this is done.
> >>>
> >>> Royston
> >>>
> >>> -----Original Message-----
> >>> From: Ted Yu [mailto:yuzhihong@gmail.com]
> >>> Sent: 03 January 2012 17:10
> >>> To: user@hbase.apache.org
> >>> Subject: Re: AggregateProtocol Help
> >>>
> >>> Royston:
> >>> Thanks for your effort trying to hunt down the problem.
> >>>
> >>> Can you add a log after this line to see if aClient is null ?
> >>>              AggregationClient aClient = new AggregationClient(conf);
> >>>
> >>> I was looking at LongColumnInterpreter.add() which is called by
> >>> aClient.sum()
> >>> Can you add a few log statements in LongColumnInterpreter.add() to see
> >>> what parameters are passed to it ?
> >>>
> >>> Cheers
> >>>
> >>> On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman <
> >>> royston.sellman@googlemail.com> wrote:
> >>>
> >>>> Hi Ted, Himanshu, Gary,
> >>>>
> >>>> Thanks again for your attention. I experimented with a shorter table
> >>>> and it looks like the timeout error was spurious...
> >>>>
> >>>> With the shorter table I now get an NPE when I call
> >>>> AggregationClient.sum().
> >>>> Here's the code snippet:
> >>>>
> >>>>               // Test the table
> >>>>               HTable table = new HTable(EDRP_TABLE);
> >>>>               Get get = new Get(Bytes.toBytes("row-aa"));
> >>>>               get.addColumn(Bytes.toBytes("EDRP"),
> >>>> Bytes.toBytes("advanceKWh"));
> >>>>               Result result = table.get(get);
> >>>>               byte [] val = result.getValue(Bytes.toBytes("EDRP"),
> >>>> Bytes.toBytes("advanceKWh"));
> >>>>               System.out.println("Row aa = " +
> >>>> Bytes.toString(val));
> >>>>
> >>>>               AggregationClient aClient = new AggregationClient(conf);
> >>>>               Scan scan = new Scan();
> >>>>                scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER);
> >>>>               scan.setStartRow(Bytes.toBytes("row-ab"));
> >>>>               scan.setStopRow(Bytes.toBytes("row-az"));
> >>>>               System.out.println(Bytes.toString(EDRP_FAMILY) + ":"
> >>>> + Bytes.toString(EDRP_QUALIFIER));
> >>>>                final ColumnInterpreter<Long, Long> ci = new
> >>>> LongColumnInterpreter();
> >>>>                long sum=-1;
> >>>>               try {
> >>>>                       sum = aClient.sum(EDRP_TABLE, ci, scan);
> >>>>               } catch (Throwable e) {
> >>>>                       // TODO Auto-generated catch block
> >>>>                       e.printStackTrace();
> >>>>               }
> >>>>               System.out.println(sum);
> >>>>
> >>>> The first part is just to check that my table is OK. It prints the
> >>>> correct value for row aa. Then I check CF:CQ is correct. Then I get
> >>>> the -1 that sum was defined as. Then the NPE. Here is the run output:
> >>>>
> >>>> Row aa = 3.0
> >>>> EDRP:advanceKWh
> >>>> -1
> >>>> java.lang.NullPointerException
> >>>> at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66)
> >>>> at EDRPAggregator.main(EDRPAggregator.java:96)
> >>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
> >>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> >>>> .j
> >>>> ava:39
> >>>> )
> >>>> at
> >>>>
> >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> >>>> ss
> >>>> orImpl
> >>>> .java:25)
> >>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>
> >>>> Line 66 is the line:
> >>>>                       sum = aClient.sum(EDRP_TABLE, ci, scan);
> >>>>
> >>>> But I'm not sure that is really the line generating the NPE because
> >>>> I get the -1 output (same result when line 66 is not wrapped in
> >>>> try-catch
> >>> BTW).
> >>>> Anyway, -1 is clearly not the correct value for the sum.
> >>>>
> >>>> [Note: I'm setting a start and end row on the scan because I saw a
> >>>> comment in AggregationClient.java saying they must be set. However,
> >>>> I think this comment is obsolete as the test code in
> >>>> TestAggregateProtocol.java sometimes does not include start & end
> >>>> rows. The run is exactly the same without the start & stop rows ]
> >>>>
> >>>> We are struggling with Aggregations not least because
> >>>> TestAggregateProtocol.java is the only source of example code we
> >>>> could find and it is not ideal because it is designed to work with
> >>>> MiniCluster rather than a real cluster like ours. Is there any other
> >>>> example code in existence?
> >>>> I think that would really help us.
> >>>>
> >>>> Meanwhile, thanks for your patience with our questions. We are
> >>>> really impressed with the AggregationProtocol
> >>>> architecture/functionality and very keen to use it.
> >>>>
> >>>> Royston (and Tom)
> >>>>
> >>>> -----Original Message-----
> >>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
> >>>> Sent: 02 January 2012 06:23
> >>>> To: user@hbase.apache.org
> >>>> Subject: Re: AggregateProtocol Help
> >>>>
> >>>> Hi Royston,
> >>>>
> >>>> Try increasing the value set for hbase.rpc.timeout (by default 60
> >>> seconds).
> >>>> Add something like this to hbase-site.xml:
> >>>>
> >>>> <property>
> >>>> <name>hbase.rpc.timeout</name>
> >>>> <value>180000</value>
> >>>> </property>
> >>>>
> >>>> This would increase the timeout value to 3 minutes, for example.
> >>>>
> >>>> But as Ted and Himanshu mention, 557k rows is not a large table and
> >>>> should not be taking very long to process, unless these happen to be
> >>>> very, very wide rows.  It sounds like there could be something else
> >>>> going on.  Does debug level logging show anything else in the region
> >>> server log?
> >>>>
> >>>> --gh
> >>>>
> >>>> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>> Thanks for the reminder Himanshu.
> >>>>>
> >>>>> Royston:
> >>>>> From this blog you can get some history on this subject:
> >>>>> http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoproce
> >>>>> ss
> >>>>> or
> >>>>> .html
> >>>>>
> >>>>>
> >>>>> On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha
> >>>>> <hvashish@cs.ualberta.ca
> >>>>>> wrote:
> >>>>>
> >>>>>> Hello Royston,
> >>>>>>
> >>>>>> Sorry to hear that you are getting trouble while using
> >>>>>> Aggregation functionalities.
> >>>>>>
> >>>>>> 557k rows seems to be a small table and a SocketTimeout does not
> >>>>>> seem to be an ok response.
> >>>>>> It will be good to know the region distribution as such. (how
> >>>>>> many
> >>>> regions?
> >>>>>> Is it a full table scan?)
> >>>>>>
> >>>>>> You are using the sum function; how are you using the
> >>> ColumnInterpreter.
> >>>>>> Can you enable the log level to debug to see why the RS is taking
> >>>>>> that long to respond (more than 113 sec).
> >>>>>> The 0 return value is the default result.
> >>>>>>
> >>>>>> Thanks for trying this out.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Himanshu
> >>>>>>
> >>>>>> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman <
> >>>>>> royston.sellman@googlemail.com> wrote:
> >>>>>>
> >>>>>>> Hi Ted,
> >>>>>>>
> >>>>>>> I think 0 is the only value we ever see (I'll check tomorrow:
> >>>>>>> the server is down right now). Our table has 557,000 rows. I'll
> >>>>>>> try a much shorter table tomorrow.
> >>>>>>>
> >>>>>>> Yes, we have RS running on the NN, but it's a test cluster and
> >>>>>>> we are
> >>>>>> used
> >>>>>>> to it :)
> >>>>>>>
> >>>>>>> Do you think using AggregationProtocol is the best strategy for
> >>>>>>> the case where we want to use basic SQL-style functions like
> >>>>>>> SUM, AVG, STD, MIN, MAX? Do you think there is a better strategy?
> >>>>>>>
> >>>>>>> Many thanks,
> >>>>>>> Royston
> >>>>>>>
> >>>>>>>
> >>>>>>> On 1 Jan 2012, at 17:58, Ted Yu wrote:
> >>>>>>>
> >>>>>>>> Royston:
> >>>>>>>> Happy New Year to you too.
> >>>>>>>>
> >>>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>>> namenode/10.0.0.235:60020
> >>>>>> failed
> >>>>>>> on
> >>>>>>>>
> >>>>>>>> It seems the namenode above actually refers to a region server.
> >>>>>>>> This
> >>>>>> is a
> >>>>>>>> little bit confusing :-)
> >>>>>>>>
> >>>>>>>> The sum value below is 0.
> >>>>>>>> Have you ever seen a value greater than 0 ?
> >>>>>>>>
> >>>>>>>> How many rows are there in this CF:CQ ?
> >>>>>>>> The timeout was reported earlier by other people where
> >>>>>>>> there're many
> >>>>>> rows
> >>>>>>>> in the table.
> >>>>>>>>
> >>>>>>>> There is a JIRA to provide streaming support for coprocessor
> >>>>>>>> but the development there has stalled.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>>
> >>>>>>>> On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman <
> >>>>>>>> royston.sellman@googlemail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Gary and Ted,
> >>>>>>>>>
> >>>>>>>>> Royston (Tom's colleague) here. Back onto this after the
> >>>>>>>>> Christmas/New
> >>>>>>> Year
> >>>>>>>>> break.
> >>>>>>>>>
> >>>>>>>>> Many thanks for your help so far. We enabled our database
> >>>>>>>>> via your hbase-site.xml mod and were able to move on. to
> >>>>>>>>> other errors. But I
> >>>>>>> think
> >>>>>>>>> we
> >>>>>>>>> are now actually getting an aggregation partially calculated
> >>>>>>>>> on our
> >>>>>>> table
> >>>>>>>>> (this feels like progress). The details:
> >>>>>>>>>
> >>>>>>>>> On running our client we now get this exception:
> >>>>>>>>> 11/12/31 17:51:09 WARN
> >>>>>>>>> client.HConnectionManager$HConnectionImplementation: Error
> >>>>>>>>> executing
> >>>>>> for
> >>>>>>>>> row
> >>>>>>>>>
> >>>>>>>>> java.util.concurrent.ExecutionException:
> >>>>>>>>> org.apache.hadoop.hbase.client.RetriesExhaustedException:
> >>>>>>>>> Failed after attempts=10, exceptions:
> >>>>>>>>> Sat Dec 31 17:41:30 GMT 2011,
> >>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>> namenode/10.0.0.235:60020
> >>>>>> failed
> >>>>>>>>> on
> >>>>>>>>> socket timeout exception: java.net.SocketTimeoutException:
> >>>>>>>>> 60000
> >>>>>> millis
> >>>>>>>>> timeout while waiting for channel to be ready for read. ch :
> >>>>>>>>> java.nio.channels.SocketChannel[connected
> >>>>>>>>> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020]
> >>>>>>>>> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09
> >>>>>>>>> GMT 2011,
> >>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>> namenode/10.0.0.235:60020
> >>>>>> failed
> >>>>>>>>> on
> >>>>>>>>> socket timeout exception: java.net.SocketTimeoutException:
> >>>>>>>>> 60000
> >>>>>> millis
> >>>>>>>>> timeout while waiting for channel to be ready for read. ch :
> >>>>>>>>> java.nio.channels.SocketChannel[connected
> >>>>>>>>> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020]
> >>>>>>>>>
> >>>>>>>>>      at
> >>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:
> >>>>>>>>> 22
> >>>>>>>>> 2)
> >>>>>>>>>      at
> >> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> >>>>>>>>> processExecs(HConnectionManager.java:1465)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java
> >>>>>> :1
> >>>>>> 55
> >>>>>> 5)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
> >>>>>> Ag
> >>>>>> gr
> >>>>>> egation
> >>>>>>>>> Client.java:229)
> >>>>>>>>>      at
> >>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >>>>>>>>>      at EDRPAggregator.main(EDRPAggregator.java:77)
> >>>>>>>>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>> Method)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>> java:39
> >>>>>>>>> )
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>> cc
> >>>>>> es
> >>>>>> sorImpl
> >>>>>>>>> .java:25)
> >>>>>>>>>      at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Looking at the log (.regionserver-namenode.log) I see this
> >>>>>>>>> debug
> >>>>>>> message:
> >>>>>>>>>
> >>>>>>>>> 2011-12-31 17:42:23,472 DEBUG
> >>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateImplementation:
> >>>>>>>>> Sum from
> >>>>>>> this
> >>>>>>>>> region is
> >>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.:
> >>>>>>> 0
> >>>>>>>>>
> >>>>>>>>> Where the sum value looks reasonable which makes me think
> >>>>>>>>> the sum of a CF:CQ worked. But I never see this value on
> >> stdout.
> >>>>>>>>> Then I see this warning:
> >>>>>>>>>
> >>>>>>>>> 2011-12-31 17:42:23,476 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>> (responseTooSlow):
> >>>>>>>>> {"processingtimems":113146,"call":"execCoprocess$
> >>>>>>>>> 2011-12-31 17:42:23,511 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>> IPC
> >>>>>>> Server
> >>>>>>>>> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$
> >>>>>>>>> 2011-12-31 17:42:23,515 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>> IPC
> >>>>>>> Server
> >>>>>>>>> handler 1 on 60020 caught: java.nio.channels.ClosedChann$
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.ja
> >>>>>> va
> >>>>>> :1
> >>>>>> 33)
> >>>>>>>>>      at
> >>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.
> >>>>>> ja
> >>>>>> va
> >>>>>> :1651)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse
> >>>>>> (H
> >>>>>> Ba
> >>>>>> seServe
> >>>>>>>>> r.java:924)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBase
> >>>>>> Se
> >>>>>> rv
> >>>>>> er.java
> >>>>>>>>> :1003)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(
> >>>>>> HB
> >>>>>> as
> >>>>>> eServer
> >>>>>>>>> .java:409)
> >>>>>>>>>      at
> >>>>>>>>>
> >>>>>>>
> >>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>> 1345)
> >>>>>>>>>
> >>>>>>>>> Have we missed out some step in the HBase/RegionServerconfig?
> >>>>>>>>> Or is
> >>>>>> our
> >>>>>>>>> client code still deficient?
> >>>>>>>>>
> >>>>>>>>> Can you offer any suggestions? Is there any example code for
> >>>>>>>>> the new Aggregations stuff.
> >>>>>>>>>
> >>>>>>>>> Thanks and Happy New Year to you guys,
> >>>>>>>>>
> >>>>>>>>> Royston (and Tom).
> >>>>>>>>>
> >>>>>>>>> (HBase 0.92, Hadoop 1.0)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
> >>>>>>>>> Sent: 23 December 2011 18:06
> >>>>>>>>> To: user@hbase.apache.org
> >>>>>>>>> Subject: Re: AggregateProtocol Help
> >>>>>>>>>
> >>>>>>>>> Hi Tom,
> >>>>>>>>>
> >>>>>>>>> The test code is not really the best guide for configuration.
> >>>>>>>>>
> >>>>>>>>> To enable the AggregateProtocol on all of your tables, add
> >>>>>>>>> this to the hbase-site.xml for the servers in your cluster:
> >>>>>>>>>
> >>>>>>>>> <property>
> >>>>>>>>>  <name>hbase.coprocessor.user.region.classes</name>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementatio
> >>>>>> n<
> >>>>>> /v
> >>>>>> alue>
> >>>>>>>>> </property>
> >>>>>>>>>
> >>>>>>>>> If you only want to use the aggregate functions on a
> >>>>>>>>> specific table
> >>>>>> (or
> >>>>>>>>> tables), then you can enable that individually for the table
> >>>>>>>>> from the
> >>>>>>>>> shell:
> >>>>>>>>>
> >>>>>>>>> 1) disable the table
> >>>>>>>>> hbase> disable 'EDRP7'
> >>>>>>>>>
> >>>>>>>>> 2) add the coprocessor
> >>>>>>>>> hbase> alter 'EDRP7', METHOD => 'table_att',
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImp
> >>>>>> le
> >>>>>> me
> >>>>>> ntation
> >>>>>>>>> ||'
> >>>>>>>>>
> >>>>>>>>> (Note that the pipes in the value string are required)
> >>>>>>>>>
> >>>>>>>>> 3) re-enable the table
> >>>>>>>>> hbase> enable 'EDRP7'
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Either way should work.  With the second approach you will
> >>>>>>>>> see the coprocessor listed when you describe the table from
> >>>>>>>>> the shell, as Ted mentioned.  With the first approach you
> >>>>>>>>> will not, but it should be
> >>>>>>> loaded
> >>>>>>>>> all the same.
> >>>>>>>>>
> >>>>>>>>> --gh
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu
> >>>>>>>>> <yuzhihong@gmail.com>
> >>>> wrote:
> >>>>>>>>>> I don't know why you chose HBaseTestingUtility to create
> >>>>>>>>>> the
> >>>> table.
> >>>>>>>>>> I guess you followed test code example.
> >>>>>>>>>>
> >>>>>>>>>> At least you should pass the conf to this ctor:
> >>>>>>>>>> public HBaseTestingUtility(Configuration conf) {
> >>>>>>>>>>
> >>>>>>>>>> If coprocessor was installed correctly, you should see
> >>>>>>>>>> something like(from
> >>>>>>>>>> HBASE-5070):
> >>>>>>>>>> coprocessor$1 =>
> >>>>>>>>>>
> >>>> '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox
> >>>>>>>>>> <Tom.Wilcox@cse.org.uk>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I am not sure how we load the AggregateImplementation into
> >>>>>>>>>>> the
> >>>>>> table.
> >>>>>>>>>>> When we are creating a table, we use the same functions as
> >>>>>>>>>>> the test
> >>>>>> as
> >>>>>>>>> follows...
> >>>>>>>>>>>
> >>>>>>>>>>> ...
> >>>>>>>>>>>>
> >>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >>>>>>>>>>>> );
> >>>>>>>>>>>>
> >>>>>>>>>>>>            // Utility.CreateHBaseTable(conf,
> >>>>>>>>>>>> otherArgs[1],
> >>>>>>>>>>> otherArgs[2],
> >>>>>>>>>>>> true);
> >>>>>>>>>>>>
> >>>>>>>>>>>>            HBaseTestingUtility util = new
> >>>>>>>>>>>> HBaseTestingUtility();
> >>>>>>>>>>>>            HTable table = util.createTable(EDRP_TABLE,
> >>>>>>>>>>>> EDRP_FAMILY);
> >>>>>>>>>>>>
> >>>>>>>>>>>>            AggregationClient aClient = new
> >>>>>>>>>>>> AggregationClient(conf);
> >>>>>>>>>>> ...
> >>>>>>>>>>>
> >>>>>>>>>>> Running DESCRIBE on a table produced shows the following
> >>> output:
> >>>>>>>>>>>
> >>>>>>>>>>> hbase(main):002:0> describe 'EDRP7'
> >>>>>>>>>>> DESCRIPTION
> >>>>>>>>>>>                                          ENABLED {NAME =>
> >>>>>>>>>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER
> >>>>>>>>>>> => 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>  true
> >>>>>>>>>>> '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =>
> >>>>>>>>>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> >>>>>>>>>>> B LOCKCACHE => 'true'}]}
> >>>>>>>>>>>
> >>>>>>>>>>> We are using the tip of 0.92 (cloned from the Git repo).
> >>>>>>>>>>> See the version string below:
> >>>>>>>>>>>
> >>>>>>>>>>> hbase(main):005:0> version 0.92.0, r1208286, Thu Dec 15
> >>>>>>>>>>> 13:16:03 GMT 2011
> >>>>>>>>>>>
> >>>>>>>>>>> We would really appreciate an example of how to create a
> >>>>>>>>>>> table that is enabled to handle Aggregation).
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ________________________________________
> >>>>>>>>>>> From: Ted Yu [yuzhihong@gmail.com]
> >>>>>>>>>>> Sent: 22 December 2011 17:03
> >>>>>>>>>>> To: user@hbase.apache.org
> >>>>>>>>>>> Subject: Re: AggregateProtocol Help
> >>>>>>>>>>>
> >>>>>>>>>>> Have you loaded AggregateImplementation into your table ?
> >>>>>>>>>>> Can you show us the contents of the following command in
> >>>>>>>>>>> hbase
> >>>>>> shell:
> >>>>>>>>>>> describe 'your-table'
> >>>>>>>>>>>
> >>>>>>>>>>> BTW are you using the tip of 0.92 ?
> >>>>>>>>>>> HBASE-4946 would be of help for dynamically loaded
> >>>>>>>>>>> coprocessors
> >>>>>> which
> >>>>>>>>>>> you might use in the future.
> >>>>>>>>>>>
> >>>>>>>>>>> Cheers
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox
> >>>>>>>>>>> <Tom.Wilcox@cse.org.uk>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> We are trying to use the aggregation functionality in
> >>>>>>>>>>>> HBase
> >>>>>>>>>>>> 0.92 and we have managed to get the test code working
> >>>>>>>>>>>> using the
> >>>>>> following
> >>>>>>>>> command:
> >>>>>>>>>>>>
> >>>>>>>>>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
> >>>>>>>>>>>> org.junit.runner.JUnitCore
> >>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
> >>>>>>>>>>>>
> >>>>>>>>>>>> Closer inspection of this test class has revealed that it
> >>>>>>>>>>>> uses a mini DFS cluster to populate and run the tests.
> >>>>>>>>>>>> These tests return
> >>>>>>>>> successfully.
> >>>>>>>>>>>>
> >>>>>>>>>>>> However, when we attempt to run similar code on our
> >>>>>>>>>>>> development HDFS cluster we experience the following error:
> >>>>>>>>>>>>
> >>>>>>>>>>>> 11/12/22 15:46:28 WARN
> >>>>>>>>>>>> client.HConnectionManager$HConnectionImplementation:
> >>>>>>>>>>>> Error executing for
> >>>>>>>>>>> row
> >>>>>>>>>>>> java.util.concurrent.ExecutionException:
> >>>>>>>>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>> ti
> >>>>>>>>>>>> on
> >>>>>>>>>>>> : No
> >>>>>>>>>>> matching
> >>>>>>>>>>>> handler for protocol
> >>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>> in region
> >>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>> .j
> >>>>>>>>>>>> av
> >>>>>>>>>>>> a:4010
> >>>>>>>>>>>> )
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>> r(
> >>>>>> HR
> >>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>> java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>> cc
> >>>>>> es
> >>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>> eR
> >>>>>> pc
> >>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>> 1325)
> >>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.
> >>>>>>>>>>>> ja
> >>>>>>>>>>>> va
> >>>>>>>>>>>> :222)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation.processExecs(HConnectionManager.java:1465)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.
> >>>>>>>>>>>> java:1
> >>>>>>>>>>>> 555)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
> >>>>>> Ag
> >>>>>> gr
> >>>>>>>>>>> egationClient.java:229)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >>>>>>>>>>>>  at EDRPAggregator.main(EDRPAggregator.java:77)
> >>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>> java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>> cc
> >>>>>> es
> >>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>>>>>>>>> Caused by:
> >>>>>>>>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>> ti
> >>>>>>>>>>>> on
> >>>>>>>>>>>> : No
> >>>>>>>>>>> matching
> >>>>>>>>>>>> handler for protocol
> >>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>> in region
> >>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>> .j
> >>>>>>>>>>>> av
> >>>>>>>>>>>> a:4010
> >>>>>>>>>>>> )
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>> r(
> >>>>>> HR
> >>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>> java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>> cc
> >>>>>> es
> >>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>> eR
> >>>>>> pc
> >>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>> 1325)
> >>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
> >>>>>>>>>>>> ti
> >>>>>>>>>>>> ve
> >>>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
> >>>>>> ru
> >>>>>> ct
> >>>>>>>>>>> orAccessorImpl.java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
> >>>>>> in
> >>>>>> gC
> >>>>>>>>>>> onstructorAccessorImpl.java:27)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >>>>>>>>>>>> 51
> >>>>>>>>>>>> 3)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
> >>>>>> io
> >>>>>> n(
> >>>>>>>>>>> RemoteExceptionHandler.java:96)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation.translateException(HConnectionManager.java:1651)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java:
> >>>>>>>>>>> 13
> >>>>>>>>>>> 27
> >>>>>>>>>>> )
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
> >>>>>>>>>>>> In
> >>>>>>>>>>>> vo
> >>>>>>>>>>>> ker.ja
> >>>>>>>>>>>> va:79)
> >>>>>>>>>>>>  at $Proxy3.getSum(Unknown Source)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>> ll
> >>>>>> (A
> >>>>>>>>>>> ggregationClient.java:233)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>> ll
> >>>>>> (A
> >>>>>>>>>>> ggregationClient.java:230)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
> >>>>>>>>>>>> ja
> >>>>>>>>>>>> va
> >>>>>>>>>>>> :303)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
> >>>>>> Ex
> >>>>>> ec
> >>>>>>>>>>> utor.java:886)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
> >>>>>> ut
> >>>>>> or
> >>>>>>>>>>> .java:908)
> >>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>> ti
> >>>>>>>>>>>> on
> >>>>>>>>>>>> : No
> >>>>>>>>>>> matching
> >>>>>>>>>>>> handler for protocol
> >>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>> in region
> >>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>> .j
> >>>>>>>>>>>> av
> >>>>>>>>>>>> a:4010
> >>>>>>>>>>>> )
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>> r(
> >>>>>> HR
> >>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>> java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>> cc
> >>>>>> es
> >>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>> eR
> >>>>>> pc
> >>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>> 1325)
> >>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
> >>>>>>>>>>>> ti
> >>>>>>>>>>>> ve
> >>>>>>>>>>>> Method)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
> >>>>>> ru
> >>>>>> ct
> >>>>>>>>>>> orAccessorImpl.java:39)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
> >>>>>> in
> >>>>>> gC
> >>>>>>>>>>> onstructorAccessorImpl.java:27)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >>>>>>>>>>>> 51
> >>>>>>>>>>>> 3)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
> >>>>>> io
> >>>>>> n(
> >>>>>>>>>>> RemoteExceptionHandler.java:96)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation.translateException(HConnectionManager.java:1651)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java:
> >>>>>>>>>>> 13
> >>>>>>>>>>> 27
> >>>>>>>>>>> )
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
> >>>>>>>>>>>> In
> >>>>>>>>>>>> vo
> >>>>>>>>>>>> ker.ja
> >>>>>>>>>>>> va:79)
> >>>>>>>>>>>>  at $Proxy3.getSum(Unknown Source)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>> ll
> >>>>>> (A
> >>>>>>>>>>> ggregationClient.java:233)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>> ll
> >>>>>> (A
> >>>>>>>>>>> ggregationClient.java:230)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>> em
> >>>>>> en
> >>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
> >>>>>>>>>>>> ja
> >>>>>>>>>>>> va
> >>>>>>>>>>>> :303)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
> >>>>>> Ex
> >>>>>> ec
> >>>>>>>>>>> utor.java:886)
> >>>>>>>>>>>>  at
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
> >>>>>> ut
> >>>>>> or
> >>>>>>>>>>> .java:908)
> >>>>>>>>>>>>  at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>>>
> >>>>>>>>>>>> The source for our class is:
> >>>>>>>>>>>>
> >>>>>>>>>>>> import static org.junit.Assert.assertEquals;
> >>>>>>>>>>>>
> >>>>>>>>>>>> import java.io.IOException;
> >>>>>>>>>>>>
> >>>>>>>>>>>> import org.apache.hadoop.conf.Configuration;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseTestingUtility;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.HConstants;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.HTableDescriptor;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.client.HTable;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.client.Put;
> >>>>>>>>>>>> import org.apache.hadoop.hbase.client.Scan;
> >>>>>>>>>>>> import
> >>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationCli
> >>>>>>>>>>>> en
> >>>>>>>>>>>> t;
> >>>>>>>>>>>> import
> >>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInte
> >>>>>>>>>>>> rp re ter; import org.apache.hadoop.hbase.util.Bytes;
> >>>>>>>>>>>> import org.apache.hadoop.util.GenericOptionsParser;
> >>>>>>>>>>>> import
> >>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
> >>>>>>>>>>>> import
> >>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
> >>>>>>>>>>>> import org.junit.Test;
> >>>>>>>>>>>>
> >>>>>>>>>>>> public class EDRPAggregator {
> >>>>>>>>>>>>
> >>>>>>>>>>>>     // private static final byte[] EDRP_FAMILY =
> >>>>>>>>>>>> Bytes.toBytes("EDRP");
> >>>>>>>>>>>>     // private static final byte[] EDRP_QUALIFIER =
> >>>>>>>>>>>> Bytes.toBytes("advanceKWh");
> >>>>>>>>>>>>
> >>>>>>>>>>>>     private static byte[] ROW = Bytes.toBytes("testRow");
> >>>>>>>>>>>>     private static final int ROWSIZE = 20;
> >>>>>>>>>>>>     private static byte[][] ROWS = makeN(ROW, ROWSIZE);
> >>>>>>>>>>>>     private static final byte[] TEST_QUALIFIER =
> >>>>>>>>>>>> Bytes.toBytes("TestQualifier");
> >>>>>>>>>>>>     private static final byte[] TEST_MULTI_CQ =
> >>>>>>>>>>>> Bytes.toBytes("TestMultiCQ");
> >>>>>>>>>>>>     private static final int rowSeperator1 = 5;
> >>>>>>>>>>>>       private static final int rowSeperator2 = 12;
> >>>>>>>>>>>>
> >>>>>>>>>>>>     public static void
> >>>>>>>>>>>> testSumWithValidRange(Configuration
> >>>>>>>>>>>> conf,
> >>>>>>>>>>>>                   String[] otherArgs) throws Throwable {
> >>>>>>>>>>>>            byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
> >>>>>>>>>>>>            byte[] EDRP_FAMILY =
> >>>>>>>>>>>> Bytes.toBytes(otherArgs[2]);
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >>>>>>>>>>>> );
> >>>>>>>>>>>>
> >>>>>>>>>>>>            // Utility.CreateHBaseTable(conf,
> >>>>>>>>>>>> otherArgs[1],
> >>>>>>>>>>> otherArgs[2],
> >>>>>>>>>>>> true);
> >>>>>>>>>>>>
> >>>>>>>>>>>>            HBaseTestingUtility util = new
> >>>>>>>>>>>> HBaseTestingUtility();
> >>>>>>>>>>>>            HTable table = util.createTable(EDRP_TABLE,
> >>>>>>>>>>>> EDRP_FAMILY);
> >>>>>>>>>>>>
> >>>>>>>>>>>>            AggregationClient aClient = new
> >>>>>>>>>>>> AggregationClient(conf);
> >>>>>>>>>>>>            Scan scan = new Scan();
> >>>>>>>>>>>>            scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
> >>>>>>>>>>>>            final ColumnInterpreter<Long, Long> ci = new
> >>>>>>>>>>>> LongColumnInterpreter();
> >>>>>>>>>>>>            long sum =
> >>>>>>>>>>>> aClient.sum(Bytes.toBytes(otherArgs[0]),
> >>>>>>>>>>>> ci,
> >>>>>>>>>>> scan);
> >>>>>>>>>>>>            System.out.println(sum);
> >>>>>>>>>>>>     }
> >>>>>>>>>>>>
> >>>>>>>>>>>>     /**
> >>>>>>>>>>>>     * Main entry point.
> >>>>>>>>>>>>     *
> >>>>>>>>>>>>      * @param argsThe
> >>>>>>>>>>>>     *            command line parameters.
> >>>>>>>>>>>>     * @throws Exception
> >>>>>>>>>>>>     *             When running the job fails.
> >>>>>>>>>>>>     */
> >>>>>>>>>>>>     public static void main(String[] args) throws
> >>>>>>>>>>>> Exception {
> >>>>>>>>>>>>            Configuration conf =
> >>>>>>>>>>>> HBaseConfiguration.create();
> >>>>>>>>>>>>
> >>>>>>>>>>>>            String[] otherArgs = new
> >>>>>>>>>>>> GenericOptionsParser(conf,
> >>>>>>>>>>>> args)
> >>>>>>>>>>>>                         .getRemainingArgs();
> >>>>>>>>>>>>            if (otherArgs.length != 3) {
> >>>>>>>>>>>>                   System.err
> >>>>>>>>>>>>                                .println("Wrong number of
> >>>>>>>>>>>> arguments: " + otherArgs.length);
> >>>>>>>>>>>>                   System.err.println("Usage: " +
> >>>>>>>>>>>> "<tablename> <colfam> <qualifier>");
> >>>>>>>>>>>>                   System.exit(-1);
> >>>>>>>>>>>>            }
> >>>>>>>>>>>>
> >>>>>>>>>>>>            try {
> >>>>>>>>>>>>                   testSumWithValidRange(conf, otherArgs);
> >>>>>>>>>>>>            } catch (Throwable e) {
> >>>>>>>>>>>>                   e.printStackTrace();
> >>>>>>>>>>>>            }
> >>>>>>>>>>>>     }
> >>>>>>>>>>>>
> >>>>>>>>>>>>     /**
> >>>>>>>>>>>>     * an infrastructure method to prepare rows for the
> >>>> testtable.
> >>>>>>>>>>>>     *
> >>>>>>>>>>>>      * @param base
> >>>>>>>>>>>>     * @param n
> >>>>>>>>>>>>     * @return
> >>>>>>>>>>>>     */
> >>>>>>>>>>>>     private static byte[][] makeN(byte[] base, int n) {
> >>>>>>>>>>>>            byte[][] ret = new byte[n][];
> >>>>>>>>>>>>            for (int i = 0; i < n; i++) {
> >>>>>>>>>>>>                   ret[i] = Bytes.add(base,
> >>>>>>>>>>>> Bytes.toBytes(i));
> >>>>>>>>>>>>            }
> >>>>>>>>>>>>            return ret;
> >>>>>>>>>>>>     }
> >>>>>>>>>>>> }
> >>>>>>>>>>>>
> >>>>>>>>>>>> Please can you suggest what might be causing and/or how
> >>>>>>>>>>>> we might fix this UnknownProtocolException?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, does anyone have any working examples using the
> >>>>>>>>>>>> aggregation
> >>>>>>>>>>> protocol
> >>>>>>>>>>>> other than the test code?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Tom
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message