hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject median aggregate Was: AggregateProtocol Help
Date Fri, 06 Jan 2012 03:31:03 GMT
Royston:
For the median aggregate, is the following what you're looking for ?
Find the median among the values of all the keyvalue for cf:qualifier
column.

There is a well known distributed method of computing median that involves
multiple roundtrips (to the region servers).

Just want to confirm the use case.

Thanks

On Wed, Jan 4, 2012 at 10:57 AM, Royston Sellman <
royston.sellman@googlemail.com> wrote:

> Great ideas. Thanks.
>
> w.r.t. 5123: I'll think about it for a day or two then make some comments.
>
> 5122 is very desirable.
>
> Best Regards,
> Royston
>
> On 4 Jan 2012, at 15:01, Ted Yu wrote:
>
> > I can see room for improvement w.r.t. ColumnInterpreters
> > I logged two JIRAs:
> > https://issues.apache.org/jira/browse/HBASE-5122 is for loading
> > ColumnInterpreters dynamically
> >
> > https://issues.apache.org/jira/browse/HBASE-5123 is for adding more
> > aggregation functions.
> >
> > Royston:
> > Feel free to elaborate on 5213 and explain what Mult aggregate should do.
> >
> > Cheers
> >
> > On Wed, Jan 4, 2012 at 3:43 AM, Royston Sellman <
> > royston.sellman@googlemail.com> wrote:
> >
> >> Ted, Himanshu and Gary,
> >>
> >> It works now! I re-created my HBase table to contain Bytes.toBytes(Long)
> >> values and that fixed it.
> >>
> >> For the time being we can convert everything to Longs and work with
> that,
> >> but we will probably write our own ColumnInterpreters soon for our data
> >> types, so thanks for the pointer to HBASE-4946. There are also
> Functions we
> >> need (e.g. Median, Weighted Median, Mult) which might best be placed in
> the
> >> Aggregations Protocol. We'll be sure to discuss this with you when we
> start.
> >>
> >> Meanwhile, thanks again for all your help!
> >>
> >> Royston
> >>
> >>
> >> On 3 Jan 2012, at 18:58, Ted Yu wrote:
> >>
> >>> I like long messages :-) because they provide more clues.
> >>>
> >>> For part 1, you don't have to call Bytes.toxxx as long as the
> interpreter
> >>> uses method consistent with the way you write values into HBase tables.
> >>>
> >>> For part 2, HBASE-4946 is related.
> >>> Basically you need to place the jar containing your coprocessor and
> >>> interpreter code on hdfs so that you can load it into your HBase table.
> >>> Look at this for details:
> >>> https://issues.apache.org/jira/browse/HBASE-4554
> >>>
> >>> Cheers
> >>>
> >>> On Tue, Jan 3, 2012 at 10:42 AM, Royston Sellman <
> >>> royston.sellman@googlemail.com> wrote:
> >>>
> >>>> Hi Ted,
> >>>>
> >>>> PART 1
> >>>> =====
> >>>> Thanks for the hint. I think maybe you have given me some inspiration!
> >>>>
> >>>> It looks like getValue will return null if the table value is not the
> >>>> length
> >>>> of a long. When we created our table (batch loading CSVs using the
> >>>> SampleUploader example) we simply have this as our put():
> >>>> put.add(family, Bytes.toBytes("advanceKWh"), advanceKWh);
> >>>> [note we do no Bytes.toxxx casts to the advanceKWh value. The values
> >> look
> >>>> OK
> >>>> from HBase shell though :-)]
> >>>>
> >>>> but I looked at TestAggregateProtocol.java again and I see there puts
> >> like:
> >>>> p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(l)),
> >>>> Bytes.toBytes(l * 10));
> >>>>
> >>>> So my hypothesis is that we need to do something like:
> >>>> Long l = new Long(1);
> >>>> put.add(family, Bytes.toBytes("advanceKWh"), Bytes.toBytes(l *
> >>>> advanceKWh));
> >>>> when we create the table.
> >>>>
> >>>> Do you think my hypothesis is correct? Did we build our table
> >> incorrectly
> >>>> for reading longs from it?
> >>>>
> >>>> PART 2
> >>>> =====
> >>>> Anyway we will obviously need to make our own interpreters. but we
> >> failed
> >>>> at
> >>>> this task so far:
> >>>> In order to implement our own ColumnInterpretter, we first attempted
> >> simply
> >>>> extending the LongColumnInterpreter and passing that as a parameter to
> >>>> aClient.sum().
> >>>> import
> org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter;
> >>>>
> >>>> public class LCI extends LongColumnInterpreter {
> >>>> public Long add(Long a, Long b) {
> >>>> System.out.print("LCI.add("+a+","+b+")");
> >>>> Long c = super.add(a,b);
> >>>> System.out.println(" -> "+c);
> >>>> return c;
> >>>> }
> >>>> };
> >>>> ....
> >>>> final ColumnInterpreter<Long, Long> ci = new LCI();
> >>>> long sum=-1;
> >>>> try {
> >>>> sum = aClient.sum(EDRP_TABLE, ci, scan);
> >>>> ...
> >>>> However, we received class not found errors in our regionserver log
> >> when we
> >>>> ran this code
> >>>> Clearly we are missing something
> >>>> We've started looking at modifying the HBase source and rebuilding
> that
> >>>> But I think this is not the way we should be working
> >>>> It seems that we should be implementing the ColumnInterpreter
> interface
> >> and
> >>>> passing an instance of that implementation class too as a parameter.
>  Is
> >>>> this correct?  Could you provide us with an example?  Any ideas why we
> >>>> might
> >>>> be getting class not found exception from the regionserver?
> >>>>
> >>>> Sorry for the long message!
> >>>>
> >>>> Royston (and Tom)
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: Ted Yu [mailto:yuzhihong@gmail.com]
> >>>> Sent: 03 January 2012 18:00
> >>>> To: user@hbase.apache.org
> >>>> Subject: Re: AggregateProtocol Help
> >>>>
> >>>> My previous email might not be hitting the root cause.
> >>>> I think the following method in LCI may be giving you the null:
> >>>>
> >>>> public Long getValue(byte[] colFamily, byte[] colQualifier, KeyValue
> kv)
> >>>>    throws IOException {
> >>>>  if (kv == null || kv.getValueLength() != Bytes.SIZEOF_LONG)
> >>>>    return null;
> >>>>  return Bytes.toLong(kv.getBuffer(), kv.getValueOffset());
> >>>>
> >>>> Look at the if statement above carefully.
> >>>> If it doesn't match how you store values in HBase, feel free to
> subclass
> >>>> LongColumnInterpreter and provide the correct interpretation.
> >>>>
> >>>> BTW you don't need to restart cluster just because you need to use
> your
> >> own
> >>>> interpreter :-)
> >>>>
> >>>> On Tue, Jan 3, 2012 at 9:48 AM, Royston Sellman <
> >>>> royston.sellman@googlemail.com> wrote:
> >>>>
> >>>>> Hi Ted,
> >>>>>
> >>>>> Here is the output. As you can see aClient is not nul:
> >>>>>
> >>>>> AggregationClient aClient = new AggregationClient(conf);
> >>>>> System.err.println("aClient: "+aClient);
> >>>>>
> >>>>> <<<    aClient:
> >>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16
> >>>>>
> >>>>> It will take us a little while to add log code to LCI... we have to
> >>>>> edit the source, rebuild 0.92, redistribute round our cluster,
> restart
> >>>>> ;) We'll get back to you when this is done.
> >>>>>
> >>>>> Royston
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: Ted Yu [mailto:yuzhihong@gmail.com]
> >>>>> Sent: 03 January 2012 17:10
> >>>>> To: user@hbase.apache.org
> >>>>> Subject: Re: AggregateProtocol Help
> >>>>>
> >>>>> Royston:
> >>>>> Thanks for your effort trying to hunt down the problem.
> >>>>>
> >>>>> Can you add a log after this line to see if aClient is null ?
> >>>>>             AggregationClient aClient = new AggregationClient(conf);
> >>>>>
> >>>>> I was looking at LongColumnInterpreter.add() which is called by
> >>>>> aClient.sum()
> >>>>> Can you add a few log statements in LongColumnInterpreter.add() to
> see
> >>>>> what parameters are passed to it ?
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman <
> >>>>> royston.sellman@googlemail.com> wrote:
> >>>>>
> >>>>>> Hi Ted, Himanshu, Gary,
> >>>>>>
> >>>>>> Thanks again for your attention. I experimented with a shorter table
> >>>>>> and it looks like the timeout error was spurious...
> >>>>>>
> >>>>>> With the shorter table I now get an NPE when I call
> >>>>>> AggregationClient.sum().
> >>>>>> Here's the code snippet:
> >>>>>>
> >>>>>>              // Test the table
> >>>>>>              HTable table = new HTable(EDRP_TABLE);
> >>>>>>              Get get = new Get(Bytes.toBytes("row-aa"));
> >>>>>>              get.addColumn(Bytes.toBytes("EDRP"),
> >>>>>> Bytes.toBytes("advanceKWh"));
> >>>>>>              Result result = table.get(get);
> >>>>>>              byte [] val = result.getValue(Bytes.toBytes("EDRP"),
> >>>>>> Bytes.toBytes("advanceKWh"));
> >>>>>>              System.out.println("Row aa = " +
> >>>>>> Bytes.toString(val));
> >>>>>>
> >>>>>>              AggregationClient aClient = new
> AggregationClient(conf);
> >>>>>>              Scan scan = new Scan();
> >>>>>>               scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER);
> >>>>>>              scan.setStartRow(Bytes.toBytes("row-ab"));
> >>>>>>              scan.setStopRow(Bytes.toBytes("row-az"));
> >>>>>>              System.out.println(Bytes.toString(EDRP_FAMILY) + ":"
> >>>>>> + Bytes.toString(EDRP_QUALIFIER));
> >>>>>>               final ColumnInterpreter<Long, Long> ci = new
> >>>>>> LongColumnInterpreter();
> >>>>>>               long sum=-1;
> >>>>>>              try {
> >>>>>>                      sum = aClient.sum(EDRP_TABLE, ci, scan);
> >>>>>>              } catch (Throwable e) {
> >>>>>>                      // TODO Auto-generated catch block
> >>>>>>                      e.printStackTrace();
> >>>>>>              }
> >>>>>>              System.out.println(sum);
> >>>>>>
> >>>>>> The first part is just to check that my table is OK. It prints the
> >>>>>> correct value for row aa. Then I check CF:CQ is correct. Then I get
> >>>>>> the -1 that sum was defined as. Then the NPE. Here is the run
> output:
> >>>>>>
> >>>>>> Row aa = 3.0
> >>>>>> EDRP:advanceKWh
> >>>>>> -1
> >>>>>> java.lang.NullPointerException
> >>>>>> at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66)
> >>>>>> at EDRPAggregator.main(EDRPAggregator.java:96)
> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at
> >>>>>>
> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> >>>>>> .j
> >>>>>> ava:39
> >>>>>> )
> >>>>>> at
> >>>>>>
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> >>>>>> ss
> >>>>>> orImpl
> >>>>>> .java:25)
> >>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>>>
> >>>>>> Line 66 is the line:
> >>>>>>                      sum = aClient.sum(EDRP_TABLE, ci, scan);
> >>>>>>
> >>>>>> But I'm not sure that is really the line generating the NPE because
> >>>>>> I get the -1 output (same result when line 66 is not wrapped in
> >>>>>> try-catch
> >>>>> BTW).
> >>>>>> Anyway, -1 is clearly not the correct value for the sum.
> >>>>>>
> >>>>>> [Note: I'm setting a start and end row on the scan because I saw a
> >>>>>> comment in AggregationClient.java saying they must be set. However,
> >>>>>> I think this comment is obsolete as the test code in
> >>>>>> TestAggregateProtocol.java sometimes does not include start & end
> >>>>>> rows. The run is exactly the same without the start & stop rows ]
> >>>>>>
> >>>>>> We are struggling with Aggregations not least because
> >>>>>> TestAggregateProtocol.java is the only source of example code we
> >>>>>> could find and it is not ideal because it is designed to work with
> >>>>>> MiniCluster rather than a real cluster like ours. Is there any other
> >>>>>> example code in existence?
> >>>>>> I think that would really help us.
> >>>>>>
> >>>>>> Meanwhile, thanks for your patience with our questions. We are
> >>>>>> really impressed with the AggregationProtocol
> >>>>>> architecture/functionality and very keen to use it.
> >>>>>>
> >>>>>> Royston (and Tom)
> >>>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
> >>>>>> Sent: 02 January 2012 06:23
> >>>>>> To: user@hbase.apache.org
> >>>>>> Subject: Re: AggregateProtocol Help
> >>>>>>
> >>>>>> Hi Royston,
> >>>>>>
> >>>>>> Try increasing the value set for hbase.rpc.timeout (by default 60
> >>>>> seconds).
> >>>>>> Add something like this to hbase-site.xml:
> >>>>>>
> >>>>>> <property>
> >>>>>> <name>hbase.rpc.timeout</name>
> >>>>>> <value>180000</value>
> >>>>>> </property>
> >>>>>>
> >>>>>> This would increase the timeout value to 3 minutes, for example.
> >>>>>>
> >>>>>> But as Ted and Himanshu mention, 557k rows is not a large table and
> >>>>>> should not be taking very long to process, unless these happen to be
> >>>>>> very, very wide rows.  It sounds like there could be something else
> >>>>>> going on.  Does debug level logging show anything else in the region
> >>>>> server log?
> >>>>>>
> >>>>>> --gh
> >>>>>>
> >>>>>> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> >>>>>>> Thanks for the reminder Himanshu.
> >>>>>>>
> >>>>>>> Royston:
> >>>>>>> From this blog you can get some history on this subject:
> >>>>>>> http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoproce
> >>>>>>> ss
> >>>>>>> or
> >>>>>>> .html
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha
> >>>>>>> <hvashish@cs.ualberta.ca
> >>>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello Royston,
> >>>>>>>>
> >>>>>>>> Sorry to hear that you are getting trouble while using
> >>>>>>>> Aggregation functionalities.
> >>>>>>>>
> >>>>>>>> 557k rows seems to be a small table and a SocketTimeout does not
> >>>>>>>> seem to be an ok response.
> >>>>>>>> It will be good to know the region distribution as such. (how
> >>>>>>>> many
> >>>>>> regions?
> >>>>>>>> Is it a full table scan?)
> >>>>>>>>
> >>>>>>>> You are using the sum function; how are you using the
> >>>>> ColumnInterpreter.
> >>>>>>>> Can you enable the log level to debug to see why the RS is taking
> >>>>>>>> that long to respond (more than 113 sec).
> >>>>>>>> The 0 return value is the default result.
> >>>>>>>>
> >>>>>>>> Thanks for trying this out.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Himanshu
> >>>>>>>>
> >>>>>>>> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman <
> >>>>>>>> royston.sellman@googlemail.com> wrote:
> >>>>>>>>
> >>>>>>>>> Hi Ted,
> >>>>>>>>>
> >>>>>>>>> I think 0 is the only value we ever see (I'll check tomorrow:
> >>>>>>>>> the server is down right now). Our table has 557,000 rows. I'll
> >>>>>>>>> try a much shorter table tomorrow.
> >>>>>>>>>
> >>>>>>>>> Yes, we have RS running on the NN, but it's a test cluster and
> >>>>>>>>> we are
> >>>>>>>> used
> >>>>>>>>> to it :)
> >>>>>>>>>
> >>>>>>>>> Do you think using AggregationProtocol is the best strategy for
> >>>>>>>>> the case where we want to use basic SQL-style functions like
> >>>>>>>>> SUM, AVG, STD, MIN, MAX? Do you think there is a better strategy?
> >>>>>>>>>
> >>>>>>>>> Many thanks,
> >>>>>>>>> Royston
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 1 Jan 2012, at 17:58, Ted Yu wrote:
> >>>>>>>>>
> >>>>>>>>>> Royston:
> >>>>>>>>>> Happy New Year to you too.
> >>>>>>>>>>
> >>>>>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>>>>> namenode/10.0.0.235:60020
> >>>>>>>> failed
> >>>>>>>>> on
> >>>>>>>>>>
> >>>>>>>>>> It seems the namenode above actually refers to a region server.
> >>>>>>>>>> This
> >>>>>>>> is a
> >>>>>>>>>> little bit confusing :-)
> >>>>>>>>>>
> >>>>>>>>>> The sum value below is 0.
> >>>>>>>>>> Have you ever seen a value greater than 0 ?
> >>>>>>>>>>
> >>>>>>>>>> How many rows are there in this CF:CQ ?
> >>>>>>>>>> The timeout was reported earlier by other people where
> >>>>>>>>>> there're many
> >>>>>>>> rows
> >>>>>>>>>> in the table.
> >>>>>>>>>>
> >>>>>>>>>> There is a JIRA to provide streaming support for coprocessor
> >>>>>>>>>> but the development there has stalled.
> >>>>>>>>>>
> >>>>>>>>>> Cheers
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman <
> >>>>>>>>>> royston.sellman@googlemail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Hi Gary and Ted,
> >>>>>>>>>>>
> >>>>>>>>>>> Royston (Tom's colleague) here. Back onto this after the
> >>>>>>>>>>> Christmas/New
> >>>>>>>>> Year
> >>>>>>>>>>> break.
> >>>>>>>>>>>
> >>>>>>>>>>> Many thanks for your help so far. We enabled our database
> >>>>>>>>>>> via your hbase-site.xml mod and were able to move on. to
> >>>>>>>>>>> other errors. But I
> >>>>>>>>> think
> >>>>>>>>>>> we
> >>>>>>>>>>> are now actually getting an aggregation partially calculated
> >>>>>>>>>>> on our
> >>>>>>>>> table
> >>>>>>>>>>> (this feels like progress). The details:
> >>>>>>>>>>>
> >>>>>>>>>>> On running our client we now get this exception:
> >>>>>>>>>>> 11/12/31 17:51:09 WARN
> >>>>>>>>>>> client.HConnectionManager$HConnectionImplementation: Error
> >>>>>>>>>>> executing
> >>>>>>>> for
> >>>>>>>>>>> row
> >>>>>>>>>>>
> >>>>>>>>>>> java.util.concurrent.ExecutionException:
> >>>>>>>>>>> org.apache.hadoop.hbase.client.RetriesExhaustedException:
> >>>>>>>>>>> Failed after attempts=10, exceptions:
> >>>>>>>>>>> Sat Dec 31 17:41:30 GMT 2011,
> >>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >>>>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>>>> namenode/10.0.0.235:60020
> >>>>>>>> failed
> >>>>>>>>>>> on
> >>>>>>>>>>> socket timeout exception: java.net.SocketTimeoutException:
> >>>>>>>>>>> 60000
> >>>>>>>> millis
> >>>>>>>>>>> timeout while waiting for channel to be ready for read. ch :
> >>>>>>>>>>> java.nio.channels.SocketChannel[connected
> >>>>>>>>>>> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020]
> >>>>>>>>>>> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09
> >>>>>>>>>>> GMT 2011,
> >>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >>>>>>>>>>> java.net.SocketTimeoutException: Call to
> >>>>>>>>>>> namenode/10.0.0.235:60020
> >>>>>>>> failed
> >>>>>>>>>>> on
> >>>>>>>>>>> socket timeout exception: java.net.SocketTimeoutException:
> >>>>>>>>>>> 60000
> >>>>>>>> millis
> >>>>>>>>>>> timeout while waiting for channel to be ready for read. ch :
> >>>>>>>>>>> java.nio.channels.SocketChannel[connected
> >>>>>>>>>>> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020]
> >>>>>>>>>>>
> >>>>>>>>>>>     at
> >>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:
> >>>>>>>>>>> 22
> >>>>>>>>>>> 2)
> >>>>>>>>>>>     at
> >>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> >>>>>>>>>>> processExecs(HConnectionManager.java:1465)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java
> >>>>>>>> :1
> >>>>>>>> 55
> >>>>>>>> 5)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
> >>>>>>>> Ag
> >>>>>>>> gr
> >>>>>>>> egation
> >>>>>>>>>>> Client.java:229)
> >>>>>>>>>>>     at
> >>>>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >>>>>>>>>>>     at EDRPAggregator.main(EDRPAggregator.java:77)
> >>>>>>>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>> Method)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>> java:39
> >>>>>>>>>>> )
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>>>> cc
> >>>>>>>> es
> >>>>>>>> sorImpl
> >>>>>>>>>>> .java:25)
> >>>>>>>>>>>     at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Looking at the log (.regionserver-namenode.log) I see this
> >>>>>>>>>>> debug
> >>>>>>>>> message:
> >>>>>>>>>>>
> >>>>>>>>>>> 2011-12-31 17:42:23,472 DEBUG
> >>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateImplementation:
> >>>>>>>>>>> Sum from
> >>>>>>>>> this
> >>>>>>>>>>> region is
> >>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.:
> >>>>>>>>> 0
> >>>>>>>>>>>
> >>>>>>>>>>> Where the sum value looks reasonable which makes me think
> >>>>>>>>>>> the sum of a CF:CQ worked. But I never see this value on
> >>>> stdout.
> >>>>>>>>>>> Then I see this warning:
> >>>>>>>>>>>
> >>>>>>>>>>> 2011-12-31 17:42:23,476 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>>>> (responseTooSlow):
> >>>>>>>>>>> {"processingtimems":113146,"call":"execCoprocess$
> >>>>>>>>>>> 2011-12-31 17:42:23,511 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>>>> IPC
> >>>>>>>>> Server
> >>>>>>>>>>> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$
> >>>>>>>>>>> 2011-12-31 17:42:23,515 WARN org.apache.hadoop.ipc.HBaseServer:
> >>>>>>>>>>> IPC
> >>>>>>>>> Server
> >>>>>>>>>>> handler 1 on 60020 caught: java.nio.channels.ClosedChann$
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.ja
> >>>>>>>> va
> >>>>>>>> :1
> >>>>>>>> 33)
> >>>>>>>>>>>     at
> >>>>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.
> >>>>>>>> ja
> >>>>>>>> va
> >>>>>>>> :1651)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse
> >>>>>>>> (H
> >>>>>>>> Ba
> >>>>>>>> seServe
> >>>>>>>>>>> r.java:924)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBase
> >>>>>>>> Se
> >>>>>>>> rv
> >>>>>>>> er.java
> >>>>>>>>>>> :1003)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(
> >>>>>>>> HB
> >>>>>>>> as
> >>>>>>>> eServer
> >>>>>>>>>>> .java:409)
> >>>>>>>>>>>     at
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>> 1345)
> >>>>>>>>>>>
> >>>>>>>>>>> Have we missed out some step in the HBase/RegionServerconfig?
> >>>>>>>>>>> Or is
> >>>>>>>> our
> >>>>>>>>>>> client code still deficient?
> >>>>>>>>>>>
> >>>>>>>>>>> Can you offer any suggestions? Is there any example code for
> >>>>>>>>>>> the new Aggregations stuff.
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks and Happy New Year to you guys,
> >>>>>>>>>>>
> >>>>>>>>>>> Royston (and Tom).
> >>>>>>>>>>>
> >>>>>>>>>>> (HBase 0.92, Hadoop 1.0)
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
> >>>>>>>>>>> Sent: 23 December 2011 18:06
> >>>>>>>>>>> To: user@hbase.apache.org
> >>>>>>>>>>> Subject: Re: AggregateProtocol Help
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Tom,
> >>>>>>>>>>>
> >>>>>>>>>>> The test code is not really the best guide for configuration.
> >>>>>>>>>>>
> >>>>>>>>>>> To enable the AggregateProtocol on all of your tables, add
> >>>>>>>>>>> this to the hbase-site.xml for the servers in your cluster:
> >>>>>>>>>>>
> >>>>>>>>>>> <property>
> >>>>>>>>>>> <name>hbase.coprocessor.user.region.classes</name>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementatio
> >>>>>>>> n<
> >>>>>>>> /v
> >>>>>>>> alue>
> >>>>>>>>>>> </property>
> >>>>>>>>>>>
> >>>>>>>>>>> If you only want to use the aggregate functions on a
> >>>>>>>>>>> specific table
> >>>>>>>> (or
> >>>>>>>>>>> tables), then you can enable that individually for the table
> >>>>>>>>>>> from the
> >>>>>>>>>>> shell:
> >>>>>>>>>>>
> >>>>>>>>>>> 1) disable the table
> >>>>>>>>>>> hbase> disable 'EDRP7'
> >>>>>>>>>>>
> >>>>>>>>>>> 2) add the coprocessor
> >>>>>>>>>>> hbase> alter 'EDRP7', METHOD => 'table_att',
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImp
> >>>>>>>> le
> >>>>>>>> me
> >>>>>>>> ntation
> >>>>>>>>>>> ||'
> >>>>>>>>>>>
> >>>>>>>>>>> (Note that the pipes in the value string are required)
> >>>>>>>>>>>
> >>>>>>>>>>> 3) re-enable the table
> >>>>>>>>>>> hbase> enable 'EDRP7'
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Either way should work.  With the second approach you will
> >>>>>>>>>>> see the coprocessor listed when you describe the table from
> >>>>>>>>>>> the shell, as Ted mentioned.  With the first approach you
> >>>>>>>>>>> will not, but it should be
> >>>>>>>>> loaded
> >>>>>>>>>>> all the same.
> >>>>>>>>>>>
> >>>>>>>>>>> --gh
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu
> >>>>>>>>>>> <yuzhihong@gmail.com>
> >>>>>> wrote:
> >>>>>>>>>>>> I don't know why you chose HBaseTestingUtility to create
> >>>>>>>>>>>> the
> >>>>>> table.
> >>>>>>>>>>>> I guess you followed test code example.
> >>>>>>>>>>>>
> >>>>>>>>>>>> At least you should pass the conf to this ctor:
> >>>>>>>>>>>> public HBaseTestingUtility(Configuration conf) {
> >>>>>>>>>>>>
> >>>>>>>>>>>> If coprocessor was installed correctly, you should see
> >>>>>>>>>>>> something like(from
> >>>>>>>>>>>> HBASE-5070):
> >>>>>>>>>>>> coprocessor$1 =>
> >>>>>>>>>>>>
> >>>>>>
> '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox
> >>>>>>>>>>>> <Tom.Wilcox@cse.org.uk>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I am not sure how we load the AggregateImplementation into
> >>>>>>>>>>>>> the
> >>>>>>>> table.
> >>>>>>>>>>>>> When we are creating a table, we use the same functions as
> >>>>>>>>>>>>> the test
> >>>>>>>> as
> >>>>>>>>>>> follows...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >>>>>>>>>>>>>> );
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           // Utility.CreateHBaseTable(conf,
> >>>>>>>>>>>>>> otherArgs[1],
> >>>>>>>>>>>>> otherArgs[2],
> >>>>>>>>>>>>>> true);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           HBaseTestingUtility util = new
> >>>>>>>>>>>>>> HBaseTestingUtility();
> >>>>>>>>>>>>>>           HTable table = util.createTable(EDRP_TABLE,
> >>>>>>>>>>>>>> EDRP_FAMILY);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           AggregationClient aClient = new
> >>>>>>>>>>>>>> AggregationClient(conf);
> >>>>>>>>>>>>> ...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Running DESCRIBE on a table produced shows the following
> >>>>> output:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hbase(main):002:0> describe 'EDRP7'
> >>>>>>>>>>>>> DESCRIPTION
> >>>>>>>>>>>>>                                         ENABLED {NAME =>
> >>>>>>>>>>>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER
> >>>>>>>>>>>>> => 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>  true
> >>>>>>>>>>>>> '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =>
> >>>>>>>>>>>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false',
> >>>>>>>>>>>>> B LOCKCACHE => 'true'}]}
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We are using the tip of 0.92 (cloned from the Git repo).
> >>>>>>>>>>>>> See the version string below:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> hbase(main):005:0> version 0.92.0, r1208286, Thu Dec 15
> >>>>>>>>>>>>> 13:16:03 GMT 2011
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We would really appreciate an example of how to create a
> >>>>>>>>>>>>> table that is enabled to handle Aggregation).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ________________________________________
> >>>>>>>>>>>>> From: Ted Yu [yuzhihong@gmail.com]
> >>>>>>>>>>>>> Sent: 22 December 2011 17:03
> >>>>>>>>>>>>> To: user@hbase.apache.org
> >>>>>>>>>>>>> Subject: Re: AggregateProtocol Help
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Have you loaded AggregateImplementation into your table ?
> >>>>>>>>>>>>> Can you show us the contents of the following command in
> >>>>>>>>>>>>> hbase
> >>>>>>>> shell:
> >>>>>>>>>>>>> describe 'your-table'
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> BTW are you using the tip of 0.92 ?
> >>>>>>>>>>>>> HBASE-4946 would be of help for dynamically loaded
> >>>>>>>>>>>>> coprocessors
> >>>>>>>> which
> >>>>>>>>>>>>> you might use in the future.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cheers
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox
> >>>>>>>>>>>>> <Tom.Wilcox@cse.org.uk>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> We are trying to use the aggregation functionality in
> >>>>>>>>>>>>>> HBase
> >>>>>>>>>>>>>> 0.92 and we have managed to get the test code working
> >>>>>>>>>>>>>> using the
> >>>>>>>> following
> >>>>>>>>>>> command:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
> >>>>>>>>>>>>>> org.junit.runner.JUnitCore
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Closer inspection of this test class has revealed that it
> >>>>>>>>>>>>>> uses a mini DFS cluster to populate and run the tests.
> >>>>>>>>>>>>>> These tests return
> >>>>>>>>>>> successfully.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> However, when we attempt to run similar code on our
> >>>>>>>>>>>>>> development HDFS cluster we experience the following error:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 11/12/22 15:46:28 WARN
> >>>>>>>>>>>>>> client.HConnectionManager$HConnectionImplementation:
> >>>>>>>>>>>>>> Error executing for
> >>>>>>>>>>>>> row
> >>>>>>>>>>>>>> java.util.concurrent.ExecutionException:
> >>>>>>>>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>>>> ti
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>> : No
> >>>>>>>>>>>>> matching
> >>>>>>>>>>>>>> handler for protocol
> >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>>>> in region
> >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>>>> .j
> >>>>>>>>>>>>>> av
> >>>>>>>>>>>>>> a:4010
> >>>>>>>>>>>>>> )
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>>>> r(
> >>>>>>>> HR
> >>>>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>>>> java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>>>> cc
> >>>>>>>> es
> >>>>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>>>> eR
> >>>>>>>> pc
> >>>>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>>>> 1325)
> >>>>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.
> >>>>>>>>>>>>>> ja
> >>>>>>>>>>>>>> va
> >>>>>>>>>>>>>> :222)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation.processExecs(HConnectionManager.java:1465)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.
> >>>>>>>>>>>>>> java:1
> >>>>>>>>>>>>>> 555)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
> >>>>>>>> Ag
> >>>>>>>> gr
> >>>>>>>>>>>>> egationClient.java:229)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >>>>>>>>>>>>>> at EDRPAggregator.main(EDRPAggregator.java:77)
> >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>>>> java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>>>> cc
> >>>>>>>> es
> >>>>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >>>>>>>>>>>>>> Caused by:
> >>>>>>>>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>>>> ti
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>> : No
> >>>>>>>>>>>>> matching
> >>>>>>>>>>>>>> handler for protocol
> >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>>>> in region
> >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>>>> .j
> >>>>>>>>>>>>>> av
> >>>>>>>>>>>>>> a:4010
> >>>>>>>>>>>>>> )
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>>>> r(
> >>>>>>>> HR
> >>>>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>>>> java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>>>> cc
> >>>>>>>> es
> >>>>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>>>> eR
> >>>>>>>> pc
> >>>>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>>>> 1325)
> >>>>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
> >>>>>>>>>>>>>> ti
> >>>>>>>>>>>>>> ve
> >>>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
> >>>>>>>> ru
> >>>>>>>> ct
> >>>>>>>>>>>>> orAccessorImpl.java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
> >>>>>>>> in
> >>>>>>>> gC
> >>>>>>>>>>>>> onstructorAccessorImpl.java:27)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >>>>>>>>>>>>>> 51
> >>>>>>>>>>>>>> 3)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
> >>>>>>>> io
> >>>>>>>> n(
> >>>>>>>>>>>>> RemoteExceptionHandler.java:96)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation.translateException(HConnectionManager.java:1651)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java:
> >>>>>>>>>>>>> 13
> >>>>>>>>>>>>> 27
> >>>>>>>>>>>>> )
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
> >>>>>>>>>>>>>> In
> >>>>>>>>>>>>>> vo
> >>>>>>>>>>>>>> ker.ja
> >>>>>>>>>>>>>> va:79)
> >>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>>>> ll
> >>>>>>>> (A
> >>>>>>>>>>>>> ggregationClient.java:233)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>>>> ll
> >>>>>>>> (A
> >>>>>>>>>>>>> ggregationClient.java:230)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
> >>>>>>>>>>>>>> ja
> >>>>>>>>>>>>>> va
> >>>>>>>>>>>>>> :303)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
> >>>>>>>> Ex
> >>>>>>>> ec
> >>>>>>>>>>>>> utor.java:886)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
> >>>>>>>> ut
> >>>>>>>> or
> >>>>>>>>>>>>> .java:908)
> >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
> >>>>>>>>>>>>>> ti
> >>>>>>>>>>>>>> on
> >>>>>>>>>>>>>> : No
> >>>>>>>>>>>>> matching
> >>>>>>>>>>>>>> handler for protocol
> >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >>>>>>>>>>>>>> in region
> >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
> >>>>>>>>>>>>>> .j
> >>>>>>>>>>>>>> av
> >>>>>>>>>>>>>> a:4010
> >>>>>>>>>>>>>> )
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
> >>>>>>>> r(
> >>>>>>>> HR
> >>>>>>>>>>>>> egionServer.java:3040)
> >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >>>>>>>>>>>>> java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
> >>>>>>>> cc
> >>>>>>>> es
> >>>>>>>>>>>>> sorImpl.java:25)
> >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
> >>>>>>>> eR
> >>>>>>>> pc
> >>>>>>>>>>>>> Engine.java:364)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >>>>>>>>>>>>> 1325)
> >>>>>>>>>>>>>> [sshexec]
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
> >>>>>>>>>>>>>> ti
> >>>>>>>>>>>>>> ve
> >>>>>>>>>>>>>> Method)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
> >>>>>>>> ru
> >>>>>>>> ct
> >>>>>>>>>>>>> orAccessorImpl.java:39)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
> >>>>>>>> in
> >>>>>>>> gC
> >>>>>>>>>>>>> onstructorAccessorImpl.java:27)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >>>>>>>>>>>>>> 51
> >>>>>>>>>>>>>> 3)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
> >>>>>>>> io
> >>>>>>>> n(
> >>>>>>>>>>>>> RemoteExceptionHandler.java:96)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation.translateException(HConnectionManager.java:1651)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java:
> >>>>>>>>>>>>> 13
> >>>>>>>>>>>>> 27
> >>>>>>>>>>>>> )
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
> >>>>>>>>>>>>>> In
> >>>>>>>>>>>>>> vo
> >>>>>>>>>>>>>> ker.ja
> >>>>>>>>>>>>>> va:79)
> >>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>>>> ll
> >>>>>>>> (A
> >>>>>>>>>>>>> ggregationClient.java:233)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
> >>>>>>>> ll
> >>>>>>>> (A
> >>>>>>>>>>>>> ggregationClient.java:230)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
> >>>>>>>> em
> >>>>>>>> en
> >>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
> >>>>>>>>>>>>>> ja
> >>>>>>>>>>>>>> va
> >>>>>>>>>>>>>> :303)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
> >>>>>>>> Ex
> >>>>>>>> ec
> >>>>>>>>>>>>> utor.java:886)
> >>>>>>>>>>>>>> at
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
> >>>>>>>> ut
> >>>>>>>> or
> >>>>>>>>>>>>> .java:908)
> >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> The source for our class is:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> import static org.junit.Assert.assertEquals;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> import java.io.IOException;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> import org.apache.hadoop.conf.Configuration;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseTestingUtility;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HConstants;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HTableDescriptor;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.HTable;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Put;
> >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Scan;
> >>>>>>>>>>>>>> import
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationCli
> >>>>>>>>>>>>>> en
> >>>>>>>>>>>>>> t;
> >>>>>>>>>>>>>> import
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInte
> >>>>>>>>>>>>>> rp re ter; import org.apache.hadoop.hbase.util.Bytes;
> >>>>>>>>>>>>>> import org.apache.hadoop.util.GenericOptionsParser;
> >>>>>>>>>>>>>> import
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
> >>>>>>>>>>>>>> import
> >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
> >>>>>>>>>>>>>> import org.junit.Test;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> public class EDRPAggregator {
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    // private static final byte[] EDRP_FAMILY =
> >>>>>>>>>>>>>> Bytes.toBytes("EDRP");
> >>>>>>>>>>>>>>    // private static final byte[] EDRP_QUALIFIER =
> >>>>>>>>>>>>>> Bytes.toBytes("advanceKWh");
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    private static byte[] ROW = Bytes.toBytes("testRow");
> >>>>>>>>>>>>>>    private static final int ROWSIZE = 20;
> >>>>>>>>>>>>>>    private static byte[][] ROWS = makeN(ROW, ROWSIZE);
> >>>>>>>>>>>>>>    private static final byte[] TEST_QUALIFIER =
> >>>>>>>>>>>>>> Bytes.toBytes("TestQualifier");
> >>>>>>>>>>>>>>    private static final byte[] TEST_MULTI_CQ =
> >>>>>>>>>>>>>> Bytes.toBytes("TestMultiCQ");
> >>>>>>>>>>>>>>    private static final int rowSeperator1 = 5;
> >>>>>>>>>>>>>>      private static final int rowSeperator2 = 12;
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    public static void
> >>>>>>>>>>>>>> testSumWithValidRange(Configuration
> >>>>>>>>>>>>>> conf,
> >>>>>>>>>>>>>>                  String[] otherArgs) throws Throwable {
> >>>>>>>>>>>>>>           byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
> >>>>>>>>>>>>>>           byte[] EDRP_FAMILY =
> >>>>>>>>>>>>>> Bytes.toBytes(otherArgs[2]);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >>>>>>>>>>>>>> );
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           // Utility.CreateHBaseTable(conf,
> >>>>>>>>>>>>>> otherArgs[1],
> >>>>>>>>>>>>> otherArgs[2],
> >>>>>>>>>>>>>> true);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           HBaseTestingUtility util = new
> >>>>>>>>>>>>>> HBaseTestingUtility();
> >>>>>>>>>>>>>>           HTable table = util.createTable(EDRP_TABLE,
> >>>>>>>>>>>>>> EDRP_FAMILY);
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           AggregationClient aClient = new
> >>>>>>>>>>>>>> AggregationClient(conf);
> >>>>>>>>>>>>>>           Scan scan = new Scan();
> >>>>>>>>>>>>>>           scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
> >>>>>>>>>>>>>>           final ColumnInterpreter<Long, Long> ci = new
> >>>>>>>>>>>>>> LongColumnInterpreter();
> >>>>>>>>>>>>>>           long sum =
> >>>>>>>>>>>>>> aClient.sum(Bytes.toBytes(otherArgs[0]),
> >>>>>>>>>>>>>> ci,
> >>>>>>>>>>>>> scan);
> >>>>>>>>>>>>>>           System.out.println(sum);
> >>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    /**
> >>>>>>>>>>>>>>    * Main entry point.
> >>>>>>>>>>>>>>    *
> >>>>>>>>>>>>>>     * @param argsThe
> >>>>>>>>>>>>>>    *            command line parameters.
> >>>>>>>>>>>>>>    * @throws Exception
> >>>>>>>>>>>>>>    *             When running the job fails.
> >>>>>>>>>>>>>>    */
> >>>>>>>>>>>>>>    public static void main(String[] args) throws
> >>>>>>>>>>>>>> Exception {
> >>>>>>>>>>>>>>           Configuration conf =
> >>>>>>>>>>>>>> HBaseConfiguration.create();
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           String[] otherArgs = new
> >>>>>>>>>>>>>> GenericOptionsParser(conf,
> >>>>>>>>>>>>>> args)
> >>>>>>>>>>>>>>                        .getRemainingArgs();
> >>>>>>>>>>>>>>           if (otherArgs.length != 3) {
> >>>>>>>>>>>>>>                  System.err
> >>>>>>>>>>>>>>                               .println("Wrong number of
> >>>>>>>>>>>>>> arguments: " + otherArgs.length);
> >>>>>>>>>>>>>>                  System.err.println("Usage: " +
> >>>>>>>>>>>>>> "<tablename> <colfam> <qualifier>");
> >>>>>>>>>>>>>>                  System.exit(-1);
> >>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>           try {
> >>>>>>>>>>>>>>                  testSumWithValidRange(conf, otherArgs);
> >>>>>>>>>>>>>>           } catch (Throwable e) {
> >>>>>>>>>>>>>>                  e.printStackTrace();
> >>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>    /**
> >>>>>>>>>>>>>>    * an infrastructure method to prepare rows for the
> >>>>>> testtable.
> >>>>>>>>>>>>>>    *
> >>>>>>>>>>>>>>     * @param base
> >>>>>>>>>>>>>>    * @param n
> >>>>>>>>>>>>>>    * @return
> >>>>>>>>>>>>>>    */
> >>>>>>>>>>>>>>    private static byte[][] makeN(byte[] base, int n) {
> >>>>>>>>>>>>>>           byte[][] ret = new byte[n][];
> >>>>>>>>>>>>>>           for (int i = 0; i < n; i++) {
> >>>>>>>>>>>>>>                  ret[i] = Bytes.add(base,
> >>>>>>>>>>>>>> Bytes.toBytes(i));
> >>>>>>>>>>>>>>           }
> >>>>>>>>>>>>>>           return ret;
> >>>>>>>>>>>>>>    }
> >>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please can you suggest what might be causing and/or how
> >>>>>>>>>>>>>> we might fix this UnknownProtocolException?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, does anyone have any working examples using the
> >>>>>>>>>>>>>> aggregation
> >>>>>>>>>>>>> protocol
> >>>>>>>>>>>>>> other than the test code?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Tom
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message