hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshu Vashishtha <hvash...@cs.ualberta.ca>
Subject Re: AggregateProtocol Help
Date Tue, 03 Jan 2012 17:11:08 GMT
Hello Royston (and Tom),

Its a bit confusing. -1 is defined in your code, and the default value
(in case it was returning just default) from the RS is 0.
Can you enable DEBUG at Region Server and paste it to pastebin.
I assume the conf object is correctly configured?

Thanks,
Himanshu


On Tue, Jan 3, 2012 at 9:32 AM, Royston Sellman
<royston.sellman@googlemail.com> wrote:
> Hi Ted, Himanshu, Gary,
>
> Thanks again for your attention. I experimented with a shorter table and it
> looks like the timeout error was spurious...
>
> With the shorter table I now get an NPE when I call AggregationClient.sum().
> Here's the code snippet:
>
>                // Test the table
>                HTable table = new HTable(EDRP_TABLE);
>                Get get = new Get(Bytes.toBytes("row-aa"));
>                get.addColumn(Bytes.toBytes("EDRP"),
> Bytes.toBytes("advanceKWh"));
>                Result result = table.get(get);
>                byte [] val = result.getValue(Bytes.toBytes("EDRP"),
> Bytes.toBytes("advanceKWh"));
>                System.out.println("Row aa = " + Bytes.toString(val));
>
>                AggregationClient aClient = new AggregationClient(conf);
>                Scan scan = new Scan();
>                scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER);
>                scan.setStartRow(Bytes.toBytes("row-ab"));
>                scan.setStopRow(Bytes.toBytes("row-az"));
>                System.out.println(Bytes.toString(EDRP_FAMILY) + ":" +
> Bytes.toString(EDRP_QUALIFIER));
>                final ColumnInterpreter<Long, Long> ci = new
> LongColumnInterpreter();
>                long sum=-1;
>                try {
>                        sum = aClient.sum(EDRP_TABLE, ci, scan);
>                } catch (Throwable e) {
>                        // TODO Auto-generated catch block
>                        e.printStackTrace();
>                }
>                System.out.println(sum);
>
> The first part is just to check that my table is OK. It prints the correct
> value for row aa. Then I check CF:CQ is correct. Then I get the -1 that sum
> was defined as. Then the NPE. Here is the run output:
>
> Row aa = 3.0
> EDRP:advanceKWh
> -1
> java.lang.NullPointerException
>  at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66)
>  at EDRPAggregator.main(EDRPAggregator.java:96)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
> )
>  at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>  at java.lang.reflect.Method.invoke(Method.java:597)
>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Line 66 is the line:
>                        sum = aClient.sum(EDRP_TABLE, ci, scan);
>
> But I'm not sure that is really the line generating the NPE because I get
> the -1 output (same result when line 66 is not wrapped in try-catch BTW).
> Anyway, -1 is clearly not the correct value for the sum.
>
> [Note: I'm setting a start and end row on the scan because I saw a comment
> in AggregationClient.java saying they must be set. However, I think this
> comment is obsolete as the test code in TestAggregateProtocol.java sometimes
> does not include start & end rows. The run is exactly the same without the
> start & stop rows ]
>
> We are struggling with Aggregations not least because
> TestAggregateProtocol.java is the only source of example code we could find
> and it is not ideal because it is designed to work with MiniCluster rather
> than a real cluster like ours. Is there any other example code in existence?
> I think that would really help us.
>
> Meanwhile, thanks for your patience with our questions. We are really
> impressed with the AggregationProtocol architecture/functionality and very
> keen to use it.
>
> Royston (and Tom)
>
> -----Original Message-----
> From: Gary Helmling [mailto:ghelmling@gmail.com]
> Sent: 02 January 2012 06:23
> To: user@hbase.apache.org
> Subject: Re: AggregateProtocol Help
>
> Hi Royston,
>
> Try increasing the value set for hbase.rpc.timeout (by default 60 seconds).
> Add something like this to hbase-site.xml:
>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>180000</value>
> </property>
>
> This would increase the timeout value to 3 minutes, for example.
>
> But as Ted and Himanshu mention, 557k rows is not a large table and should
> not be taking very long to process, unless these happen to be very, very
> wide rows.  It sounds like there could be something else going on.  Does
> debug level logging show anything else in the region server log?
>
> --gh
>
> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>> Thanks for the reminder Himanshu.
>>
>> Royston:
>> From this blog you can get some history on this subject:
>> http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoprocessor
>> .html
>>
>>
>> On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha
>> <hvashish@cs.ualberta.ca
>>> wrote:
>>
>>> Hello Royston,
>>>
>>> Sorry to hear that you are getting trouble while using Aggregation
>>> functionalities.
>>>
>>> 557k rows seems to be a small table and a SocketTimeout does not seem
>>> to be an ok response.
>>> It will be good to know the region distribution as such. (how many
> regions?
>>> Is it a full table scan?)
>>>
>>> You are using the sum function; how are you using the ColumnInterpreter.
>>> Can you enable the log level to debug to see why the RS is taking
>>> that long to respond (more than 113 sec).
>>> The 0 return value is the default result.
>>>
>>> Thanks for trying this out.
>>>
>>> Thanks,
>>> Himanshu
>>>
>>> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman <
>>> royston.sellman@googlemail.com> wrote:
>>>
>>> > Hi Ted,
>>> >
>>> > I think 0 is the only value we ever see (I'll check tomorrow: the
>>> > server is down right now). Our table has 557,000 rows. I'll try a
>>> > much shorter table tomorrow.
>>> >
>>> > Yes, we have RS running on the NN, but it's a test cluster and we
>>> > are
>>> used
>>> > to it :)
>>> >
>>> > Do you think using AggregationProtocol is the best strategy for the
>>> > case where we want to use basic SQL-style functions like SUM, AVG,
>>> > STD, MIN, MAX? Do you think there is a better strategy?
>>> >
>>> > Many thanks,
>>> > Royston
>>> >
>>> >
>>> > On 1 Jan 2012, at 17:58, Ted Yu wrote:
>>> >
>>> > > Royston:
>>> > > Happy New Year to you too.
>>> > >
>>> > >>> java.net.SocketTimeoutException: Call to
>>> > >>> namenode/10.0.0.235:60020
>>> failed
>>> > on
>>> > >
>>> > > It seems the namenode above actually refers to a region server.
>>> > > This
>>> is a
>>> > > little bit confusing :-)
>>> > >
>>> > > The sum value below is 0.
>>> > > Have you ever seen a value greater than 0 ?
>>> > >
>>> > > How many rows are there in this CF:CQ ?
>>> > > The timeout was reported earlier by other people where there're
>>> > > many
>>> rows
>>> > > in the table.
>>> > >
>>> > > There is a JIRA to provide streaming support for coprocessor but
>>> > > the development there has stalled.
>>> > >
>>> > > Cheers
>>> > >
>>> > > On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman <
>>> > > royston.sellman@googlemail.com> wrote:
>>> > >
>>> > >> Hi Gary and Ted,
>>> > >>
>>> > >> Royston (Tom's colleague) here. Back onto this after the
>>> > >> Christmas/New
>>> > Year
>>> > >> break.
>>> > >>
>>> > >> Many thanks for your help so far. We enabled our database via
>>> > >> your hbase-site.xml mod and were able to move on. to other
>>> > >> errors. But I
>>> > think
>>> > >> we
>>> > >> are now actually getting an aggregation partially calculated on
>>> > >> our
>>> > table
>>> > >> (this feels like progress). The details:
>>> > >>
>>> > >> On running our client we now get this exception:
>>> > >> 11/12/31 17:51:09 WARN
>>> > >> client.HConnectionManager$HConnectionImplementation: Error
>>> > >> executing
>>> for
>>> > >> row
>>> > >>
>>> > >>  java.util.concurrent.ExecutionException:
>>> > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
>>> > >> after attempts=10, exceptions:
>>> > >> Sat Dec 31 17:41:30 GMT 2011,
>>> > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
>>> > >> java.net.SocketTimeoutException: Call to
>>> > >> namenode/10.0.0.235:60020
>>> failed
>>> > >> on
>>> > >> socket timeout exception: java.net.SocketTimeoutException: 60000
>>> millis
>>> > >> timeout while waiting for channel to be ready for read. ch :
>>> > >> java.nio.channels.SocketChannel[connected
>>> > >> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020]
>>> > >> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09 GMT
>>> > >> 2011, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
>>> > >> java.net.SocketTimeoutException: Call to
>>> > >> namenode/10.0.0.235:60020
>>> failed
>>> > >> on
>>> > >> socket timeout exception: java.net.SocketTimeoutException: 60000
>>> millis
>>> > >> timeout while waiting for channel to be ready for read. ch :
>>> > >> java.nio.channels.SocketChannel[connected
>>> > >> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020]
>>> > >>
>>> > >>       at
>>> > >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:22
>>> > >> 2)
>>> > >>       at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
>>> > >> processExecs(HConnectionManager.java:1465)
>>> > >>       at
>>> > >>
>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:155
>>> 5)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(Aggr
>>> egation
>>> > >> Client.java:229)
>>> > >>       at
>>> > >> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
>>> > >>       at EDRPAggregator.main(EDRPAggregator.java:77)
>>> > >>       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> > >> Method)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> java:39
>>> > >> )
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> sorImpl
>>> > >> .java:25)
>>> > >>       at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >>       at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > >>
>>> > >>
>>> > >> Looking at the log (.regionserver-namenode.log) I see this debug
>>> > message:
>>> > >>
>>> > >> 2011-12-31 17:42:23,472 DEBUG
>>> > >> org.apache.hadoop.hbase.coprocessor.AggregateImplementation: Sum
>>> > >> from
>>> > this
>>> > >> region is
>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.:
>>> > 0
>>> > >>
>>> > >> Where the sum value looks reasonable which makes me think the
>>> > >> sum of a CF:CQ worked. But I never see this value on stdout.
>>> > >> Then I see this warning:
>>> > >>
>>> > >> 2011-12-31 17:42:23,476 WARN org.apache.hadoop.ipc.HBaseServer:
>>> > >> (responseTooSlow):
>>> > >> {"processingtimems":113146,"call":"execCoprocess$
>>> > >> 2011-12-31 17:42:23,511 WARN org.apache.hadoop.ipc.HBaseServer:
>>> > >> IPC
>>> > Server
>>> > >> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$
>>> > >> 2011-12-31 17:42:23,515 WARN org.apache.hadoop.ipc.HBaseServer:
>>> > >> IPC
>>> > Server
>>> > >> handler 1 on 60020 caught: java.nio.channels.ClosedChann$
>>> > >>       at
>>> > >>
>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:1
>>> 33)
>>> > >>       at
>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>>> > >>       at
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.java
>>> :1651)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(HBa
>>> seServe
>>> > >> r.java:924)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseServ
>>> er.java
>>> > >> :1003)
>>> > >>       at
>>> > >>
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HBas
>>> eServer
>>> > >> .java:409)
>>> > >>       at
>>> > >>
>>> >
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>> 1345)
>>> > >>
>>> > >> Have we missed out some step in the HBase/RegionServerconfig? Or
>>> > >> is
>>> our
>>> > >> client code still deficient?
>>> > >>
>>> > >> Can you offer any suggestions? Is there any example code for the
>>> > >> new Aggregations stuff.
>>> > >>
>>> > >> Thanks and Happy New Year to you guys,
>>> > >>
>>> > >> Royston (and Tom).
>>> > >>
>>> > >> (HBase 0.92, Hadoop 1.0)
>>> > >>
>>> > >>
>>> > >> -----Original Message-----
>>> > >> From: Gary Helmling [mailto:ghelmling@gmail.com]
>>> > >> Sent: 23 December 2011 18:06
>>> > >> To: user@hbase.apache.org
>>> > >> Subject: Re: AggregateProtocol Help
>>> > >>
>>> > >> Hi Tom,
>>> > >>
>>> > >> The test code is not really the best guide for configuration.
>>> > >>
>>> > >> To enable the AggregateProtocol on all of your tables, add this
>>> > >> to the hbase-site.xml for the servers in your cluster:
>>> > >>
>>> > >> <property>
>>> > >>   <name>hbase.coprocessor.user.region.classes</name>
>>> > >>
>>> > >>
>>> >
>>> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation</v
>>> alue>
>>> > >> </property>
>>> > >>
>>> > >> If you only want to use the aggregate functions on a specific
>>> > >> table
>>> (or
>>> > >> tables), then you can enable that individually for the table
>>> > >> from the
>>> > >> shell:
>>> > >>
>>> > >> 1) disable the table
>>> > >> hbase> disable 'EDRP7'
>>> > >>
>>> > >> 2) add the coprocessor
>>> > >> hbase> alter 'EDRP7', METHOD => 'table_att',
>>> > >>
>>> > >>
>>> > >>
>>> >
>>> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImpleme
>>> ntation
>>> > >> ||'
>>> > >>
>>> > >> (Note that the pipes in the value string are required)
>>> > >>
>>> > >> 3) re-enable the table
>>> > >> hbase> enable 'EDRP7'
>>> > >>
>>> > >>
>>> > >> Either way should work.  With the second approach you will see
>>> > >> the coprocessor listed when you describe the table from the
>>> > >> shell, as Ted mentioned.  With the first approach you will not,
>>> > >> but it should be
>>> > loaded
>>> > >> all the same.
>>> > >>
>>> > >> --gh
>>> > >>
>>> > >>
>>> > >> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>>> > >>> I don't know why you chose HBaseTestingUtility to create the table.
>>> > >>> I guess you followed test code example.
>>> > >>>
>>> > >>> At least you should pass the conf to this ctor:
>>> > >>> public HBaseTestingUtility(Configuration conf) {
>>> > >>>
>>> > >>> If coprocessor was installed correctly, you should see
>>> > >>> something like(from
>>> > >>> HBASE-5070):
>>> > >>> coprocessor$1 =>
>>> > >>>
> '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'
>>> > >>>
>>> > >>> Cheers
>>> > >>>
>>> > >>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox
>>> > >>> <Tom.Wilcox@cse.org.uk>
>>> > >> wrote:
>>> > >>>
>>> > >>>> Hi,
>>> > >>>>
>>> > >>>> I am not sure how we load the AggregateImplementation into the
>>> table.
>>> > >>>> When we are creating a table, we use the same functions as the
>>> > >>>> test
>>> as
>>> > >> follows...
>>> > >>>>
>>> > >>>> ...
>>> > >>>>>
>>> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>>> > >>>>>
>>> > >>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
>>> > >>>>> );
>>> > >>>>>
>>> > >>>>>             // Utility.CreateHBaseTable(conf, otherArgs[1],
>>> > >>>> otherArgs[2],
>>> > >>>>> true);
>>> > >>>>>
>>> > >>>>>             HBaseTestingUtility util = new
>>> > >>>>> HBaseTestingUtility();
>>> > >>>>>             HTable table = util.createTable(EDRP_TABLE,
>>> > >>>>> EDRP_FAMILY);
>>> > >>>>>
>>> > >>>>>             AggregationClient aClient = new
>>> > >>>>> AggregationClient(conf);
>>> > >>>> ...
>>> > >>>>
>>> > >>>> Running DESCRIBE on a table produced shows the following output:
>>> > >>>>
>>> > >>>> hbase(main):002:0> describe 'EDRP7'
>>> > >>>> DESCRIPTION
>>> > >>>>                                           ENABLED {NAME =>
>>> > >>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER =>
>>> > >>>> 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>  true '3',
>>> > >>>> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =>
>>> > >>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', B
>>> > >>>> LOCKCACHE => 'true'}]}
>>> > >>>>
>>> > >>>> We are using the tip of 0.92 (cloned from the Git repo). See
>>> > >>>> the version string below:
>>> > >>>>
>>> > >>>> hbase(main):005:0> version
>>> > >>>> 0.92.0, r1208286, Thu Dec 15 13:16:03 GMT 2011
>>> > >>>>
>>> > >>>> We would really appreciate an example of how to create a table
>>> > >>>> that is enabled to handle Aggregation).
>>> > >>>>
>>> > >>>> Thanks
>>> > >>>>
>>> > >>>>
>>> > >>>> ________________________________________
>>> > >>>> From: Ted Yu [yuzhihong@gmail.com]
>>> > >>>> Sent: 22 December 2011 17:03
>>> > >>>> To: user@hbase.apache.org
>>> > >>>> Subject: Re: AggregateProtocol Help
>>> > >>>>
>>> > >>>> Have you loaded AggregateImplementation into your table ?
>>> > >>>> Can you show us the contents of the following command in hbase
>>> shell:
>>> > >>>> describe 'your-table'
>>> > >>>>
>>> > >>>> BTW are you using the tip of 0.92 ?
>>> > >>>> HBASE-4946 would be of help for dynamically loaded
>>> > >>>> coprocessors
>>> which
>>> > >>>> you might use in the future.
>>> > >>>>
>>> > >>>> Cheers
>>> > >>>>
>>> > >>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox
>>> > >>>> <Tom.Wilcox@cse.org.uk>
>>> > >> wrote:
>>> > >>>>
>>> > >>>>> Hi,
>>> > >>>>>
>>> > >>>>> We are trying to use the aggregation functionality in HBase
>>> > >>>>> 0.92 and we have managed to get the test code working using
>>> > >>>>> the
>>> following
>>> > >> command:
>>> > >>>>>
>>> > >>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
>>> > >>>>> org.junit.runner.JUnitCore
>>> > >>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
>>> > >>>>>
>>> > >>>>> Closer inspection of this test class has revealed that it
>>> > >>>>> uses a mini DFS cluster to populate and run the tests. These
>>> > >>>>> tests return
>>> > >> successfully.
>>> > >>>>>
>>> > >>>>> However, when we attempt to run similar code on our
>>> > >>>>> development HDFS cluster we experience the following error:
>>> > >>>>>
>>> > >>>>> 11/12/22 15:46:28 WARN
>>> > >>>>> client.HConnectionManager$HConnectionImplementation: Error
>>> > >>>>> executing for
>>> > >>>> row
>>> > >>>>> java.util.concurrent.ExecutionException:
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException
>>> > >>>>> : No
>>> > >>>> matching
>>> > >>>>> handler for protocol
>>> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>> > >>>>> in region
>>> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.jav
>>> > >>>>> a:4010
>>> > >>>>> )
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HR
>>> > >>>> egionServer.java:3040)
>>> > >>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> > >>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> > >>>> java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> > >>>> sorImpl.java:25)
>>> > >>>>>   at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>>> > >>>> Engine.java:364)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>> > >>>> 1325)
>>> > >>>>> [sshexec]
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java
>>> > >>>>> :222)
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation.processExecs(HConnectionManager.java:1465)
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.
>>> > >>>>> java:1
>>> > >>>>> 555)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(Aggr
>>> > >>>> egationClient.java:229)
>>> > >>>>>   at
>>> > >>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
>>> > >>>>>   at EDRPAggregator.main(EDRPAggregator.java:77)
>>> > >>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> > >>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> > >>>> java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> > >>>> sorImpl.java:25)
>>> > >>>>>   at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >>>>>   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>> > >>>>> Caused by:
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException
>>> > >>>>> : No
>>> > >>>> matching
>>> > >>>>> handler for protocol
>>> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>> > >>>>> in region
>>> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.jav
>>> > >>>>> a:4010
>>> > >>>>> )
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HR
>>> > >>>> egionServer.java:3040)
>>> > >>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> > >>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> > >>>> java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> > >>>> sorImpl.java:25)
>>> > >>>>>   at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>>> > >>>> Engine.java:364)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>> > >>>> 1325)
>>> > >>>>> [sshexec]
>>> > >>>>>   at
>>> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> > >>>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
>>> > >>>> orAccessorImpl.java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
>>> > >>>> onstructorAccessorImpl.java:27)
>>> > >>>>>   at
>>> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:51
>>> > >>>>> 3)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(
>>> > >>>> RemoteExceptionHandler.java:96)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation.translateException(HConnectionManager.java:1651)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java:1327
>>> > >>>> )
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvo
>>> > >>>>> ker.ja
>>> > >>>>> va:79)
>>> > >>>>>   at $Proxy3.getSum(Unknown Source)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(A
>>> > >>>> ggregationClient.java:233)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(A
>>> > >>>> ggregationClient.java:230)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation$4.call(HConnectionManager.java:1453)
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java
>>> > >>>>> :303)
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
>>> > >>>> utor.java:886)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
>>> > >>>> .java:908)
>>> > >>>>>   at java.lang.Thread.run(Thread.java:662)
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException
>>> > >>>>> : No
>>> > >>>> matching
>>> > >>>>> handler for protocol
>>> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>> > >>>>> in region
>>> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.jav
>>> > >>>>> a:4010
>>> > >>>>> )
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HR
>>> > >>>> egionServer.java:3040)
>>> > >>>>>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> > >>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> > >>>> java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> > >>>> sorImpl.java:25)
>>> > >>>>>   at java.lang.reflect.Method.invoke(Method.java:597)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpc
>>> > >>>> Engine.java:364)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>> > >>>> 1325)
>>> > >>>>> [sshexec]
>>> > >>>>>   at
>>> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>> > >>>>> Method)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
>>> > >>>> orAccessorImpl.java:39)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
>>> > >>>> onstructorAccessorImpl.java:27)
>>> > >>>>>   at
>>> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:51
>>> > >>>>> 3)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(
>>> > >>>> RemoteExceptionHandler.java:96)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation.translateException(HConnectionManager.java:1651)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java:1327
>>> > >>>> )
>>> > >>>>>   at
>>> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvo
>>> > >>>>> ker.ja
>>> > >>>>> va:79)
>>> > >>>>>   at $Proxy3.getSum(Unknown Source)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(A
>>> > >>>> ggregationClient.java:233)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(A
>>> > >>>> ggregationClient.java:230)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen
>>> > >>>> tation$4.call(HConnectionManager.java:1453)
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java
>>> > >>>>> :303)
>>> > >>>>>   at
>>> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec
>>> > >>>> utor.java:886)
>>> > >>>>>   at
>>> > >>>>>
>>> > >>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
>>> > >>>> .java:908)
>>> > >>>>>   at java.lang.Thread.run(Thread.java:662)
>>> > >>>>>
>>> > >>>>> The source for our class is:
>>> > >>>>>
>>> > >>>>> import static org.junit.Assert.assertEquals;
>>> > >>>>>
>>> > >>>>> import java.io.IOException;
>>> > >>>>>
>>> > >>>>> import org.apache.hadoop.conf.Configuration;
>>> > >>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
>>> > >>>>> import org.apache.hadoop.hbase.HBaseTestingUtility;
>>> > >>>>> import org.apache.hadoop.hbase.HConstants;
>>> > >>>>> import org.apache.hadoop.hbase.HTableDescriptor;
>>> > >>>>> import org.apache.hadoop.hbase.client.HTable;
>>> > >>>>> import org.apache.hadoop.hbase.client.Put;
>>> > >>>>> import org.apache.hadoop.hbase.client.Scan;
>>> > >>>>> import
>>> > >>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient;
>>> > >>>>> import
>>> > >>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpre
>>> > >>>>> ter; import org.apache.hadoop.hbase.util.Bytes;
>>> > >>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>> > >>>>> import org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
>>> > >>>>> import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
>>> > >>>>> import org.junit.Test;
>>> > >>>>>
>>> > >>>>> public class EDRPAggregator {
>>> > >>>>>
>>> > >>>>>      // private static final byte[] EDRP_FAMILY =
>>> > >>>>> Bytes.toBytes("EDRP");
>>> > >>>>>      // private static final byte[] EDRP_QUALIFIER =
>>> > >>>>> Bytes.toBytes("advanceKWh");
>>> > >>>>>
>>> > >>>>>      private static byte[] ROW = Bytes.toBytes("testRow");
>>> > >>>>>      private static final int ROWSIZE = 20;
>>> > >>>>>      private static byte[][] ROWS = makeN(ROW, ROWSIZE);
>>> > >>>>>      private static final byte[] TEST_QUALIFIER =
>>> > >>>>> Bytes.toBytes("TestQualifier");
>>> > >>>>>      private static final byte[] TEST_MULTI_CQ =
>>> > >>>>> Bytes.toBytes("TestMultiCQ");
>>> > >>>>>      private static final int rowSeperator1 = 5;
>>> > >>>>>        private static final int rowSeperator2 = 12;
>>> > >>>>>
>>> > >>>>>      public static void testSumWithValidRange(Configuration
>>> > >>>>> conf,
>>> > >>>>>                    String[] otherArgs) throws Throwable {
>>> > >>>>>             byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
>>> > >>>>>             byte[] EDRP_FAMILY = Bytes.toBytes(otherArgs[2]);
>>> > >>>>>
>>> > >>>>>
>>> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>>> > >>>>>
>>> > >>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
>>> > >>>>> );
>>> > >>>>>
>>> > >>>>>             // Utility.CreateHBaseTable(conf, otherArgs[1],
>>> > >>>> otherArgs[2],
>>> > >>>>> true);
>>> > >>>>>
>>> > >>>>>             HBaseTestingUtility util = new
>>> > >>>>> HBaseTestingUtility();
>>> > >>>>>             HTable table = util.createTable(EDRP_TABLE,
>>> > >>>>> EDRP_FAMILY);
>>> > >>>>>
>>> > >>>>>             AggregationClient aClient = new
>>> > >>>>> AggregationClient(conf);
>>> > >>>>>             Scan scan = new Scan();
>>> > >>>>>             scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
>>> > >>>>>             final ColumnInterpreter<Long, Long> ci = new
>>> > >>>>> LongColumnInterpreter();
>>> > >>>>>             long sum =
>>> > >>>>> aClient.sum(Bytes.toBytes(otherArgs[0]),
>>> > >>>>> ci,
>>> > >>>> scan);
>>> > >>>>>             System.out.println(sum);
>>> > >>>>>      }
>>> > >>>>>
>>> > >>>>>      /**
>>> > >>>>>      * Main entry point.
>>> > >>>>>      *
>>> > >>>>>       * @param argsThe
>>> > >>>>>      *            command line parameters.
>>> > >>>>>      * @throws Exception
>>> > >>>>>      *             When running the job fails.
>>> > >>>>>      */
>>> > >>>>>      public static void main(String[] args) throws Exception
>>> > >>>>> {
>>> > >>>>>             Configuration conf = HBaseConfiguration.create();
>>> > >>>>>
>>> > >>>>>             String[] otherArgs = new
>>> > >>>>> GenericOptionsParser(conf,
>>> > >>>>> args)
>>> > >>>>>                          .getRemainingArgs();
>>> > >>>>>             if (otherArgs.length != 3) {
>>> > >>>>>                    System.err
>>> > >>>>>                                 .println("Wrong number of
>>> > >>>>> arguments: " + otherArgs.length);
>>> > >>>>>                    System.err.println("Usage: " +
>>> > >>>>> "<tablename> <colfam> <qualifier>");
>>> > >>>>>                    System.exit(-1);
>>> > >>>>>             }
>>> > >>>>>
>>> > >>>>>             try {
>>> > >>>>>                    testSumWithValidRange(conf, otherArgs);
>>> > >>>>>             } catch (Throwable e) {
>>> > >>>>>                    e.printStackTrace();
>>> > >>>>>             }
>>> > >>>>>      }
>>> > >>>>>
>>> > >>>>>      /**
>>> > >>>>>      * an infrastructure method to prepare rows for the
> testtable.
>>> > >>>>>      *
>>> > >>>>>       * @param base
>>> > >>>>>      * @param n
>>> > >>>>>      * @return
>>> > >>>>>      */
>>> > >>>>>      private static byte[][] makeN(byte[] base, int n) {
>>> > >>>>>             byte[][] ret = new byte[n][];
>>> > >>>>>             for (int i = 0; i < n; i++) {
>>> > >>>>>                    ret[i] = Bytes.add(base,
>>> > >>>>> Bytes.toBytes(i));
>>> > >>>>>             }
>>> > >>>>>             return ret;
>>> > >>>>>      }
>>> > >>>>> }
>>> > >>>>>
>>> > >>>>> Please can you suggest what might be causing and/or how we
>>> > >>>>> might fix this UnknownProtocolException?
>>> > >>>>>
>>> > >>>>> Also, does anyone have any working examples using the
>>> > >>>>> aggregation
>>> > >>>> protocol
>>> > >>>>> other than the test code?
>>> > >>>>>
>>> > >>>>> Thanks,
>>> > >>>>> Tom
>>> > >>>>>
>>> > >>>>>
>>> > >>>>
>>> > >>
>>> > >>
>>> >
>>> >
>>>
>

Mime
View raw message