Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: domain of royston.sellman@googlemail.com
 designates 74.125.82.169 as permitted sender)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: median aggregate Was: AggregateProtocol Help
From: Royston Sellman <royston.sellman@googlemail.com>
In-Reply-To: <19ED0BD2-D67D-4FE0-8A94-43917F38D847@gmail.com>
Date: Fri, 6 Jan 2012 15:09:01 +0000
Content-Transfer-Encoding: quoted-printable
Message-Id: <D407890B-386A-4A84-9B5E-4A14DB3932A2@gmail.com>
References: 
 <CALte62yus86e-mNt4VrVkXo_NqR3z6_hdoMkyEb2Kkzji86ksg@mail.gmail.com>
 <725F3D3F-AC1E-4DAD-8E79-D7CE35598C3F@gmail.com>
 <19ED0BD2-D67D-4FE0-8A94-43917F38D847@gmail.com>
To: user@hbase.apache.org

Done.

Thanks,
Royston


> Do you mind putting what you wrote below on hbase-5123 ?
>=20
> Thanks
>=20
>=20
>=20
> On Jan 6, 2012, at 6:22 AM, Royston Sellman =
<royston.sellman@googlemail.com> wrote:
>=20
>> Hi Ted,
>>=20
>> Yes, that is the use case I am thinking of.=20
>>=20
>> Re: 5123 I have also had some time to think about other aggregation =
functions (Please be aware that I am new to HBase, Coprocessors, and the =
Aggregation Protocol and I have little knowledge of distributed =
numerical algorithms!). It seems to me the pattern in AP is to return a =
SINGLE value from a SINGLE column (CF:CQ) of a table. In future one =
might wish to extend AP to return MULTIPLE values from MULTIPLE columns, =
so it is good to keep this in mind for the SINGLE value/SINGLE column =
(SVSC) case.=20
>>=20
>> So, common SVSC aggregation functions (AP supported first):
>> min
>> max
>> sum
>> count
>> avg (arithmetic mean)
>> std
>> median
>> mode=20
>> quantile/ntile
>> mult/product
>>=20
>> for column values of all numeric types, returning values of that =
type.
>>=20
>> Some thoughts on the future possibilities:
>> An example of a future SINGLE value MULTIPLE column use case could be =
weighted versions of the above functions i.e. a column of weights =
applied to the column of values then the new aggregation derived.
>> (note: there is a very good description of Weighted Median in the R =
language documentation:
>> =
http://www.stat.ucl.ac.be/ISdidactique/Rhelp/library/R.basic/html/weighted=
.median.html)
>>=20
>> An example of future MULTIPLE value SINGLE column could be range: =
return all rows with a column value between two values. Maybe this is a =
bad example because there could be better HBase ways to do it with =
filters/scans at a higher level. Perhaps binning is a better example? =
i.e. return an array containing values derived from applying one of the =
SVSC functions to a binned column e.g:
>> int bins =3D 100;
>> aClient.sum(table, ci, scan, bins); =3D> {12.3, 14.5...}
>> Another example (common in several programming languages) is to map =
an arbitrary function over a column and return the new vector. Of =
course, again this may be a bad example in the case of long HBase =
columns but it seems like an appropriate thing to do with coprocessors.
>>=20
>> MULTIPLE value MULTIPLE column examples are common in spatial data =
processing but I see there has been a lot of spatial/GIS discussion =
around HBase which I have not read yet. So I'll keep quiet for now.
>>=20
>> I hope these thoughts strike a balance between my (special interest) =
use case of statistical/spatial functions on tables and general purpose =
(but coprocessor enabled/regionserver distributed) HBase.
>>=20
>> Best regards,
>> Royston
>>=20
>>=20
>> On 6 Jan 2012, at 03:31, Ted Yu wrote:
>>=20
>>> Royston:
>>> For the median aggregate, is the following what you're looking for ?
>>> Find the median among the values of all the keyvalue for =
cf:qualifier
>>> column.
>>>=20
>>> There is a well known distributed method of computing median that =
involves
>>> multiple roundtrips (to the region servers).
>>>=20
>>> Just want to confirm the use case.
>>>=20
>>> Thanks
>>>=20
>>> On Wed, Jan 4, 2012 at 10:57 AM, Royston Sellman <
>>> royston.sellman@googlemail.com> wrote:
>>>=20
>>>> Great ideas. Thanks.
>>>>=20
>>>> w.r.t. 5123: I'll think about it for a day or two then make some =
comments.
>>>>=20
>>>> 5122 is very desirable.
>>>>=20
>>>> Best Regards,
>>>> Royston
>>>>=20
>>>> On 4 Jan 2012, at 15:01, Ted Yu wrote:
>>>>=20
>>>>> I can see room for improvement w.r.t. ColumnInterpreters
>>>>> I logged two JIRAs:
>>>>> https://issues.apache.org/jira/browse/HBASE-5122 is for loading
>>>>> ColumnInterpreters dynamically
>>>>>=20
>>>>> https://issues.apache.org/jira/browse/HBASE-5123 is for adding =
more
>>>>> aggregation functions.
>>>>>=20
>>>>> Royston:
>>>>> Feel free to elaborate on 5213 and explain what Mult aggregate =
should do.
>>>>>=20
>>>>> Cheers
>>>>>=20
>>>>> On Wed, Jan 4, 2012 at 3:43 AM, Royston Sellman <
>>>>> royston.sellman@googlemail.com> wrote:
>>>>>=20
>>>>>> Ted, Himanshu and Gary,
>>>>>>=20
>>>>>> It works now! I re-created my HBase table to contain =
Bytes.toBytes(Long)
>>>>>> values and that fixed it.
>>>>>>=20
>>>>>> For the time being we can convert everything to Longs and work =
with
>>>> that,
>>>>>> but we will probably write our own ColumnInterpreters soon for =
our data
>>>>>> types, so thanks for the pointer to HBASE-4946. There are also
>>>> Functions we
>>>>>> need (e.g. Median, Weighted Median, Mult) which might best be =
placed in
>>>> the
>>>>>> Aggregations Protocol. We'll be sure to discuss this with you =
when we
>>>> start.
>>>>>>=20
>>>>>> Meanwhile, thanks again for all your help!
>>>>>>=20
>>>>>> Royston
>>>>>>=20
>>>>>>=20
>>>>>> On 3 Jan 2012, at 18:58, Ted Yu wrote:
>>>>>>=20
>>>>>>> I like long messages :-) because they provide more clues.
>>>>>>>=20
>>>>>>> For part 1, you don't have to call Bytes.toxxx as long as the
>>>> interpreter
>>>>>>> uses method consistent with the way you write values into HBase =
tables.
>>>>>>>=20
>>>>>>> For part 2, HBASE-4946 is related.
>>>>>>> Basically you need to place the jar containing your coprocessor =
and
>>>>>>> interpreter code on hdfs so that you can load it into your HBase =
table.
>>>>>>> Look at this for details:
>>>>>>> https://issues.apache.org/jira/browse/HBASE-4554
>>>>>>>=20
>>>>>>> Cheers
>>>>>>>=20
>>>>>>> On Tue, Jan 3, 2012 at 10:42 AM, Royston Sellman <
>>>>>>> royston.sellman@googlemail.com> wrote:
>>>>>>>=20
>>>>>>>> Hi Ted,
>>>>>>>>=20
>>>>>>>> PART 1
>>>>>>>> =3D=3D=3D=3D=3D
>>>>>>>> Thanks for the hint. I think maybe you have given me some =
inspiration!
>>>>>>>>=20
>>>>>>>> It looks like getValue will return null if the table value is =
not the
>>>>>>>> length
>>>>>>>> of a long. When we created our table (batch loading CSVs using =
the
>>>>>>>> SampleUploader example) we simply have this as our put():
>>>>>>>> put.add(family, Bytes.toBytes("advanceKWh"), advanceKWh);
>>>>>>>> [note we do no Bytes.toxxx casts to the advanceKWh value. The =
values
>>>>>> look
>>>>>>>> OK
>>>>>>>> from HBase shell though :-)]
>>>>>>>>=20
>>>>>>>> but I looked at TestAggregateProtocol.java again and I see =
there puts
>>>>>> like:
>>>>>>>> p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(l)),
>>>>>>>> Bytes.toBytes(l * 10));
>>>>>>>>=20
>>>>>>>> So my hypothesis is that we need to do something like:
>>>>>>>> Long l =3D new Long(1);
>>>>>>>> put.add(family, Bytes.toBytes("advanceKWh"), Bytes.toBytes(l *
>>>>>>>> advanceKWh));
>>>>>>>> when we create the table.
>>>>>>>>=20
>>>>>>>> Do you think my hypothesis is correct? Did we build our table
>>>>>> incorrectly
>>>>>>>> for reading longs from it?
>>>>>>>>=20
>>>>>>>> PART 2
>>>>>>>> =3D=3D=3D=3D=3D
>>>>>>>> Anyway we will obviously need to make our own interpreters. but =
we
>>>>>> failed
>>>>>>>> at
>>>>>>>> this task so far:
>>>>>>>> In order to implement our own ColumnInterpretter, we first =
attempted
>>>>>> simply
>>>>>>>> extending the LongColumnInterpreter and passing that as a =
parameter to
>>>>>>>> aClient.sum().
>>>>>>>> import
>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter;
>>>>>>>>=20
>>>>>>>> public class LCI extends LongColumnInterpreter {
>>>>>>>> public Long add(Long a, Long b) {
>>>>>>>> System.out.print("LCI.add("+a+","+b+")");
>>>>>>>> Long c =3D super.add(a,b);
>>>>>>>> System.out.println(" -> "+c);
>>>>>>>> return c;
>>>>>>>> }
>>>>>>>> };
>>>>>>>> ....
>>>>>>>> final ColumnInterpreter<Long, Long> ci =3D new LCI();
>>>>>>>> long sum=3D-1;
>>>>>>>> try {
>>>>>>>> sum =3D aClient.sum(EDRP_TABLE, ci, scan);
>>>>>>>> ...
>>>>>>>> However, we received class not found errors in our regionserver =
log
>>>>>> when we
>>>>>>>> ran this code
>>>>>>>> Clearly we are missing something
>>>>>>>> We've started looking at modifying the HBase source and =
rebuilding
>>>> that
>>>>>>>> But I think this is not the way we should be working
>>>>>>>> It seems that we should be implementing the ColumnInterpreter
>>>> interface
>>>>>> and
>>>>>>>> passing an instance of that implementation class too as a =
parameter.
>>>> Is
>>>>>>>> this correct?  Could you provide us with an example?  Any ideas =
why we
>>>>>>>> might
>>>>>>>> be getting class not found exception from the regionserver?
>>>>>>>>=20
>>>>>>>> Sorry for the long message!
>>>>>>>>=20
>>>>>>>> Royston (and Tom)
>>>>>>>>=20
>>>>>>>>=20
>>>>>>>> -----Original Message-----
>>>>>>>> From: Ted Yu [mailto:yuzhihong@gmail.com]
>>>>>>>> Sent: 03 January 2012 18:00
>>>>>>>> To: user@hbase.apache.org
>>>>>>>> Subject: Re: AggregateProtocol Help
>>>>>>>>=20
>>>>>>>> My previous email might not be hitting the root cause.
>>>>>>>> I think the following method in LCI may be giving you the null:
>>>>>>>>=20
>>>>>>>> public Long getValue(byte[] colFamily, byte[] colQualifier, =
KeyValue
>>>> kv)
>>>>>>>> throws IOException {
>>>>>>>> if (kv =3D=3D null || kv.getValueLength() !=3D =
Bytes.SIZEOF_LONG)
>>>>>>>> return null;
>>>>>>>> return Bytes.toLong(kv.getBuffer(), kv.getValueOffset());
>>>>>>>>=20
>>>>>>>> Look at the if statement above carefully.
>>>>>>>> If it doesn't match how you store values in HBase, feel free to
>>>> subclass
>>>>>>>> LongColumnInterpreter and provide the correct interpretation.
>>>>>>>>=20
>>>>>>>> BTW you don't need to restart cluster just because you need to =
use
>>>> your
>>>>>> own
>>>>>>>> interpreter :-)
>>>>>>>>=20
>>>>>>>> On Tue, Jan 3, 2012 at 9:48 AM, Royston Sellman <
>>>>>>>> royston.sellman@googlemail.com> wrote:
>>>>>>>>=20
>>>>>>>>> Hi Ted,
>>>>>>>>>=20
>>>>>>>>> Here is the output. As you can see aClient is not nul:
>>>>>>>>>=20
>>>>>>>>> AggregationClient aClient =3D new AggregationClient(conf);
>>>>>>>>> System.err.println("aClient: "+aClient);
>>>>>>>>>=20
>>>>>>>>> <<<    aClient:
>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16
>>>>>>>>>=20
>>>>>>>>> It will take us a little while to add log code to LCI... we =
have to
>>>>>>>>> edit the source, rebuild 0.92, redistribute round our cluster,
>>>> restart
>>>>>>>>> ;) We'll get back to you when this is done.
>>>>>>>>>=20
>>>>>>>>> Royston
>>>>>>>>>=20
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Ted Yu [mailto:yuzhihong@gmail.com]
>>>>>>>>> Sent: 03 January 2012 17:10
>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>> Subject: Re: AggregateProtocol Help
>>>>>>>>>=20
>>>>>>>>> Royston:
>>>>>>>>> Thanks for your effort trying to hunt down the problem.
>>>>>>>>>=20
>>>>>>>>> Can you add a log after this line to see if aClient is null ?
>>>>>>>>>          AggregationClient aClient =3D new =
AggregationClient(conf);
>>>>>>>>>=20
>>>>>>>>> I was looking at LongColumnInterpreter.add() which is called =
by
>>>>>>>>> aClient.sum()
>>>>>>>>> Can you add a few log statements in =
LongColumnInterpreter.add() to
>>>> see
>>>>>>>>> what parameters are passed to it ?
>>>>>>>>>=20
>>>>>>>>> Cheers
>>>>>>>>>=20
>>>>>>>>> On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman <
>>>>>>>>> royston.sellman@googlemail.com> wrote:
>>>>>>>>>=20
>>>>>>>>>> Hi Ted, Himanshu, Gary,
>>>>>>>>>>=20
>>>>>>>>>> Thanks again for your attention. I experimented with a =
shorter table
>>>>>>>>>> and it looks like the timeout error was spurious...
>>>>>>>>>>=20
>>>>>>>>>> With the shorter table I now get an NPE when I call
>>>>>>>>>> AggregationClient.sum().
>>>>>>>>>> Here's the code snippet:
>>>>>>>>>>=20
>>>>>>>>>>           // Test the table
>>>>>>>>>>           HTable table =3D new HTable(EDRP_TABLE);
>>>>>>>>>>           Get get =3D new Get(Bytes.toBytes("row-aa"));
>>>>>>>>>>           get.addColumn(Bytes.toBytes("EDRP"),
>>>>>>>>>> Bytes.toBytes("advanceKWh"));
>>>>>>>>>>           Result result =3D table.get(get);
>>>>>>>>>>           byte [] val =3D =
result.getValue(Bytes.toBytes("EDRP"),
>>>>>>>>>> Bytes.toBytes("advanceKWh"));
>>>>>>>>>>           System.out.println("Row aa =3D " +
>>>>>>>>>> Bytes.toString(val));
>>>>>>>>>>=20
>>>>>>>>>>           AggregationClient aClient =3D new
>>>> AggregationClient(conf);
>>>>>>>>>>           Scan scan =3D new Scan();
>>>>>>>>>>            scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER);
>>>>>>>>>>           scan.setStartRow(Bytes.toBytes("row-ab"));
>>>>>>>>>>           scan.setStopRow(Bytes.toBytes("row-az"));
>>>>>>>>>>           System.out.println(Bytes.toString(EDRP_FAMILY) + =
":"
>>>>>>>>>> + Bytes.toString(EDRP_QUALIFIER));
>>>>>>>>>>            final ColumnInterpreter<Long, Long> ci =3D new
>>>>>>>>>> LongColumnInterpreter();
>>>>>>>>>>            long sum=3D-1;
>>>>>>>>>>           try {
>>>>>>>>>>                   sum =3D aClient.sum(EDRP_TABLE, ci, scan);
>>>>>>>>>>           } catch (Throwable e) {
>>>>>>>>>>                   // TODO Auto-generated catch block
>>>>>>>>>>                   e.printStackTrace();
>>>>>>>>>>           }
>>>>>>>>>>           System.out.println(sum);
>>>>>>>>>>=20
>>>>>>>>>> The first part is just to check that my table is OK. It =
prints the
>>>>>>>>>> correct value for row aa. Then I check CF:CQ is correct. Then =
I get
>>>>>>>>>> the -1 that sum was defined as. Then the NPE. Here is the run
>>>> output:
>>>>>>>>>>=20
>>>>>>>>>> Row aa =3D 3.0
>>>>>>>>>> EDRP:advanceKWh
>>>>>>>>>> -1
>>>>>>>>>> java.lang.NullPointerException
>>>>>>>>>> at =
EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66)
>>>>>>>>>> at EDRPAggregator.main(EDRPAggregator.java:96)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native =
Method)  at
>>>>>>>>>>=20
>>>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
>>>>>>>>>> .j
>>>>>>>>>> ava:39
>>>>>>>>>> )
>>>>>>>>>> at
>>>>>>>>>>=20
>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
>>>>>>>>>> ss
>>>>>>>>>> orImpl
>>>>>>>>>> .java:25)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>=20
>>>>>>>>>> Line 66 is the line:
>>>>>>>>>>                   sum =3D aClient.sum(EDRP_TABLE, ci, scan);
>>>>>>>>>>=20
>>>>>>>>>> But I'm not sure that is really the line generating the NPE =
because
>>>>>>>>>> I get the -1 output (same result when line 66 is not wrapped =
in
>>>>>>>>>> try-catch
>>>>>>>>> BTW).
>>>>>>>>>> Anyway, -1 is clearly not the correct value for the sum.
>>>>>>>>>>=20
>>>>>>>>>> [Note: I'm setting a start and end row on the scan because I =
saw a
>>>>>>>>>> comment in AggregationClient.java saying they must be set. =
However,
>>>>>>>>>> I think this comment is obsolete as the test code in
>>>>>>>>>> TestAggregateProtocol.java sometimes does not include start & =
end
>>>>>>>>>> rows. The run is exactly the same without the start & stop =
rows ]
>>>>>>>>>>=20
>>>>>>>>>> We are struggling with Aggregations not least because
>>>>>>>>>> TestAggregateProtocol.java is the only source of example code =
we
>>>>>>>>>> could find and it is not ideal because it is designed to work =
with
>>>>>>>>>> MiniCluster rather than a real cluster like ours. Is there =
any other
>>>>>>>>>> example code in existence?
>>>>>>>>>> I think that would really help us.
>>>>>>>>>>=20
>>>>>>>>>> Meanwhile, thanks for your patience with our questions. We =
are
>>>>>>>>>> really impressed with the AggregationProtocol
>>>>>>>>>> architecture/functionality and very keen to use it.
>>>>>>>>>>=20
>>>>>>>>>> Royston (and Tom)
>>>>>>>>>>=20
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
>>>>>>>>>> Sent: 02 January 2012 06:23
>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>> Subject: Re: AggregateProtocol Help
>>>>>>>>>>=20
>>>>>>>>>> Hi Royston,
>>>>>>>>>>=20
>>>>>>>>>> Try increasing the value set for hbase.rpc.timeout (by =
default 60
>>>>>>>>> seconds).
>>>>>>>>>> Add something like this to hbase-site.xml:
>>>>>>>>>>=20
>>>>>>>>>> <property>
>>>>>>>>>> <name>hbase.rpc.timeout</name>
>>>>>>>>>> <value>180000</value>
>>>>>>>>>> </property>
>>>>>>>>>>=20
>>>>>>>>>> This would increase the timeout value to 3 minutes, for =
example.
>>>>>>>>>>=20
>>>>>>>>>> But as Ted and Himanshu mention, 557k rows is not a large =
table and
>>>>>>>>>> should not be taking very long to process, unless these =
happen to be
>>>>>>>>>> very, very wide rows.  It sounds like there could be =
something else
>>>>>>>>>> going on.  Does debug level logging show anything else in the =
region
>>>>>>>>> server log?
>>>>>>>>>>=20
>>>>>>>>>> --gh
>>>>>>>>>>=20
>>>>>>>>>> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <yuzhihong@gmail.com> =
wrote:
>>>>>>>>>>> Thanks for the reminder Himanshu.
>>>>>>>>>>>=20
>>>>>>>>>>> Royston:
>>>>>>>>>>> =46rom this blog you can get some history on this subject:
>>>>>>>>>>> =
http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoproce
>>>>>>>>>>> ss
>>>>>>>>>>> or
>>>>>>>>>>> .html
>>>>>>>>>>>=20
>>>>>>>>>>>=20
>>>>>>>>>>> On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha
>>>>>>>>>>> <hvashish@cs.ualberta.ca
>>>>>>>>>>>> wrote:
>>>>>>>>>>>=20
>>>>>>>>>>>> Hello Royston,
>>>>>>>>>>>>=20
>>>>>>>>>>>> Sorry to hear that you are getting trouble while using
>>>>>>>>>>>> Aggregation functionalities.
>>>>>>>>>>>>=20
>>>>>>>>>>>> 557k rows seems to be a small table and a SocketTimeout =
does not
>>>>>>>>>>>> seem to be an ok response.
>>>>>>>>>>>> It will be good to know the region distribution as such. =
(how
>>>>>>>>>>>> many
>>>>>>>>>> regions?
>>>>>>>>>>>> Is it a full table scan?)
>>>>>>>>>>>>=20
>>>>>>>>>>>> You are using the sum function; how are you using the
>>>>>>>>> ColumnInterpreter.
>>>>>>>>>>>> Can you enable the log level to debug to see why the RS is =
taking
>>>>>>>>>>>> that long to respond (more than 113 sec).
>>>>>>>>>>>> The 0 return value is the default result.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Thanks for trying this out.
>>>>>>>>>>>>=20
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Himanshu
>>>>>>>>>>>>=20
>>>>>>>>>>>> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman <
>>>>>>>>>>>> royston.sellman@googlemail.com> wrote:
>>>>>>>>>>>>=20
>>>>>>>>>>>>> Hi Ted,
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> I think 0 is the only value we ever see (I'll check =
tomorrow:
>>>>>>>>>>>>> the server is down right now). Our table has 557,000 rows. =
I'll
>>>>>>>>>>>>> try a much shorter table tomorrow.
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Yes, we have RS running on the NN, but it's a test cluster =
and
>>>>>>>>>>>>> we are
>>>>>>>>>>>> used
>>>>>>>>>>>>> to it :)
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Do you think using AggregationProtocol is the best =
strategy for
>>>>>>>>>>>>> the case where we want to use basic SQL-style functions =
like
>>>>>>>>>>>>> SUM, AVG, STD, MIN, MAX? Do you think there is a better =
strategy?
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> Many thanks,
>>>>>>>>>>>>> Royston
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>> On 1 Jan 2012, at 17:58, Ted Yu wrote:
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Royston:
>>>>>>>>>>>>>> Happy New Year to you too.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> java.net.SocketTimeoutException: Call to
>>>>>>>>>>>>>>>> namenode/10.0.0.235:60020
>>>>>>>>>>>> failed
>>>>>>>>>>>>> on
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> It seems the namenode above actually refers to a region =
server.
>>>>>>>>>>>>>> This
>>>>>>>>>>>> is a
>>>>>>>>>>>>>> little bit confusing :-)
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> The sum value below is 0.
>>>>>>>>>>>>>> Have you ever seen a value greater than 0 ?
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> How many rows are there in this CF:CQ ?
>>>>>>>>>>>>>> The timeout was reported earlier by other people where
>>>>>>>>>>>>>> there're many
>>>>>>>>>>>> rows
>>>>>>>>>>>>>> in the table.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> There is a JIRA to provide streaming support for =
coprocessor
>>>>>>>>>>>>>> but the development there has stalled.
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>> On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman <
>>>>>>>>>>>>>> royston.sellman@googlemail.com> wrote:
>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Hi Gary and Ted,
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Royston (Tom's colleague) here. Back onto this after the
>>>>>>>>>>>>>>> Christmas/New
>>>>>>>>>>>>> Year
>>>>>>>>>>>>>>> break.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Many thanks for your help so far. We enabled our =
database
>>>>>>>>>>>>>>> via your hbase-site.xml mod and were able to move on. to
>>>>>>>>>>>>>>> other errors. But I
>>>>>>>>>>>>> think
>>>>>>>>>>>>>>> we
>>>>>>>>>>>>>>> are now actually getting an aggregation partially =
calculated
>>>>>>>>>>>>>>> on our
>>>>>>>>>>>>> table
>>>>>>>>>>>>>>> (this feels like progress). The details:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> On running our client we now get this exception:
>>>>>>>>>>>>>>> 11/12/31 17:51:09 WARN
>>>>>>>>>>>>>>> client.HConnectionManager$HConnectionImplementation: =
Error
>>>>>>>>>>>>>>> executing
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> row
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> java.util.concurrent.ExecutionException:
>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.RetriesExhaustedException:
>>>>>>>>>>>>>>> Failed after attempts=3D10, exceptions:
>>>>>>>>>>>>>>> Sat Dec 31 17:41:30 GMT 2011,
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
>>>>>>>>>>>>>>> java.net.SocketTimeoutException: Call to
>>>>>>>>>>>>>>> namenode/10.0.0.235:60020
>>>>>>>>>>>> failed
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> socket timeout exception: =
java.net.SocketTimeoutException:
>>>>>>>>>>>>>>> 60000
>>>>>>>>>>>> millis
>>>>>>>>>>>>>>> timeout while waiting for channel to be ready for read. =
ch :
>>>>>>>>>>>>>>> java.nio.channels.SocketChannel[connected
>>>>>>>>>>>>>>> local=3D/10.0.0.235:59999 =
remote=3Dnamenode/10.0.0.235:60020]
>>>>>>>>>>>>>>> (8 more of these, making for 10 tries) Sat Dec 31 =
17:51:09
>>>>>>>>>>>>>>> GMT 2011,
>>>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
>>>>>>>>>>>>>>> java.net.SocketTimeoutException: Call to
>>>>>>>>>>>>>>> namenode/10.0.0.235:60020
>>>>>>>>>>>> failed
>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>> socket timeout exception: =
java.net.SocketTimeoutException:
>>>>>>>>>>>>>>> 60000
>>>>>>>>>>>> millis
>>>>>>>>>>>>>>> timeout while waiting for channel to be ready for read. =
ch :
>>>>>>>>>>>>>>> java.nio.channels.SocketChannel[connected
>>>>>>>>>>>>>>> local=3D/10.0.0.235:59364 =
remote=3Dnamenode/10.0.0.235:60020]
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:
>>>>>>>>>>>>>>> 22
>>>>>>>>>>>>>>> 2)
>>>>>>>>>>>>>>>  at
>>>>>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>=20
>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio=
n.
>>>>>>>>>>>>>>> processExecs(HConnectionManager.java:1465)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java
>>>>>>>>>>>> :1
>>>>>>>>>>>> 55
>>>>>>>>>>>> 5)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
>>>>>>>>>>>> Ag
>>>>>>>>>>>> gr
>>>>>>>>>>>> egation
>>>>>>>>>>>>>>> Client.java:229)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>> =
EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
>>>>>>>>>>>>>>>  at EDRPAggregator.main(EDRPAggregator.java:77)
>>>>>>>>>>>>>>>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>>>>>>> java:39
>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>>> cc
>>>>>>>>>>>> es
>>>>>>>>>>>> sorImpl
>>>>>>>>>>>>>>> .java:25)
>>>>>>>>>>>>>>>  at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>  at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Looking at the log (.regionserver-namenode.log) I see =
this
>>>>>>>>>>>>>>> debug
>>>>>>>>>>>>> message:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> 2011-12-31 17:42:23,472 DEBUG
>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.coprocessor.AggregateImplementation:
>>>>>>>>>>>>>>> Sum from
>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> region is
>>>>>>>>>>>> =
EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.:
>>>>>>>>>>>>> 0
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Where the sum value looks reasonable which makes me =
think
>>>>>>>>>>>>>>> the sum of a CF:CQ worked. But I never see this value on
>>>>>>>> stdout.
>>>>>>>>>>>>>>> Then I see this warning:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> 2011-12-31 17:42:23,476 WARN =
org.apache.hadoop.ipc.HBaseServer:
>>>>>>>>>>>>>>> (responseTooSlow):
>>>>>>>>>>>>>>> {"processingtimems":113146,"call":"execCoprocess$
>>>>>>>>>>>>>>> 2011-12-31 17:42:23,511 WARN =
org.apache.hadoop.ipc.HBaseServer:
>>>>>>>>>>>>>>> IPC
>>>>>>>>>>>>> Server
>>>>>>>>>>>>>>> Responder, call execCoprocessor([B@4b22fad6, =
getSum(org.$
>>>>>>>>>>>>>>> 2011-12-31 17:42:23,515 WARN =
org.apache.hadoop.ipc.HBaseServer:
>>>>>>>>>>>>>>> IPC
>>>>>>>>>>>>> Server
>>>>>>>>>>>>>>> handler 1 on 60020 caught: =
java.nio.channels.ClosedChann$
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.ja
>>>>>>>>>>>> va
>>>>>>>>>>>> :1
>>>>>>>>>>>> 33)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>> =
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.
>>>>>>>>>>>> ja
>>>>>>>>>>>> va
>>>>>>>>>>>> :1651)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse
>>>>>>>>>>>> (H
>>>>>>>>>>>> Ba
>>>>>>>>>>>> seServe
>>>>>>>>>>>>>>> r.java:924)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBase
>>>>>>>>>>>> Se
>>>>>>>>>>>> rv
>>>>>>>>>>>> er.java
>>>>>>>>>>>>>>> :1003)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(
>>>>>>>>>>>> HB
>>>>>>>>>>>> as
>>>>>>>>>>>> eServer
>>>>>>>>>>>>>>> .java:409)
>>>>>>>>>>>>>>>  at
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>>>>>>>>>>> 1345)
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Have we missed out some step in the =
HBase/RegionServerconfig?
>>>>>>>>>>>>>>> Or is
>>>>>>>>>>>> our
>>>>>>>>>>>>>>> client code still deficient?
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Can you offer any suggestions? Is there any example code =
for
>>>>>>>>>>>>>>> the new Aggregations stuff.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Thanks and Happy New Year to you guys,
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Royston (and Tom).
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> (HBase 0.92, Hadoop 1.0)
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>> From: Gary Helmling [mailto:ghelmling@gmail.com]
>>>>>>>>>>>>>>> Sent: 23 December 2011 18:06
>>>>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>>>>> Subject: Re: AggregateProtocol Help
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Hi Tom,
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> The test code is not really the best guide for =
configuration.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> To enable the AggregateProtocol on all of your tables, =
add
>>>>>>>>>>>>>>> this to the hbase-site.xml for the servers in your =
cluster:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> <property>
>>>>>>>>>>>>>>> <name>hbase.coprocessor.user.region.classes</name>
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
<value>org.apache.hadoop.hbase.coprocessor.AggregateImplementatio
>>>>>>>>>>>> n<
>>>>>>>>>>>> /v
>>>>>>>>>>>> alue>
>>>>>>>>>>>>>>> </property>
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> If you only want to use the aggregate functions on a
>>>>>>>>>>>>>>> specific table
>>>>>>>>>>>> (or
>>>>>>>>>>>>>>> tables), then you can enable that individually for the =
table
>>>>>>>>>>>>>>> from the
>>>>>>>>>>>>>>> shell:
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> 1) disable the table
>>>>>>>>>>>>>>> hbase> disable 'EDRP7'
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> 2) add the coprocessor
>>>>>>>>>>>>>>> hbase> alter 'EDRP7', METHOD =3D> 'table_att',
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
'coprocessor'=3D>'|org.apache.hadoop.hbase.coprocessor.AggregateImp
>>>>>>>>>>>> le
>>>>>>>>>>>> me
>>>>>>>>>>>> ntation
>>>>>>>>>>>>>>> ||'
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> (Note that the pipes in the value string are required)
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> 3) re-enable the table
>>>>>>>>>>>>>>> hbase> enable 'EDRP7'
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> Either way should work.  With the second approach you =
will
>>>>>>>>>>>>>>> see the coprocessor listed when you describe the table =
from
>>>>>>>>>>>>>>> the shell, as Ted mentioned.  With the first approach =
you
>>>>>>>>>>>>>>> will not, but it should be
>>>>>>>>>>>>> loaded
>>>>>>>>>>>>>>> all the same.
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> --gh
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu
>>>>>>>>>>>>>>> <yuzhihong@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> I don't know why you chose HBaseTestingUtility to =
create
>>>>>>>>>>>>>>>> the
>>>>>>>>>> table.
>>>>>>>>>>>>>>>> I guess you followed test code example.
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> At least you should pass the conf to this ctor:
>>>>>>>>>>>>>>>> public HBaseTestingUtility(Configuration conf) {
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> If coprocessor was installed correctly, you should see
>>>>>>>>>>>>>>>> something like(from
>>>>>>>>>>>>>>>> HBASE-5070):
>>>>>>>>>>>>>>>> coprocessor$1 =3D>
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>> =
'|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox
>>>>>>>>>>>>>>>> <Tom.Wilcox@cse.org.uk>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> I am not sure how we load the AggregateImplementation =
into
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>> table.
>>>>>>>>>>>>>>>>> When we are creating a table, we use the same =
functions as
>>>>>>>>>>>>>>>>> the test
>>>>>>>>>>>> as
>>>>>>>>>>>>>>> follows...
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
>>>>>>>>>>>>>>>>>> );
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        // Utility.CreateHBaseTable(conf,
>>>>>>>>>>>>>>>>>> otherArgs[1],
>>>>>>>>>>>>>>>>> otherArgs[2],
>>>>>>>>>>>>>>>>>> true);
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        HBaseTestingUtility util =3D new
>>>>>>>>>>>>>>>>>> HBaseTestingUtility();
>>>>>>>>>>>>>>>>>>        HTable table =3D util.createTable(EDRP_TABLE,
>>>>>>>>>>>>>>>>>> EDRP_FAMILY);
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        AggregationClient aClient =3D new
>>>>>>>>>>>>>>>>>> AggregationClient(conf);
>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Running DESCRIBE on a table produced shows the =
following
>>>>>>>>> output:
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> hbase(main):002:0> describe 'EDRP7'
>>>>>>>>>>>>>>>>> DESCRIPTION
>>>>>>>>>>>>>>>>>                                      ENABLED {NAME =3D>
>>>>>>>>>>>>>>>>> 'EDRP7', FAMILIES =3D> [{NAME =3D> 'advanceKWh', =
BLOOMFILTER
>>>>>>>>>>>>>>>>> =3D> 'NONE', REPLICATION_SCOPE =3D> '0', VERSIONS =3D> =
 true
>>>>>>>>>>>>>>>>> '3', COMPRESSION =3D> 'NONE', MIN_VERSIONS =3D> '0', =
TTL =3D>
>>>>>>>>>>>>>>>>> '2147483647', BLOCKSIZE =3D> '65536', IN_MEMORY =3D> =
'false',
>>>>>>>>>>>>>>>>> B LOCKCACHE =3D> 'true'}]}
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> We are using the tip of 0.92 (cloned from the Git =
repo).
>>>>>>>>>>>>>>>>> See the version string below:
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> hbase(main):005:0> version 0.92.0, r1208286, Thu Dec =
15
>>>>>>>>>>>>>>>>> 13:16:03 GMT 2011
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> We would really appreciate an example of how to create =
a
>>>>>>>>>>>>>>>>> table that is enabled to handle Aggregation).
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>>> From: Ted Yu [yuzhihong@gmail.com]
>>>>>>>>>>>>>>>>> Sent: 22 December 2011 17:03
>>>>>>>>>>>>>>>>> To: user@hbase.apache.org
>>>>>>>>>>>>>>>>> Subject: Re: AggregateProtocol Help
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Have you loaded AggregateImplementation into your =
table ?
>>>>>>>>>>>>>>>>> Can you show us the contents of the following command =
in
>>>>>>>>>>>>>>>>> hbase
>>>>>>>>>>>> shell:
>>>>>>>>>>>>>>>>> describe 'your-table'
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> BTW are you using the tip of 0.92 ?
>>>>>>>>>>>>>>>>> HBASE-4946 would be of help for dynamically loaded
>>>>>>>>>>>>>>>>> coprocessors
>>>>>>>>>>>> which
>>>>>>>>>>>>>>>>> you might use in the future.
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> Cheers
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox
>>>>>>>>>>>>>>>>> <Tom.Wilcox@cse.org.uk>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> We are trying to use the aggregation functionality in
>>>>>>>>>>>>>>>>>> HBase
>>>>>>>>>>>>>>>>>> 0.92 and we have managed to get the test code working
>>>>>>>>>>>>>>>>>> using the
>>>>>>>>>>>> following
>>>>>>>>>>>>>>> command:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
>>>>>>>>>>>>>>>>>> org.junit.runner.JUnitCore
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Closer inspection of this test class has revealed =
that it
>>>>>>>>>>>>>>>>>> uses a mini DFS cluster to populate and run the =
tests.
>>>>>>>>>>>>>>>>>> These tests return
>>>>>>>>>>>>>>> successfully.
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> However, when we attempt to run similar code on our
>>>>>>>>>>>>>>>>>> development HDFS cluster we experience the following =
error:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> 11/12/22 15:46:28 WARN
>>>>>>>>>>>>>>>>>> client.HConnectionManager$HConnectionImplementation:
>>>>>>>>>>>>>>>>>> Error executing for
>>>>>>>>>>>>>>>>> row
>>>>>>>>>>>>>>>>>> java.util.concurrent.ExecutionException:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
>>>>>>>>>>>>>>>>>> ti
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> : No
>>>>>>>>>>>>>>>>> matching
>>>>>>>>>>>>>>>>>> handler for protocol
>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>>>>>>>>>>>>>>>>> in region
>>>>>>>>>>>>> =
EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
>>>>>>>>>>>>>>>>>> .j
>>>>>>>>>>>>>>>>>> av
>>>>>>>>>>>>>>>>>> a:4010
>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
>>>>>>>>>>>> r(
>>>>>>>>>>>> HR
>>>>>>>>>>>>>>>>> egionServer.java:3040)
>>>>>>>>>>>>>>>>>> at =
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>>>>>>>>>>>> java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>>> cc
>>>>>>>>>>>> es
>>>>>>>>>>>>>>>>> sorImpl.java:25)
>>>>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
>>>>>>>>>>>> eR
>>>>>>>>>>>> pc
>>>>>>>>>>>>>>>>> Engine.java:364)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>>>>>>>>>>>>>>>> 1325)
>>>>>>>>>>>>>>>>>> [sshexec]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.
>>>>>>>>>>>>>>>>>> ja
>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>> :222)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> tation.processExecs(HConnectionManager.java:1465)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.
>>>>>>>>>>>>>>>>>> java:1
>>>>>>>>>>>>>>>>>> 555)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(
>>>>>>>>>>>> Ag
>>>>>>>>>>>> gr
>>>>>>>>>>>>>>>>> egationClient.java:229)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
>>>>>>>>>>>>>>>>>> at EDRPAggregator.main(EDRPAggregator.java:77)
>>>>>>>>>>>>>>>>>> at =
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>>>>>>>>>>>> java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>>> cc
>>>>>>>>>>>> es
>>>>>>>>>>>>>>>>> sorImpl.java:25)
>>>>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>>>> at =
org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>>>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
>>>>>>>>>>>>>>>>>> ti
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> : No
>>>>>>>>>>>>>>>>> matching
>>>>>>>>>>>>>>>>>> handler for protocol
>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>>>>>>>>>>>>>>>>> in region
>>>>>>>>>>>>> =
EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
>>>>>>>>>>>>>>>>>> .j
>>>>>>>>>>>>>>>>>> av
>>>>>>>>>>>>>>>>>> a:4010
>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
>>>>>>>>>>>> r(
>>>>>>>>>>>> HR
>>>>>>>>>>>>>>>>> egionServer.java:3040)
>>>>>>>>>>>>>>>>>> at =
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>>>>>>>>>>>> java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>>> cc
>>>>>>>>>>>> es
>>>>>>>>>>>>>>>>> sorImpl.java:25)
>>>>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
>>>>>>>>>>>> eR
>>>>>>>>>>>> pc
>>>>>>>>>>>>>>>>> Engine.java:364)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>>>>>>>>>>>>>>>> 1325)
>>>>>>>>>>>>>>>>>> [sshexec]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
>>>>>>>>>>>>>>>>>> ti
>>>>>>>>>>>>>>>>>> ve
>>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
>>>>>>>>>>>> ru
>>>>>>>>>>>> ct
>>>>>>>>>>>>>>>>> orAccessorImpl.java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
>>>>>>>>>>>> in
>>>>>>>>>>>> gC
>>>>>>>>>>>>>>>>> onstructorAccessorImpl.java:27)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.lang.reflect.Constructor.newInstance(Constructor.java:
>>>>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>>>>> 3)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
>>>>>>>>>>>> io
>>>>>>>>>>>> n(
>>>>>>>>>>>>>>>>> RemoteExceptionHandler.java:96)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> =
tation.translateException(HConnectionManager.java:1651)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> =
tation.getRegionServerWithRetries(HConnectionManager.java:
>>>>>>>>>>>>>>>>> 13
>>>>>>>>>>>>>>>>> 27
>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>> vo
>>>>>>>>>>>>>>>>>> ker.ja
>>>>>>>>>>>>>>>>>> va:79)
>>>>>>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
>>>>>>>>>>>> ll
>>>>>>>>>>>> (A
>>>>>>>>>>>>>>>>> ggregationClient.java:233)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
>>>>>>>>>>>> ll
>>>>>>>>>>>> (A
>>>>>>>>>>>>>>>>> ggregationClient.java:230)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
>>>>>>>>>>>>>>>>>> ja
>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>> :303)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
>>>>>>>>>>>> Ex
>>>>>>>>>>>> ec
>>>>>>>>>>>>>>>>> utor.java:886)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
>>>>>>>>>>>> ut
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> .java:908)
>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep
>>>>>>>>>>>>>>>>>> ti
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>>>>>> : No
>>>>>>>>>>>>>>>>> matching
>>>>>>>>>>>>>>>>>> handler for protocol
>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
>>>>>>>>>>>>>>>>>> in region
>>>>>>>>>>>>> =
EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion
>>>>>>>>>>>>>>>>>> .j
>>>>>>>>>>>>>>>>>> av
>>>>>>>>>>>>>>>>>> a:4010
>>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso
>>>>>>>>>>>> r(
>>>>>>>>>>>> HR
>>>>>>>>>>>>>>>>> egionServer.java:3040)
>>>>>>>>>>>>>>>>>> at =
sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>>>>>>>>>>>>>>>> java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA
>>>>>>>>>>>> cc
>>>>>>>>>>>> es
>>>>>>>>>>>>>>>>> sorImpl.java:25)
>>>>>>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl
>>>>>>>>>>>> eR
>>>>>>>>>>>> pc
>>>>>>>>>>>>>>>>> Engine.java:364)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>> =
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
>>>>>>>>>>>>>>>>> 1325)
>>>>>>>>>>>>>>>>>> [sshexec]
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na
>>>>>>>>>>>>>>>>>> ti
>>>>>>>>>>>>>>>>>> ve
>>>>>>>>>>>>>>>>>> Method)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst
>>>>>>>>>>>> ru
>>>>>>>>>>>> ct
>>>>>>>>>>>>>>>>> orAccessorImpl.java:39)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat
>>>>>>>>>>>> in
>>>>>>>>>>>> gC
>>>>>>>>>>>>>>>>> onstructorAccessorImpl.java:27)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.lang.reflect.Constructor.newInstance(Constructor.java:
>>>>>>>>>>>>>>>>>> 51
>>>>>>>>>>>>>>>>>> 3)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept
>>>>>>>>>>>> io
>>>>>>>>>>>> n(
>>>>>>>>>>>>>>>>> RemoteExceptionHandler.java:96)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> =
tation.translateException(HConnectionManager.java:1651)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> =
tation.getRegionServerWithRetries(HConnectionManager.java:
>>>>>>>>>>>>>>>>> 13
>>>>>>>>>>>>>>>>> 27
>>>>>>>>>>>>>>>>> )
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC
>>>>>>>>>>>>>>>>>> In
>>>>>>>>>>>>>>>>>> vo
>>>>>>>>>>>>>>>>>> ker.ja
>>>>>>>>>>>>>>>>>> va:79)
>>>>>>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
>>>>>>>>>>>> ll
>>>>>>>>>>>> (A
>>>>>>>>>>>>>>>>> ggregationClient.java:233)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca
>>>>>>>>>>>> ll
>>>>>>>>>>>> (A
>>>>>>>>>>>>>>>>> ggregationClient.java:230)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl
>>>>>>>>>>>> em
>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.
>>>>>>>>>>>>>>>>>> ja
>>>>>>>>>>>>>>>>>> va
>>>>>>>>>>>>>>>>>> :303)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>> =
java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool
>>>>>>>>>>>> Ex
>>>>>>>>>>>> ec
>>>>>>>>>>>>>>>>> utor.java:886)
>>>>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>> =
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec
>>>>>>>>>>>> ut
>>>>>>>>>>>> or
>>>>>>>>>>>>>>>>> .java:908)
>>>>>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> The source for our class is:
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> import static org.junit.Assert.assertEquals;
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> import java.io.IOException;
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.conf.Configuration;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseTestingUtility;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.HConstants;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.HTableDescriptor;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.HTable;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Put;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Scan;
>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.AggregationCli
>>>>>>>>>>>>>>>>>> en
>>>>>>>>>>>>>>>>>> t;
>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.client.coprocessor.LongColumnInte
>>>>>>>>>>>>>>>>>> rp re ter; import org.apache.hadoop.hbase.util.Bytes;
>>>>>>>>>>>>>>>>>> import org.apache.hadoop.util.GenericOptionsParser;
>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>> =
org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
>>>>>>>>>>>>>>>>>> import
>>>>>>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
>>>>>>>>>>>>>>>>>> import org.junit.Test;
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> public class EDRPAggregator {
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> // private static final byte[] EDRP_FAMILY =3D
>>>>>>>>>>>>>>>>>> Bytes.toBytes("EDRP");
>>>>>>>>>>>>>>>>>> // private static final byte[] EDRP_QUALIFIER =3D
>>>>>>>>>>>>>>>>>> Bytes.toBytes("advanceKWh");
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> private static byte[] ROW =3D =
Bytes.toBytes("testRow");
>>>>>>>>>>>>>>>>>> private static final int ROWSIZE =3D 20;
>>>>>>>>>>>>>>>>>> private static byte[][] ROWS =3D makeN(ROW, ROWSIZE);
>>>>>>>>>>>>>>>>>> private static final byte[] TEST_QUALIFIER =3D
>>>>>>>>>>>>>>>>>> Bytes.toBytes("TestQualifier");
>>>>>>>>>>>>>>>>>> private static final byte[] TEST_MULTI_CQ =3D
>>>>>>>>>>>>>>>>>> Bytes.toBytes("TestMultiCQ");
>>>>>>>>>>>>>>>>>> private static final int rowSeperator1 =3D 5;
>>>>>>>>>>>>>>>>>>   private static final int rowSeperator2 =3D 12;
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> public static void
>>>>>>>>>>>>>>>>>> testSumWithValidRange(Configuration
>>>>>>>>>>>>>>>>>> conf,
>>>>>>>>>>>>>>>>>>               String[] otherArgs) throws Throwable {
>>>>>>>>>>>>>>>>>>        byte[] EDRP_TABLE =3D =
Bytes.toBytes(otherArgs[1]);
>>>>>>>>>>>>>>>>>>        byte[] EDRP_FAMILY =3D
>>>>>>>>>>>>>>>>>> Bytes.toBytes(otherArgs[2]);
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
>>>>>>>>>>>>>>>>>> );
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        // Utility.CreateHBaseTable(conf,
>>>>>>>>>>>>>>>>>> otherArgs[1],
>>>>>>>>>>>>>>>>> otherArgs[2],
>>>>>>>>>>>>>>>>>> true);
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        HBaseTestingUtility util =3D new
>>>>>>>>>>>>>>>>>> HBaseTestingUtility();
>>>>>>>>>>>>>>>>>>        HTable table =3D util.createTable(EDRP_TABLE,
>>>>>>>>>>>>>>>>>> EDRP_FAMILY);
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        AggregationClient aClient =3D new
>>>>>>>>>>>>>>>>>> AggregationClient(conf);
>>>>>>>>>>>>>>>>>>        Scan scan =3D new Scan();
>>>>>>>>>>>>>>>>>>        scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
>>>>>>>>>>>>>>>>>>        final ColumnInterpreter<Long, Long> ci =3D new
>>>>>>>>>>>>>>>>>> LongColumnInterpreter();
>>>>>>>>>>>>>>>>>>        long sum =3D
>>>>>>>>>>>>>>>>>> aClient.sum(Bytes.toBytes(otherArgs[0]),
>>>>>>>>>>>>>>>>>> ci,
>>>>>>>>>>>>>>>>> scan);
>>>>>>>>>>>>>>>>>>        System.out.println(sum);
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> /**
>>>>>>>>>>>>>>>>>> * Main entry point.
>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>  * @param argsThe
>>>>>>>>>>>>>>>>>> *            command line parameters.
>>>>>>>>>>>>>>>>>> * @throws Exception
>>>>>>>>>>>>>>>>>> *             When running the job fails.
>>>>>>>>>>>>>>>>>> */
>>>>>>>>>>>>>>>>>> public static void main(String[] args) throws
>>>>>>>>>>>>>>>>>> Exception {
>>>>>>>>>>>>>>>>>>        Configuration conf =3D
>>>>>>>>>>>>>>>>>> HBaseConfiguration.create();
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        String[] otherArgs =3D new
>>>>>>>>>>>>>>>>>> GenericOptionsParser(conf,
>>>>>>>>>>>>>>>>>> args)
>>>>>>>>>>>>>>>>>>                     .getRemainingArgs();
>>>>>>>>>>>>>>>>>>        if (otherArgs.length !=3D 3) {
>>>>>>>>>>>>>>>>>>               System.err
>>>>>>>>>>>>>>>>>>                            .println("Wrong number of
>>>>>>>>>>>>>>>>>> arguments: " + otherArgs.length);
>>>>>>>>>>>>>>>>>>               System.err.println("Usage: " +
>>>>>>>>>>>>>>>>>> "<tablename> <colfam> <qualifier>");
>>>>>>>>>>>>>>>>>>               System.exit(-1);
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>        try {
>>>>>>>>>>>>>>>>>>               testSumWithValidRange(conf, otherArgs);
>>>>>>>>>>>>>>>>>>        } catch (Throwable e) {
>>>>>>>>>>>>>>>>>>               e.printStackTrace();
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> /**
>>>>>>>>>>>>>>>>>> * an infrastructure method to prepare rows for the
>>>>>>>>>> testtable.
>>>>>>>>>>>>>>>>>> *
>>>>>>>>>>>>>>>>>>  * @param base
>>>>>>>>>>>>>>>>>> * @param n
>>>>>>>>>>>>>>>>>> * @return
>>>>>>>>>>>>>>>>>> */
>>>>>>>>>>>>>>>>>> private static byte[][] makeN(byte[] base, int n) {
>>>>>>>>>>>>>>>>>>        byte[][] ret =3D new byte[n][];
>>>>>>>>>>>>>>>>>>        for (int i =3D 0; i < n; i++) {
>>>>>>>>>>>>>>>>>>               ret[i] =3D Bytes.add(base,
>>>>>>>>>>>>>>>>>> Bytes.toBytes(i));
>>>>>>>>>>>>>>>>>>        }
>>>>>>>>>>>>>>>>>>        return ret;
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Please can you suggest what might be causing and/or =
how
>>>>>>>>>>>>>>>>>> we might fix this UnknownProtocolException?
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Also, does anyone have any working examples using the
>>>>>>>>>>>>>>>>>> aggregation
>>>>>>>>>>>>>>>>> protocol
>>>>>>>>>>>>>>>>>> other than the test code?
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>> Tom
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>>=20
>>>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>>=20
>>>>>>>>=20
>>>>>>>>=20
>>>>>>=20
>>>>>>=20
>>>>=20
>>>>=20
>>=20