hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Wilcox <Tom.Wil...@cse.org.uk>
Subject RE: AggregateProtocol Help
Date Fri, 23 Dec 2011 11:02:11 GMT
Hi,

I am not sure how we load the AggregateImplementation into the table. When we are creating
a table, we use the same functions as the test as follows...

...
>              conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>
> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation");
>
>              // Utility.CreateHBaseTable(conf, otherArgs[1], otherArgs[2],
> true);
>
>              HBaseTestingUtility util = new HBaseTestingUtility();
>              HTable table = util.createTable(EDRP_TABLE, EDRP_FAMILY);
>
>              AggregationClient aClient = new AggregationClient(conf);
...

Running DESCRIBE on a table produced shows the following output:

hbase(main):002:0> describe 'EDRP7'
DESCRIPTION                                                                              
                           ENABLED
 {NAME => 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS =>  true
 '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', BLOCKSIZE
=> '65536', IN_MEMORY => 'false', B
 LOCKCACHE => 'true'}]}

We are using the tip of 0.92 (cloned from the Git repo). See the version string below:

hbase(main):005:0> version
0.92.0, r1208286, Thu Dec 15 13:16:03 GMT 2011

We would really appreciate an example of how to create a table that is enabled to handle Aggregation).

Thanks


________________________________________
From: Ted Yu [yuzhihong@gmail.com]
Sent: 22 December 2011 17:03
To: user@hbase.apache.org
Subject: Re: AggregateProtocol Help

Have you loaded AggregateImplementation into your table ?
Can you show us the contents of the following command in hbase shell:
describe 'your-table'

BTW are you using the tip of 0.92 ?
HBASE-4946 would be of help for dynamically loaded coprocessors which you
might use in the future.

Cheers

On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox <Tom.Wilcox@cse.org.uk> wrote:

> Hi,
>
> We are trying to use the aggregation functionality in HBase 0.92  and we
> have managed to get the test code working using the following command:
>
> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
> org.junit.runner.JUnitCore
> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
>
> Closer inspection of this test class has revealed that it uses a mini DFS
> cluster to populate and run the tests. These tests return successfully.
>
> However, when we attempt to run similar code on our development HDFS
> cluster we experience the following error:
>
> [sshexec] 11/12/22 15:46:28 WARN
> client.HConnectionManager$HConnectionImplementation: Error executing for row
>  [sshexec] java.util.concurrent.ExecutionException:
> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No matching
> handler for protocol org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
>  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
>  [sshexec]
>  [sshexec]   at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>  [sshexec]   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processExecs(HConnectionManager.java:1465)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1555)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(AggregationClient.java:229)
>  [sshexec]   at
> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
>  [sshexec]   at EDRPAggregator.main(EDRPAggregator.java:77)
>  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>  [sshexec] Caused by:
> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No matching
> handler for protocol org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
>  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
>  [sshexec]
>  [sshexec]   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  [sshexec]   at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  [sshexec]   at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.translateException(HConnectionManager.java:1651)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1327)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
>  [sshexec]   at $Proxy3.getSum(Unknown Source)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:233)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:230)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1453)
>  [sshexec]   at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  [sshexec]   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  [sshexec]   at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  [sshexec]   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  [sshexec]   at java.lang.Thread.run(Thread.java:662)
>  [sshexec] org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No matching
> handler for protocol org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
>  [sshexec]   at
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
>  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
>  [sshexec]
>  [sshexec]   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>  [sshexec]   at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>  [sshexec]   at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>  [sshexec]   at
> java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>  [sshexec]   at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.translateException(HConnectionManager.java:1651)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1327)
>  [sshexec]   at
> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
>  [sshexec]   at $Proxy3.getSum(Unknown Source)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:233)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:230)
>  [sshexec]   at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1453)
>  [sshexec]   at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>  [sshexec]   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>  [sshexec]   at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>  [sshexec]   at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>  [sshexec]   at java.lang.Thread.run(Thread.java:662)
>
> The source for our class is:
>
> import static org.junit.Assert.assertEquals;
>
> import java.io.IOException;
>
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.HBaseConfiguration;
> import org.apache.hadoop.hbase.HBaseTestingUtility;
> import org.apache.hadoop.hbase.HConstants;
> import org.apache.hadoop.hbase.HTableDescriptor;
> import org.apache.hadoop.hbase.client.HTable;
> import org.apache.hadoop.hbase.client.Put;
> import org.apache.hadoop.hbase.client.Scan;
> import org.apache.hadoop.hbase.client.coprocessor.AggregationClient;
> import org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter;
> import org.apache.hadoop.hbase.util.Bytes;
> import org.apache.hadoop.util.GenericOptionsParser;
> import org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
> import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
> import org.junit.Test;
>
> public class EDRPAggregator {
>
>       // private static final byte[] EDRP_FAMILY = Bytes.toBytes("EDRP");
>       // private static final byte[] EDRP_QUALIFIER =
> Bytes.toBytes("advanceKWh");
>
>       private static byte[] ROW = Bytes.toBytes("testRow");
>       private static final int ROWSIZE = 20;
>       private static byte[][] ROWS = makeN(ROW, ROWSIZE);
>       private static final byte[] TEST_QUALIFIER =
> Bytes.toBytes("TestQualifier");
>       private static final byte[] TEST_MULTI_CQ =
> Bytes.toBytes("TestMultiCQ");
>       private static final int rowSeperator1 = 5;
>         private static final int rowSeperator2 = 12;
>
>       public static void testSumWithValidRange(Configuration conf,
>                     String[] otherArgs) throws Throwable {
>              byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
>              byte[] EDRP_FAMILY = Bytes.toBytes(otherArgs[2]);
>
>              conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
>
> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation");
>
>              // Utility.CreateHBaseTable(conf, otherArgs[1], otherArgs[2],
> true);
>
>              HBaseTestingUtility util = new HBaseTestingUtility();
>              HTable table = util.createTable(EDRP_TABLE, EDRP_FAMILY);
>
>              AggregationClient aClient = new AggregationClient(conf);
>              Scan scan = new Scan();
>              scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
>              final ColumnInterpreter<Long, Long> ci = new
> LongColumnInterpreter();
>              long sum = aClient.sum(Bytes.toBytes(otherArgs[0]), ci, scan);
>              System.out.println(sum);
>       }
>
>       /**
>       * Main entry point.
>       *
>        * @param argsThe
>       *            command line parameters.
>       * @throws Exception
>       *             When running the job fails.
>       */
>       public static void main(String[] args) throws Exception {
>              Configuration conf = HBaseConfiguration.create();
>
>              String[] otherArgs = new GenericOptionsParser(conf, args)
>                           .getRemainingArgs();
>              if (otherArgs.length != 3) {
>                     System.err
>                                  .println("Wrong number of arguments: " +
> otherArgs.length);
>                     System.err.println("Usage: " + "<tablename> <colfam>
> <qualifier>");
>                     System.exit(-1);
>              }
>
>              try {
>                     testSumWithValidRange(conf, otherArgs);
>              } catch (Throwable e) {
>                     e.printStackTrace();
>              }
>       }
>
>       /**
>       * an infrastructure method to prepare rows for the testtable.
>       *
>        * @param base
>       * @param n
>       * @return
>       */
>       private static byte[][] makeN(byte[] base, int n) {
>              byte[][] ret = new byte[n][];
>              for (int i = 0; i < n; i++) {
>                     ret[i] = Bytes.add(base, Bytes.toBytes(i));
>              }
>              return ret;
>       }
> }
>
> Please can you suggest what might be causing and/or how we might fix this
> UnknownProtocolException?
>
> Also, does anyone have any working examples using the aggregation protocol
> other than the test code?
>
> Thanks,
> Tom
>
>

Mime
View raw message