hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: AggregateProtocol Help
Date Fri, 23 Dec 2011 15:04:46 GMT
I don't know why you chose HBaseTestingUtility to create the table.
I guess you followed test code example.

At least you should pass the conf to this ctor:
  public HBaseTestingUtility(Configuration conf) {

If coprocessor was installed correctly, you should see something like(from
HBASE-5070):
coprocessor$1 =>
'|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'

Cheers

On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox <Tom.Wilcox@cse.org.uk> wrote:

> Hi,
>
> I am not sure how we load the AggregateImplementation into the table. When
> we are creating a table, we use the same functions as the test as follows...
>
> ...
> >              conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >
> > "org.apache.hadoop.hbase.coprocessor.AggregateImplementation");
> >
> >              // Utility.CreateHBaseTable(conf, otherArgs[1],
> otherArgs[2],
> > true);
> >
> >              HBaseTestingUtility util = new HBaseTestingUtility();
> >              HTable table = util.createTable(EDRP_TABLE, EDRP_FAMILY);
> >
> >              AggregationClient aClient = new AggregationClient(conf);
> ...
>
> Running DESCRIBE on a table produced shows the following output:
>
> hbase(main):002:0> describe 'EDRP7'
> DESCRIPTION
>                                            ENABLED
>  {NAME => 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER =>
> 'NONE', REPLICATION_SCOPE => '0', VERSIONS =>  true
>  '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', B
>  LOCKCACHE => 'true'}]}
>
> We are using the tip of 0.92 (cloned from the Git repo). See the version
> string below:
>
> hbase(main):005:0> version
> 0.92.0, r1208286, Thu Dec 15 13:16:03 GMT 2011
>
> We would really appreciate an example of how to create a table that is
> enabled to handle Aggregation).
>
> Thanks
>
>
> ________________________________________
> From: Ted Yu [yuzhihong@gmail.com]
> Sent: 22 December 2011 17:03
> To: user@hbase.apache.org
> Subject: Re: AggregateProtocol Help
>
> Have you loaded AggregateImplementation into your table ?
> Can you show us the contents of the following command in hbase shell:
> describe 'your-table'
>
> BTW are you using the tip of 0.92 ?
> HBASE-4946 would be of help for dynamically loaded coprocessors which you
> might use in the future.
>
> Cheers
>
> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox <Tom.Wilcox@cse.org.uk> wrote:
>
> > Hi,
> >
> > We are trying to use the aggregation functionality in HBase 0.92  and we
> > have managed to get the test code working using the following command:
> >
> > java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
> > org.junit.runner.JUnitCore
> > org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
> >
> > Closer inspection of this test class has revealed that it uses a mini DFS
> > cluster to populate and run the tests. These tests return successfully.
> >
> > However, when we attempt to run similar code on our development HDFS
> > cluster we experience the following error:
> >
> > [sshexec] 11/12/22 15:46:28 WARN
> > client.HConnectionManager$HConnectionImplementation: Error executing for
> row
> >  [sshexec] java.util.concurrent.ExecutionException:
> > org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> > org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No
> matching
> > handler for protocol
> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> > in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >  [sshexec]   at
> > org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
> >  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
> >  [sshexec]
> >  [sshexec]   at
> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >  [sshexec]   at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processExecs(HConnectionManager.java:1465)
> >  [sshexec]   at
> > org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1555)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(AggregationClient.java:229)
> >  [sshexec]   at
> > EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >  [sshexec]   at EDRPAggregator.main(EDRPAggregator.java:77)
> >  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
> >  [sshexec]   at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >  [sshexec] Caused by:
> > org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> > org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No
> matching
> > handler for protocol
> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> > in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >  [sshexec]   at
> > org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
> >  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
> >  [sshexec]
> >  [sshexec]   at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >  [sshexec]   at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.translateException(HConnectionManager.java:1651)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1327)
> >  [sshexec]   at
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
> >  [sshexec]   at $Proxy3.getSum(Unknown Source)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:233)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:230)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1453)
> >  [sshexec]   at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >  [sshexec]   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >  [sshexec]   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >  [sshexec]   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >  [sshexec]   at java.lang.Thread.run(Thread.java:662)
> >  [sshexec] org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> > org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: No
> matching
> > handler for protocol
> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> > in region EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >  [sshexec]   at
> > org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:4010)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3040)
> >  [sshexec]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >  [sshexec]   at java.lang.reflect.Method.invoke(Method.java:597)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1325)
> >  [sshexec]
> >  [sshexec]   at
> > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> >  [sshexec]   at
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> >  [sshexec]   at
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> >  [sshexec]   at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.translateException(HConnectionManager.java:1651)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1327)
> >  [sshexec]   at
> > org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79)
> >  [sshexec]   at $Proxy3.getSum(Unknown Source)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:233)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:230)
> >  [sshexec]   at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1453)
> >  [sshexec]   at
> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >  [sshexec]   at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >  [sshexec]   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >  [sshexec]   at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >  [sshexec]   at java.lang.Thread.run(Thread.java:662)
> >
> > The source for our class is:
> >
> > import static org.junit.Assert.assertEquals;
> >
> > import java.io.IOException;
> >
> > import org.apache.hadoop.conf.Configuration;
> > import org.apache.hadoop.hbase.HBaseConfiguration;
> > import org.apache.hadoop.hbase.HBaseTestingUtility;
> > import org.apache.hadoop.hbase.HConstants;
> > import org.apache.hadoop.hbase.HTableDescriptor;
> > import org.apache.hadoop.hbase.client.HTable;
> > import org.apache.hadoop.hbase.client.Put;
> > import org.apache.hadoop.hbase.client.Scan;
> > import org.apache.hadoop.hbase.client.coprocessor.AggregationClient;
> > import org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter;
> > import org.apache.hadoop.hbase.util.Bytes;
> > import org.apache.hadoop.util.GenericOptionsParser;
> > import org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
> > import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
> > import org.junit.Test;
> >
> > public class EDRPAggregator {
> >
> >       // private static final byte[] EDRP_FAMILY = Bytes.toBytes("EDRP");
> >       // private static final byte[] EDRP_QUALIFIER =
> > Bytes.toBytes("advanceKWh");
> >
> >       private static byte[] ROW = Bytes.toBytes("testRow");
> >       private static final int ROWSIZE = 20;
> >       private static byte[][] ROWS = makeN(ROW, ROWSIZE);
> >       private static final byte[] TEST_QUALIFIER =
> > Bytes.toBytes("TestQualifier");
> >       private static final byte[] TEST_MULTI_CQ =
> > Bytes.toBytes("TestMultiCQ");
> >       private static final int rowSeperator1 = 5;
> >         private static final int rowSeperator2 = 12;
> >
> >       public static void testSumWithValidRange(Configuration conf,
> >                     String[] otherArgs) throws Throwable {
> >              byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
> >              byte[] EDRP_FAMILY = Bytes.toBytes(otherArgs[2]);
> >
> >              conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >
> > "org.apache.hadoop.hbase.coprocessor.AggregateImplementation");
> >
> >              // Utility.CreateHBaseTable(conf, otherArgs[1],
> otherArgs[2],
> > true);
> >
> >              HBaseTestingUtility util = new HBaseTestingUtility();
> >              HTable table = util.createTable(EDRP_TABLE, EDRP_FAMILY);
> >
> >              AggregationClient aClient = new AggregationClient(conf);
> >              Scan scan = new Scan();
> >              scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
> >              final ColumnInterpreter<Long, Long> ci = new
> > LongColumnInterpreter();
> >              long sum = aClient.sum(Bytes.toBytes(otherArgs[0]), ci,
> scan);
> >              System.out.println(sum);
> >       }
> >
> >       /**
> >       * Main entry point.
> >       *
> >        * @param argsThe
> >       *            command line parameters.
> >       * @throws Exception
> >       *             When running the job fails.
> >       */
> >       public static void main(String[] args) throws Exception {
> >              Configuration conf = HBaseConfiguration.create();
> >
> >              String[] otherArgs = new GenericOptionsParser(conf, args)
> >                           .getRemainingArgs();
> >              if (otherArgs.length != 3) {
> >                     System.err
> >                                  .println("Wrong number of arguments: " +
> > otherArgs.length);
> >                     System.err.println("Usage: " + "<tablename> <colfam>
> > <qualifier>");
> >                     System.exit(-1);
> >              }
> >
> >              try {
> >                     testSumWithValidRange(conf, otherArgs);
> >              } catch (Throwable e) {
> >                     e.printStackTrace();
> >              }
> >       }
> >
> >       /**
> >       * an infrastructure method to prepare rows for the testtable.
> >       *
> >        * @param base
> >       * @param n
> >       * @return
> >       */
> >       private static byte[][] makeN(byte[] base, int n) {
> >              byte[][] ret = new byte[n][];
> >              for (int i = 0; i < n; i++) {
> >                     ret[i] = Bytes.add(base, Bytes.toBytes(i));
> >              }
> >              return ret;
> >       }
> > }
> >
> > Please can you suggest what might be causing and/or how we might fix this
> > UnknownProtocolException?
> >
> > Also, does anyone have any working examples using the aggregation
> protocol
> > other than the test code?
> >
> > Thanks,
> > Tom
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message