Return-Path: X-Original-To: apmail-hbase-user-archive@www.apache.org Delivered-To: apmail-hbase-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6A124D41B for ; Wed, 5 Sep 2012 21:28:30 +0000 (UTC) Received: (qmail 43573 invoked by uid 500); 5 Sep 2012 21:28:28 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 43527 invoked by uid 500); 5 Sep 2012 21:28:28 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 43516 invoked by uid 99); 5 Sep 2012 21:28:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 21:28:28 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of anilgupta84@gmail.com designates 209.85.220.169 as permitted sender) Received: from [209.85.220.169] (HELO mail-vc0-f169.google.com) (209.85.220.169) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 05 Sep 2012 21:28:22 +0000 Received: by vcbfl13 with SMTP id fl13so1887309vcb.14 for ; Wed, 05 Sep 2012 14:28:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=HRD80VIQFTY4Thyo0vj/wL4QaNq0Igzi7PfLbWdRVNw=; b=fBKK+K9jL6idZAbID0V4C0UbrgR3VFZOBE58cLNApcbwx9U3VJXNCOH/pKjgTETU+s aPuuWRhIxCTNUID+KnySfge2bwx7GR7g7XRMVxCXHEy9FMFSeHMvPdNkQEb1VGUUtFMW Z30rUf7RbeFWqqsoezU1kjajwrFKTRGdPsJkRLtnpjGHkj940iTrljPg1Lb25M6XDwgU mwJOpfNduvSfDzb3fJ5+JiDo+VvTlxPYR7oIKZJcV8BPQ8pichV0pbOzDwPjQqCcaaCw Si04OYCj6LR8A8O6Suffrz5Tho1rYmbJtw1lB0jKQc/BlXmfiON76hCFkOLyq5Ll2tRz cdfA== Received: by 10.58.221.66 with SMTP id qc2mr22557663vec.30.1346880481888; Wed, 05 Sep 2012 14:28:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.58.91.136 with HTTP; Wed, 5 Sep 2012 14:27:41 -0700 (PDT) In-Reply-To: References: From: anil gupta Date: Wed, 5 Sep 2012 14:27:41 -0700 Message-ID: Subject: Re: BigDecimalColumnInterpreter To: user@hbase.apache.org Content-Type: multipart/alternative; boundary=047d7bf162e4ad890104c8fb0c4b X-Virus-Checked: Checked by ClamAV on apache.org --047d7bf162e4ad890104c8fb0c4b Content-Type: text/plain; charset=ISO-8859-1 Hi Julian, Sorry for wrong reference to the aggregation client class in my previous email. Here is the right class:org.apache.hadoop.hbase.client.coprocessor.AggregationClient. HTH, Anil On Wed, Sep 5, 2012 at 2:04 PM, anil gupta wrote: > Hi Julian, > > I have been running the same class on my distributed cluster for > aggregation. It has been working fine. The only difference is that i use > the methods provided incom.intuit.ihub.hbase.poc.aggregation.client. > AggregationClient class. IMHO, you don't need to define an Endpoint for > using the BigDecimalColumnInterpreter. > > You need to use methods of AggregationClient: > sum(Bytes.toBytes(tableName), ci, scan), avg(final byte[] tableName, final > ColumnInterpreter ci, Scan scan), etc. > > In the above method you just need to pass the BigDecimalColumnInterpreter, > Scan object and Byte Array of TableName. It should work. Let me know if it > doesn't work this way.* > * > Thanks, > Anil Gupta > * > * > > On Wed, Sep 5, 2012 at 1:30 PM, Julian Wissmann wrote: > >> Thank you! >> So this looks like the missing link here. >> I'll see if I can get it working, tomorrow morning. >> >> Cheers >> >> 2012/9/5 Ted Yu >> >> > I added one review comment on >> > HBASE-6669 >> > . >> > >> > Thanks Julian for reminding me. >> > >> > On Wed, Sep 5, 2012 at 12:49 PM, Julian Wissmann >> > wrote: >> > >> > > I get supplied with doubles from sensors, but in the end I loose too >> much >> > > precision if I do my aggregations on double, otherwise I'd go for it. >> > > I use 0.92.1, from Cloudera CDH4. >> > > I've done some initial testing with LongColumnInterpreter on a dataset >> > that >> > > I've generated, to do some testing and get accustomed to stuff, but >> that >> > > worked like a charm after some initial stupidity on my side. >> > > So now I'm trying to do some testing with the real data, which comes >> in >> > as >> > > double and gets parsed to BigDecimal before writing. >> > > >> > > 2012/9/5 Ted Yu >> > > >> > > > And your HBase version is ? >> > > > >> > > > Since you use Double.parseDouble(), looks like it would be more >> > efficient >> > > > to develop DoubleColumnInterpreter. >> > > > >> > > > On Wed, Sep 5, 2012 at 12:07 PM, Julian Wissmann >> > > > wrote: >> > > > >> > > > > Hi, >> > > > > the schema looks like this: >> > > > > RowKey: id,timerange_timestamp,offset (String) >> > > > > Qualifier: Offset (long) >> > > > > Timestamp: timestamp (long) >> > > > > Value:number (BigDecimal) >> > > > > >> > > > > Or as code when I read data from csv:byte[] value = >> > > > > Bytes.toBytes(BigDecimal.valueOf(Double.parseDouble(cData[2]))); >> > > > > >> > > > > Cheers, >> > > > > >> > > > > Julian >> > > > > >> > > > > 2012/9/5 Ted Yu >> > > > > >> > > > > > You haven't told us the schema of your table yet. >> > > > > > Your table should have column whose value can be interpreted by >> > > > > > BigDecimalColumnInterpreter. >> > > > > > >> > > > > > Cheers >> > > > > > >> > > > > > On Wed, Sep 5, 2012 at 9:17 AM, Julian Wissmann < >> > > > > julian.wissmann@sdace.de >> > > > > > >wrote: >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > I am currently experimenting with the >> BigDecimalColumnInterpreter >> > > > from >> > > > > > > https://issues.apache.org/jira/browse/HBASE-6669. >> > > > > > > >> > > > > > > I was thinking the best way for me to work with it would be to >> > use >> > > > the >> > > > > > Java >> > > > > > > class and just use that as is. >> > > > > > > >> > > > > > > Imported it into my project and tried to work with it as is, >> by >> > > just >> > > > > > > instantiating the ColumnInterpreter as >> > BigDecimalColumnInterpreter. >> > > > > Okay, >> > > > > > > threw errors and also complained about not knowing where to >> find >> > > > such a >> > > > > > > class. >> > > > > > > >> > > > > > > So I did some reading and found out, that I'd need to have an >> > > > Endpoint >> > > > > > for >> > > > > > > it. So I imported AggregateImplementation and >> AggregateProtocol >> > > into >> > > > my >> > > > > > > workspace, renamed them, and refactored them where necessary >> to >> > > take >> > > > > > > BigDecimal. Re-exported the jar, then and had another try. >> > > > > > > >> > > > > > > So when I call: >> > > > > > > ------ >> > > > > > > final Scan scan = new Scan((metricID + "," + >> > > > > basetime_begin).getBytes(), >> > > > > > > (metricID + "," + basetime_end).getBytes()); >> > > > > > > scan.addFamily(family.getBytes()); >> > > > > > > final ColumnInterpreter ci = new >> > > > > > > BigDecimalColumnInterpreter(); >> > > > > > > Map results = >> > > > > > > table.coprocessorExec(BigDecimalProtocol.class, null, null, >> > > > > > > new Batch.Call() { >> > > > > > > public BigDecimal call(BigDecimalProtocol >> instance)throws >> > > > > > > IOException{ >> > > > > > > return instance.getMax(ci, scan); >> > > > > > > } >> > > > > > > }); >> > > > > > > ------ >> > > > > > > I get errors in the log again, that it can't find >> > > > > > > BigDecimalColumnInterpreter... okay, so I tried >> > > > > > > ------ >> > > > > > > Scan scan = new Scan((metricID + "," + >> > basetime_begin).getBytes(), >> > > > > > > (metricID + "," + basetime_end).getBytes()); >> > > > > > > scan.addFamily(family.getBytes()); >> > > > > > > final ColumnInterpreter ci = new >> > > > > > > BigDecimalColumnInterpreter(); >> > > > > > > AggregationClient ag = new AggregationClient(config); >> > > > > > > BigDecimal max = ag.max(Bytes.toBytes(tableName), ci, scan); >> > > > > > > ------ >> > > > > > > I don't get errors recored in the log anymore, but a load of >> Java >> > > > error >> > > > > > > output: >> > > > > > > ------ >> > > > > > > >> > > > > > > java.util.concurrent.ExecutionException: >> > > > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Failed >> > > > after >> > > > > > > attempts=10, exceptions: >> > > > > > > Wed Sep 05 18:13:43 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:44 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:45 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:46 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:49 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:51 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:55 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:59 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:07 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:23 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > >> > > > > > > at >> > > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >> > > > > > > at java.util.concurrent.FutureTask.get(FutureTask.java:111) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processExecs(HConnectionManager.java:1434) >> > > > > > > at >> > > > > > >> > > >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1263) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(AggregationClient.java:259) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> helpers.HbaseReaderBigDecimal.getWeeksumSCAN(HbaseReaderBigDecimal.java:360) >> > > > > > > at >> > > helpers.HbaseReaderBigDecimal.main(HbaseReaderBigDecimal.java:81) >> > > > > > > Caused by: >> > > org.apache.hadoop.hbase.client.RetriesExhaustedException: >> > > > > > Failed >> > > > > > > after attempts=10, exceptions: >> > > > > > > Wed Sep 05 18:13:43 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:44 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:45 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:46 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:49 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:51 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:55 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:59 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:07 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:23 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183) >> > > > > > > at >> > > > > > > >> > > > > >> > > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79) >> > > > > > > at $Proxy7.getSum(Unknown Source) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:263) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:260) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1422) >> > > > > > > at >> > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> > > > > > > at java.lang.Thread.run(Thread.java:679) >> > > > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: >> Failed >> > > > after >> > > > > > > attempts=10, exceptions: >> > > > > > > Wed Sep 05 18:13:43 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:44 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:45 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:46 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:49 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:51 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:55 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:13:59 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:07 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > Wed Sep 05 18:14:23 CEST 2012, >> > > > > > > org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@50502819, >> > > > > > > java.io.IOException: >> > > > > > > IPC server unable to read call parameters: Error in readFields >> > > > > > > >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:183) >> > > > > > > at >> > > > > > > >> > > > > >> > > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCInvoker.java:79) >> > > > > > > at $Proxy7.getSum(Unknown Source) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:263) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call(AggregationClient.java:260) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation$4.call(HConnectionManager.java:1422) >> > > > > > > at >> > > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> > > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >> > > > > > > at >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >> > > > > > > at java.lang.Thread.run(Thread.java:679) >> > > > > > > >> > > > > > > ------ >> > > > > > > >> > > > > > > I'm not really sure about what I'm doing wrong. Does anyone >> have >> > a >> > > > hint >> > > > > > > towards the right direction? >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > > > > -- > Thanks & Regards, > Anil Gupta > -- Thanks & Regards, Anil Gupta --047d7bf162e4ad890104c8fb0c4b--