Return-Path: Delivered-To: apmail-hbase-user-archive@www.apache.org Received: (qmail 86682 invoked from network); 7 Feb 2011 16:54:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Feb 2011 16:54:24 -0000 Received: (qmail 55937 invoked by uid 500); 7 Feb 2011 16:54:23 -0000 Delivered-To: apmail-hbase-user-archive@hbase.apache.org Received: (qmail 55388 invoked by uid 500); 7 Feb 2011 16:54:20 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 55373 invoked by uid 99); 7 Feb 2011 16:54:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:54:18 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of saint.ack@gmail.com designates 209.85.215.169 as permitted sender) Received: from [209.85.215.169] (HELO mail-ey0-f169.google.com) (209.85.215.169) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Feb 2011 16:54:13 +0000 Received: by eyh6 with SMTP id 6so2392378eyh.14 for ; Mon, 07 Feb 2011 08:53:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=c0ou0V6P8BMDVpkoUNZrKd18b1ZxhI4GAFpguwNydd8=; b=BpVDygQlhscjYRsiJyThI5jhc9cyC4LJ9eAF+cqFRcsKwDeC6jWRc8d5rRci0ud8YT Oh9F6Za2BSS4dcz2JyZDGKo8w50MTII+OrDivDgWh4hbsljWSpXfyiWJ3tGIdrO7wCfd qbGCBXxotjermVIZjnhljGd4fcIuFi3E8GWOk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=virmfcjfyF0bbZh6L12DEY22nf/UUN/mFBIsUjqCa5zSc5PGfpcJTjTsRVHcFa9iSj UniyhZTuu83sFIbwS58aX93LXZTgwtzhRM2ikFyEFZzc6ed+r6kiRNplramC8Cr1QsqE AANz7etKMd2ApdB+4ho9CHpems4MEd9jqUy10= MIME-Version: 1.0 Received: by 10.216.163.203 with SMTP id a53mr13828744wel.104.1297097632314; Mon, 07 Feb 2011 08:53:52 -0800 (PST) Sender: saint.ack@gmail.com Received: by 10.216.49.11 with HTTP; Mon, 7 Feb 2011 08:53:52 -0800 (PST) In-Reply-To: References: Date: Mon, 7 Feb 2011 08:53:52 -0800 X-Google-Sender-Auth: TCF0QztoV--S-m7deFFVkFfrQDc Message-ID: Subject: Re: Type mismatch From: Stack To: user@hbase.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks for writing back the list Mark (I should have spotted that -- sorry)= . St.Ack On Sun, Feb 6, 2011 at 8:11 PM, Mark Kerzner wrote: > And the correct answer is... instead of this signature > > =A0 =A0public static class RowCounterReducer > =A0 =A0 =A0 =A0 =A0 =A0extends TableReducer ImmutableBytesWritable> > =A0 =A0{ > =A0 =A0 =A0 =A0public void reduce(Text key, > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Iterable values, > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Reducer.Context context) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0throws IOException, > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0InterruptedException { > > (WRONG!) > > I used this signature > > =A0 =A0 public static class RowCounterReducer extends > TableReducer= { > > =A0 =A0 =A0 =A0@Override > =A0 =A0 =A0 =A0public void reduce(ImmutableBytesWritable key, Iterable > values, Context context) > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0throws IOException, InterruptedException { > > RIGHT! > > Thank you, all. > > Mark > > On Fri, Feb 4, 2011 at 2:01 PM, Stack wrote: > >> (Thanks Sujee) >> >> What did you change in your src to get it going? >> >> St.Ack >> >> On Fri, Feb 4, 2011 at 10:56 AM, Mark Kerzner >> wrote: >> > I found an example that works and uses the latest HBase API, >> > http://sujee.net/tech/articles/hbase-map-reduce-freq-counter/, you mig= ht >> > know about it, but for me it was very helpful. >> > >> > Mark >> > >> > On Fri, Feb 4, 2011 at 11:55 AM, Stack wrote: >> > >> >> Its just an issue of matching your outputs to TOF. =A0 There are >> >> examples of Reducer usage in the mapreduce package. =A0They declare >> >> their types other than how you have it. =A0See PutSortReducer and >> >> ImportTsv which uses it (and configures it up). >> >> >> >> St.Ack >> >> >> >> On Fri, Feb 4, 2011 at 7:08 AM, Mark Kerzner >> >> wrote: >> >> > I tried 0.90 - same error. I am going to try to build HBase from co= de >> and >> >> > include this code in my debugging session, to step through it. But = I >> must >> >> be >> >> > doing something wrong. >> >> > >> >> > How does one write to HBase in the Reducer, is there any example!??= ? >> >> > >> >> > Thank you! >> >> > >> >> > Mark >> >> > >> >> > On Fri, Feb 4, 2011 at 12:38 AM, Stack wrote: >> >> > >> >> >> I'm not sure whats up w/ your sample above. =A0Here's some observa= tions >> >> >> that might help. >> >> >> >> >> >> Here is the code. =A0Our line numbers differ. =A0You are not on 0.= 90.0? >> >> >> Thats not important. =A0You are in this method it seems: >> >> >> >> >> >> >> >> >> http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/TableOutp= utFormat.html#124 >> >> >> =A0See the message on the end. =A0You should submit a patch where = we add >> >> >> to the IOException message a toString on the value passed so we ha= ve >> a >> >> >> better clue as to where we are off here -- so you can see class of >> >> >> object submitted (debugging, I'd add this to the log message). >> >> >> >> >> >> Looking at how you declare TOF, it doesn't look right (This helps >> with >> >> >> that: >> >> >> >> >> >> http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/TableRedu= cer.html >> >> >> ). >> >> >> =A0It seems like the declaration should be >> =A0but >> >> >> you are outputting a Text for KEYOUT, not the declared Put. =A0Thi= s is >> >> >> probably not your prob. though. >> >> >> >> >> >> Looking at IdentityTableReducer, it just passes Writables with the >> >> >> value a Delete or Put. >> >> >> >> >> >> St.Ack >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Feb 3, 2011 at 10:00 PM, Mark Kerzner > > >> >> >> wrote: >> >> >> > Thank you, St.Ack, it is very nice of you to keep helping me. He= re >> is >> >> the >> >> >> > stack :) trace, but as you can see, it is the internal Hadoop co= de. >> I >> >> see >> >> >> > this code and I see the message - I am not passing it the right >> object >> >> - >> >> >> but >> >> >> > how DO I pass the right object? >> >> >> > >> >> >> > M >> >> >> > >> >> >> > >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr= ite(TableOutputFormat.java:106) >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.wr= ite(TableOutputFormat.java:65) >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(Reduce= Task.java:512) >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> >> >> >> >> >> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutput= Context.java:80) >> >> >> > =A0 =A0 =A0 =A0at >> org.apache.hadoop.mapreduce.Reducer.reduce(Reducer.java:156) >> >> >> > =A0 =A0 =A0 =A0at org.apache.hadoop.mapreduce.Reducer.run(Reduce= r.java:176) >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:570) >> >> >> > =A0 =A0 =A0 =A0at >> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:412) >> >> >> > =A0 =A0 =A0 =A0at >> >> >> > >> >> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:2= 58) >> >> >> > >> >> >> > On Thu, Feb 3, 2011 at 11:52 PM, Stack wrote: >> >> >> > >> >> >> >> Look at the stack trace. =A0See where its being thrown. =A0Look= at >> that >> >> >> >> src code at that line offset. =A0Should give you a clue. >> >> >> >> St.Ack >> >> >> >> >> >> >> >> On Thu, Feb 3, 2011 at 9:36 PM, Mark Kerzner < >> markkerzner@gmail.com> >> >> >> >> wrote: >> >> >> >> > Thank you, that helped, but now I get this error on trying to >> write >> >> >> back >> >> >> >> to >> >> >> >> > HBase: >> >> >> >> > >> >> >> >> > java.io.IOException: Pass a Delete or a Put >> >> >> >> > >> >> >> >> > Here is a fragment on my code. Again, thanks a bunch! >> >> >> >> > >> >> >> >> > =A0 =A0public static class RowCounterReducer >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0extends TableReducer >> >> >> >> > =A0 =A0{ >> >> >> >> > =A0 =A0 =A0 =A0public void reduce(Text key, >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Iterable values, >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Reducer.Context context) >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0throws IOException, >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0InterruptedException { >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0Iterator iterator =3D va= lues.iterator(); >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0while (iterator.hasNext()) { >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0IntWritable value =3D iterator= .next(); >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Put put =3D new Put(); >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0context.write(key, put); >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0} >> >> >> >> > =A0 =A0 =A0 =A0} >> >> >> >> > =A0 =A0} >> >> >> >> > >> >> >> >> > >> >> >> >> > On Thu, Feb 3, 2011 at 2:50 PM, Stack wrot= e: >> >> >> >> > >> >> >> >> >> You are emitting a Text type. =A0Try just passing 'row' to t= he >> >> context, >> >> >> >> >> the one passed in to your map. >> >> >> >> >> St.Ack >> >> >> >> >> >> >> >> >> >> On Thu, Feb 3, 2011 at 12:23 PM, Mark Kerzner < >> >> markkerzner@gmail.com >> >> >> > >> >> >> >> >> wrote: >> >> >> >> >> > Hi, >> >> >> >> >> > >> >> >> >> >> > I have this code to read and write to HBase from MR, and i= t >> >> works >> >> >> fine >> >> >> >> >> with >> >> >> >> >> > 0 reducers, but it gives a type mismatch error when with 1 >> >> reducer. >> >> >> >> What >> >> >> >> >> > should I look at? *Thank you!* >> >> >> >> >> > >> >> >> >> >> > *Code:* >> >> >> >> >> > >> >> >> >> >> > =A0 =A0static class RowCounterMapper >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0extends TableMapper { >> >> >> >> >> > >> >> >> >> >> > =A0 =A0 =A0 =A0private static enum Counters { >> >> >> >> >> > >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0ROWS >> >> >> >> >> > =A0 =A0 =A0 =A0} >> >> >> >> >> > >> >> >> >> >> > =A0 =A0 =A0 =A0@Override >> >> >> >> >> > =A0 =A0 =A0 =A0public void map(ImmutableBytesWritable row,= Result >> >> values, >> >> >> >> Context >> >> >> >> >> > context) >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0throws IOException, Interru= ptedException { >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0for (KeyValue value : values.list()= ) { >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (value.getValue().length= > 0) { >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0Text key =3D new Te= xt(value.getValue()); >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0context.write(key, = ONE); >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0} >> >> >> >> >> > =A0 =A0 =A0 =A0 =A0 =A0} >> >> >> >> >> > =A0 =A0 =A0 =A0} >> >> >> >> >> > =A0 =A0} >> >> >> >> >> > >> >> >> >> >> > *Error: * >> >> >> >> >> > >> >> >> >> >> > java.io.IOException: Type mismatch in key from map: expect= ed >> >> >> >> >> > org.apache.hadoop.hbase.io.ImmutableBytesWritable, recieve= d >> >> >> >> >> > org.apache.hadoop.io.Text >> >> >> >> >> > >> >> >> >> >> >> >> >> >> > >> >> >> >> >> >> >> > >> >> >> >> >> > >> >> >> > >> >