hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nishant Khurana" <nishant2...@gmail.com>
Subject Re: Exception while running a mapreduce job
Date Fri, 28 Nov 2008 20:50:14 GMT
Thanks,
It worked :) . One more question. When I store Integer values as row keys or
any column values and run scan table from hbase shell, they come like this :
\000\000C|                  column=year:, timestamp=1227905036961,
value=1999
 \000\000C~                  column=name:, timestamp=1227905036962,
value=The 39 Steps
 \000\000C~                  column=yahoo_movie_id:,
timestamp=1227905036962, value=k{I\357\277\275
 \000\000C~                  column=year:, timestamp=1227905036962,
value=1935
 \000\000C\200               column=name:, timestamp=1227905036962,
value=Prophecy
 \000\000C\200               column=yahoo_movie_id:,
timestamp=1227905036962, value=k\357\277\275\n@
 \000\000C\200               column=year:, timestamp=1227905036962,
value=1979

Notice the first column and value part both of which were integers. Is it
because they get converted to ImmutableBytesWritable that they look like
this ? Can I store them in readable form ?
Thanks




On Fri, Nov 28, 2008 at 3:08 PM, stack <stack@duboce.net> wrote:

> How is job being setup?  I'd suspect you are calling initTableReduceJob in
> job setup.  Look at what it does.  It sets the reduce key type.  Maybe
> after
> calling it, reset the reduce key type to IntWritable.
> St.Ack
>
>
>
> On Fri, Nov 28, 2008 at 11:48 AM, Nishant Khurana <nishant2984@gmail.com
> >wrote:
>
> > Hi,
> > I am trying to run a map reduce job which parses a text file and fills up
> a
> > Hbase Table. Following is the code :
> >
> >
> > public class UploadMoviesList extends Configured implements Tool
> > {
> >   public static class MapClass extends MapReduceBase implements
> > Mapper<LongWritable, Text, IntWritable, MapWritable>
> >   {
> >      public void map(LongWritable key, Text value,
> > OutputCollector<IntWritable, MapWritable> output, Reporter reporter)
> throws
> > IOException
> >      {
> >         String line = value.toString();
> >         String[] result = line.split("%");
> >         MapWritable mw = new MapWritable();
> >         mw.put(new Text("year:year"), new Text(result[1].toString()));
> >         mw.put(new Text("name:name"), new Text(result[2].toString()));
> >         int a = new Integer(result[3]).intValue();
> >         mw.put(new Text("y_movie_id:y_movie_id"), new IntWritable(a));
> >         int b = new Integer(result[0]).intValue();
> >         output.collect(new IntWritable(b), mw);
> >      }
> >   }
> >
> >   public static class ReduceClass extends TableReduce<IntWritable,
> > MapWritable>
> >   {
> >    @Override
> >    public void reduce(IntWritable key, Iterator<MapWritable> values,
> > OutputCollector<ImmutableBytesWritable, BatchUpdate> output, Reporter
> > reporter) throws IOException
> >    {
> >       reporter.setStatus("Reducer committing " + key);
> >       ImmutableBytesWritable ibw = new
> > ImmutableBytesWritable(Bytes.toBytes(key.get()));
> >       BatchUpdate outval = new BatchUpdate(Bytes.toBytes(key.get()));
> >        while (values.hasNext())
> >        {
> >          MapWritable hmw = new MapWritable(values.next());
> >          outval.put("year:year",
> > Bytes.toBytes(hmw.get("year:year").toString()));
> >          outval.put("name:name",
> > Bytes.toBytes(hmw.get("name:name").toString()));
> >          IntWritable iw =
> (IntWritable)(hmw.get("y_movie_id:y_movie_id"));
> >          outval.put("y_movie_id:y_movie_id", Bytes.toBytes(iw.get()));
> >          output.collect(ibw,outval);
> >       }
> >    }
> >   }
> >
> >
> > When I try to run it, I am getting following exceptions :
> > 08/11/28 14:42:27 INFO mapred.JobClient: Task Id :
> > attempt_200811281158_0005_m_000001_0, Status : FAILED
> > java.io.IOException: Type mismatch in key from map: expected
> > org.apache.hadoop.hbase.io.ImmutableBytesWritable, recieved
> > org.apache.hadoop.io.IntWritable
> >    at
> >
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:415)
> >    at dist_q_data.UploadMoviesList$MapClass.map(UploadMoviesList.java:45)
> >    at dist_q_data.UploadMoviesList$MapClass.map(UploadMoviesList.java:1)
> >    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
> >    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> >    at
> > org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
> >
> >
> > I don't know why it says it expects a ImmutableBytesWritable key. Any
> > suggestions ?
> > Thanks
> >
> > --
> > Nishant Khurana
> > Candidate for Masters in Engineering (Dec 2009)
> > Computer and Information Science
> > School of Engineering and Applied Science
> > University of Pennsylvania
> >
>



-- 
Nishant Khurana
Candidate for Masters in Engineering (Dec 2009)
Computer and Information Science
School of Engineering and Applied Science
University of Pennsylvania

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message