hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From llpind <sonny_h...@hotmail.com>
Subject Re: Help with Map/Reduce program
Date Wed, 10 Jun 2009 17:03:33 GMT

Thanks.

Yea I've got that colFam for sure in the HBase table. 

I've been trying to play with rowcounter, and not having much luck either.

I run the command:
hadoop19/bin/hadoop org.apache.hadoop.hbase.mapred.Driver rowcounter
/home/hadoop/dev/rowcounter7 tableA colFam1:


The map/reduce finishes just like it does with my own program, but with all
part files empty in /home/hadoop/dev/rowcounter7.   

Any Ideas?

Billy Pearson-2 wrote:
> 
> You could try scanning it with shell to make sure there is data
> bin/hbase shell
> help ->  hbase> scan 't1', {COLUMNS => ['c1', 'c2'], LIMIT => 10, \
>              STARTROW => 'xyz'}
> 
> so something like
> scan 'tablename', {COLUMNS => ['col1']}
> 
> That will spit out data if there is any
> I thank you might be able to even call
> scan 'tablename'
> 
> Billy
> 
> 
> 
> 
> "llpind" <sonny_heer@hotmail.com> wrote in 
> message news:23954242.post@talk.nabble.com...
>>
>> Yeah I noticed that shortly after I posted.  I have it "colFam1:" now. 
>> I'm
>> positive my table has that column family, but my output table still has
>> nothing in it.
>>
>> I'm looking at the source code for rowcounter, and it doesn't even
>> require
>> :.  does it need to be passed in?
>>
>>
>> I may be going about this wrong, I'm open to ideas.  I need a way to 
>> iterate
>> over an entire HBase table, and count columns (column counter instead of
>> rowcounter).
>>
>> Billy Pearson-2 wrote:
>>>
>>> try with out the * in the column
>>> "colFam1:*",
>>> try
>>> "colFam1:",
>>>
>>> I do not thank the * works like a all option just leave it blank
>>> colFam1:
>>> and it will give all results
>>>
>>> Billy
>>>
>>>
>>> "llpind" <sonny_heer@hotmail.com> wrote in
>>> message 
>>> news:23952252.post@talk.nabble.com...
>>>>
>>>> Hi again,
>>>>
>>>> I need some help with a map/reduce program I have which copies data
>>>> from
>>>> one
>>>> table to another.  What I would like to do is iterate through an entire
>>>> HBase table, and for a given row key and column family count the number
>>>> of
>>>> records.  So the output table will have a single column family named
>>>> 'count'
>>>> (e.g. entry would look something like 'rowKey1',
>>>> 'count:count_for_rowkey1',
>>>> '534', where the rowkey could be the same as input table ).
>>>>
>>>> here is my first attempt:
>>>>
>>>> CONF:
>>>> =========================================================================
>>>>
>>>> c.setInputFormat(TableInputFormat.class);
>>>> c.setOutputFormat(TableOutputFormat.class);
>>>>
>>>>     TableMapReduceUtil.initTableMapJob("inputTableName", "colFam1:*",
>>>> MapperClass.class,
>>>>           ImmutableBytesWritable.class, RowResult.class, c);
>>>>
>>>>     TableMapReduceUtil.initTableReduceJob("outputTableName",
>>>> ReducerClass.class, c );
>>>>
>>>> MapperClass:
>>>> =====================================================================
>>>>
>>>> @Override
>>>> public void map(
>>>> ImmutableBytesWritable key,
>>>> RowResult row,
>>>> OutputCollector<ImmutableBytesWritable, RowResult> collector,
>>>> Reporter reporter) throws IOException {
>>>>
>>>>
>>>> reporter.incrCounter(Counters.ROWS, 1);
>>>> collector.collect(key, row);
>>>> }
>>>>
>>>> ReducerClass:================================================================
>>>>
>>>> @Override
>>>> public void reduce(ImmutableBytesWritable k,
>>>> Iterator<RowResult> v,
>>>> OutputCollector<ImmutableBytesWritable, BatchUpdate> c,
>>>> Reporter r) throws IOException {
>>>>
>>>> while (v.hasNext()){
>>>> BatchUpdate bu = new BatchUpdate(k.get());
>>>> while (v.hasNext()){
>>>> RowResult row = v.next();
>>>> bu.put(Bytes.toBytes("count:rowToCountName"),
>>>> Bytes.toBytes(row.size()));
>>>> }
>>>> c.collect(k, bu);
>>>> }
>>>>                             }
>>>>
>>>> ========================================================================
>>>>
>>>> It runs the map/reduce, but I get nothing in my output table.
>>>>
>>>> Thanks.
>>>>
>>>> llpind
>>>> -- 
>>>> View this message in context:
>>>> http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p23952252.html
>>>> Sent from the HBase User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>> -- 
>> View this message in context: 
>> http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p23954242.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>> 
> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p23966757.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message