hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From llpind <sonny_h...@hotmail.com>
Subject Re: Help with Map/Reduce program
Date Fri, 12 Jun 2009 17:04:55 GMT

I believe my map is collecting per row correcly, but reduce doesn't seem to
be doing anything:

		private RowResult previousRow = null;  //keep previous row
		private int counter = 0;  //counter for adding up like TYPE|VALUE
		public void reduce(ImmutableBytesWritable k,
				Iterator<RowResult> v,
				OutputCollector<ImmutableBytesWritable, BatchUpdate> c,
				Reporter r) throws IOException {
			//keep counter for equal entities. row takes TYPE|VALUE|LINKID form
			while (v.hasNext()){
				RowResult currentRow = v.next();
				if (previousRow == null){
					previousRow = currentRow;
				if (extractEntity(currentRow).equals(extractEntity(previousRow))){
					//commit previous row size, set previous to current & reset counter
					BatchUpdate bu = new BatchUpdate(extractEntity(previousRow));
					bu.put("count:"+counter, String.valueOf(counter).getBytes());
					c.collect(new ImmutableBytesWritable(previousRow.getRow()), bu);
					previousRow = currentRow;
					counter = 0;

What am I doing wrong?

The extract is simiply getting the TYPE|VALUE only.  

what excatly do I have in the Iterator<RowResult>  at this point?  


llpind wrote:
> If i have a tall table, what is returned in the reduce?   I'm still
> confused as to how things map up.
> for example assume I have ELECTRONICS|TV|ID2343 as the row key. There are
> millions of these (ELECTRONICS|TV|ID234324, along with other products). 
> I'd like to count the total # of IDs for all TVs.  How do I do this with
> map/reduce?  I tried a few things, but not able to get it working.
> Ryan Rawson wrote:
>> Also remember you might be able to convert to a tall table. Row keys can
>> be
>> compound and you can do partial left matches on them. Eg:
>> Userid:timestamp:eventid
>> now you have a tall table. Do prefix matches on the userid you want and
>> you
>> get results in chronological order.
>> You can build equivalent indexes in hbase as in sql. You may find a
>> design
>> like this alieviates the need for extremely wide rows.
>> Good luck!
>> On Jun 11, 2009 11:44 AM, "Billy Pearson" <sales@pearsonwholesale.com>
>> wrote:
>> That might be a good idea but you might be able to redesign you layout of
>> the table
>> using a different key then the current one worth barnstorming.
>> Billy
>> "llpind" <sonny_heer@hotmail.com> wrote in message
>> news:23975432.post@talk.nabble.com...
>> Sorry I forgot to mention the overflow then overflows into new row keys
>> per
>> 10,000 column entries ...

View this message in context: http://www.nabble.com/Help-with-Map-Reduce-program-tp23952252p24002766.html
Sent from the HBase User mailing list archive at Nabble.com.

View raw message