Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of pavlikus@gmail.com designates
 66.249.90.181 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:content-type:content-transfer-encoding
         :mime-version:subject:date:x-mailer;
        b=AigzX6fRUgR9+Ft8A74jG29rGh09N9rToDbqS+CK3EWqZbutJVy3CI4O7LZWTvY7QF
         qJG5iuV3Dhiq8SVPaExcStgG+UmJ0DeN3b6JAyRGpdKeVGdNwg5CZe1RYZtVYxrmgaNz
         pd0ks5ayJxhyUgUfPSanuwqg4v0eksQsF9uzI=
Message-Id: <1B74AFC6-67EC-40D6-A8C5-69D1E5028FFE@gmail.com>
From: Pavel Lysov <pavlikus@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v926)
Subject: to mapreduce or not to mapreduce? 
Date: Tue, 29 Jul 2008 19:51:58 +0300

Hey all!

I'd like to ask you to take a look at the stuff I have and advice is  
it right direction to proceed it with map reduce approach?

There's MESSAGES table, each message has sender and recipient. It  
works nice so far and next I want to get the following info:

Total of messages user X has sent
Total of messages user X has received
Total of messages in the system

It would be USERS table, with USER_ID as row key and with 'messages'  
column family:
   messages:total_sent 345
   messages:total_received 543

Similar to the above, I'd create SYSTEM table with 'messages:total'  
column that will hold the total count of messages.

Next I think I should implement map reduce job that will update  
'messages:total_sent/total_received' for every user by adding one to  
output collector for given user id. Next, in reduce, I'll sum them up  
and update the user's row. Is it good idea to do like that? Could it  
cause any probs if more than two concurrent reduce jobs will try to  
update the same row?

Similar question for SYSTEM table, suppose there a bunch of reduce  
jobs that try to update messages:total column at the same time?

I think table locks would help there but it seems I am missing some  
basis understanding of how that all is supposed to work. Could you  
please advice?

I appreciate your help!
Pavel Lysov
pavlikus@gmail.com