hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5166) MultiThreaded Table Mapper analogous to MultiThreaded Mapper in hadoop
Date Thu, 23 Feb 2012 06:15:50 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13214361#comment-13214361
] 

jiraposter@reviews.apache.org commented on HBASE-5166:
------------------------------------------------------



bq.  On 2012-02-23 04:32:03, Michael Stack wrote:
bq.  > This looks great.  Does it work?  Have you tried it?  +1 on commit if it works.
 Would be nice in things like PE putting up more load.
bq.  
bq.  Jai Singh wrote:
bq.      This works fine. I've tested it in the usecase  I mentioned on jira HBASE-5166.
bq.  
bq.  Michael Stack wrote:
bq.      So works nicely for your crawling then?  Mind writing a sweet release note for this?
 I'll go commit it.
bq.  
bq.  Michael Stack wrote:
bq.      Oh, mind uploading the final version of the patch to the issue itself then we can
run hadoopqa on the patch and make sure it plays well w/ rest of hbase (should be fine given
its standalone).  Thanks Jai.

Yes, It works great with web crawling scenario. 

"MultiThreadedTableMapper for [N/W] IO bound jobs"

Updated the patch on jira.

Thanks


- Jai


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3995/#review5302
-----------------------------------------------------------


On 2012-02-23 04:22:51, Jai Singh wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/3995/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-02-23 04:22:51)
bq.  
bq.  
bq.  Review request for hbase, Ted Yu and Michael Stack.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  There is no MultiThreadedTableMapper in hbase currently just like we have a MultiThreadedMapper
in Hadoop for IO Bound Jobs. 
bq.  UseCase, webcrawler: take input (urls) from a hbase table and put the content (urls,
content) back into hbase. 
bq.  Running these kind of hbase mapreduce job with normal table mapper is quite slow as we
are not utilizing CPU fully (N/W IO Bound).
bq.  
bq.  Moreover, I want to know whether It would be a good/bad idea to use HBase for these kind
of usecases ?.
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    /src/main/java/org/apache/hadoop/hbase/mapreduce/MultithreadedTableMapper.java PRE-CREATION

bq.    /src/test/java/org/apache/hadoop/hbase/mapreduce/TestMulitthreadedTableMapper.java
PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/3995/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Jai
bq.  
bq.


                
> MultiThreaded Table Mapper analogous to MultiThreaded Mapper in hadoop
> ----------------------------------------------------------------------
>
>                 Key: HBASE-5166
>                 URL: https://issues.apache.org/jira/browse/HBASE-5166
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Jai Kumar Singh
>            Priority: Minor
>              Labels: multithreaded, tablemapper
>         Attachments: 0001-Added-MultithreadedTableMapper-HBASE-5166.patch, 0003-Added-MultithreadedTableMapper-HBASE-5166.patch,
0005-HBASE-5166-Added-MultithreadedTableMapper.patch, 0006-HBASE-5166-Added-MultithreadedTableMapper.patch,
0008-HBASE-5166-Added-MultithreadedTableMapper.patch
>
>   Original Estimate: 0.5h
>  Remaining Estimate: 0.5h
>
> There is no MultiThreadedTableMapper in hbase currently just like we have a MultiThreadedMapper
in Hadoop for IO Bound Jobs. 
> UseCase, webcrawler: take input (urls) from a hbase table and put the content (urls,
content) back into hbase. 
> Running these kind of hbase mapreduce job with normal table mapper is quite slow as we
are not utilizing CPU fully (N/W IO Bound).
> Moreover, I want to know whether It would be a good/bad idea to use HBase for these kind
of usecases ?. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message