hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kay Kay (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-2368) BulkPut - Writable class compatible with TableRecordWriter for bulk puts agnostic of region server mapping at Mapper/Combiner level
Date Thu, 25 Mar 2010 00:25:27 GMT

    [ https://issues.apache.org/jira/browse/HBASE-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849549#action_12849549
] 

Kay Kay commented on HBASE-2368:
--------------------------------

| could you explain to me how this might be different than having the TableOutputFormat assume/use
a write buffer? 



This is complementary to the same, except with autoFlush set to true / writeBuffer set to
false.   We have a Mapper job that was generating 1:many puts at the Mapper level. So instead
of writing individual puts to the stream, bulk add the puts and write to the stream once.


|  (though not to TOF).

Specifically this patch addresses the TableRecordWriter (of TOF) and as you had pointed ,
this is moot when we are using HTable directly , but complementary at the Mapper / Combiner
level , when we can consolidate puts and write them in bulk. 


> BulkPut - Writable class  compatible with TableRecordWriter for bulk puts agnostic of
region server mapping at Mapper/Combiner level
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-2368
>                 URL: https://issues.apache.org/jira/browse/HBASE-2368
>             Project: Hadoop HBase
>          Issue Type: Improvement
>          Components: client
>            Reporter: Kay Kay
>             Fix For: 0.21.0
>
>         Attachments: HBASE-2368.patch
>
>
> TableRecordWriter currently accepts only a put/delete as writables. Some mapper processes
might want to consolidate the 'put's and insert them in bulk. Useful in combiners / mappers
- to send across a bunch of puts from one stage to another , while maintaining a very similar
region-server-mapping agnostic api at respective levels. 
> New type - BulkPut ( Writable ) introduced that is just a consolidation of Puts.  Eventually
, the TableRecordWriter bulk inserts the puts together into the hbase eco-system. 
> Patch made against trunk only. But since , it does not break any backward compatibility
- it can be an useful addition to the branch as well. 
> Let me know your comments on the same. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message