hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Shelukhin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7413) Convert WAL to pb
Date Tue, 16 Apr 2013 20:47:15 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633332#comment-13633332
] 

Sergey Shelukhin commented on HBASE-7413:
-----------------------------------------

The comments above add to almost a page ;)
bq. Do we have to make our own format? Can we not reuse?
We use protobufs. The only additions on top of that is: we don't want to put KVs in protobuf,
so we write them using writables as before; and we need magic at start to tell the new files
apart from old (old ones have different magic). What do you have in mind to reuse?
bq. Ideally, can take a protobuf multi message off rpc w/ cellBlocks as is and just put it
into the WAL w/o unmarshalling/marshalling making new types
WAL RPC message currently has byte buffer for KVs. That would result in extra copy of KVs...
I did lift most of the Admin.proto RPC stuff, with few changes.
bq. (cellBlocks have codecs now and they can be compressed – would be sweet if they retained
their compression/codec when in WAL – would mean though that the compression/codec used
would have to be associated w/ the WAL entry since it is currently only in the Connection
header for the particular client server-side)
Compression is retained.
bq. If we could do away with the HLog entry key, that'd be cool but it has info such as target
region, replication...
I actually moved replication scopes from WALEdit to HLogKey, so WALEdit is just KVs. I'd say
if we have some reasonable way to put KVs into PB w/o overhead there should just be one structure
because WALEdit and HLogKey are used together everywhere for practical purposes.
Another obstacle to uniting them is all the code that uses them of course :) So this should
be a separate JIRA, I did leave a TODO


                
> Convert WAL to pb
> -----------------
>
>                 Key: HBASE-7413
>                 URL: https://issues.apache.org/jira/browse/HBASE-7413
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>            Reporter: stack
>            Assignee: Sergey Shelukhin
>            Priority: Critical
>             Fix For: 0.95.1
>
>
> From HBASE-7201

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message