hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7413) Convert WAL to pb
Date Mon, 22 Apr 2013 23:41:16 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638571#comment-13638571

stack commented on HBASE-7413:

bq. The usage of cells for both request and response is not very obvious

Why boss?  We'd send Cells but return molecules?

bq. It does seem to result in extra copy though, when you build the cellblock, which is something
we want to avoid for WAL path. Why is cell block ByteBuffer?

Because we build the cell block in memory?  Yeah.  I don't see any way around it.  You have
any suggestions? Hopefully w/ a good prefix encoding and maybe a compression (more copying?)
it'll be worth the composition in memory before we put it on the wire.

Cell block is a byte array.  Building it we need to use something that can grow, the BBOutputStream.

bq. It would require, aside from the task itself, adding a way to build cellblock directly
into output without copy (and getting count from cellscanner in advance?)

Not sure how to do the above.  Certain cellblock types could do this -- i.e. a write through
of Cells -- but cellblocks that do prefix encoding will want to store up and flush at well
defined junctures.  The count of Cells you should have before you start no?

I was hoping we could go from rpc straight into the WAL with the cell blocks but this is a
bit naive since we'll want cellblocks by region?

If not enough Cells, probably no need to make a CellBlock.  If values area large -- > a
couple of kilobytes -- probably worth compressing the WAL values.

On the rest of the writeup, thanks for doing it.

Here are some comments.

Can you call it WALKey instead of HLogKey?

bq. Replication scopes have been moved to HLogKey from WALEdit, so the latter only has KVs.

Excellent!  This I see as big improvement.

Can you talk about Cells instead of KVs?  If going to all this bother, might as well go the
whole hog and do Cell.

bq. VInt with length, followed by the backing byte array....

Doing the above, I'd argue, is only doing half of the job.  If we are going to convert WAL,
lets got off KV and its current serialization.  Write the codec used into the WAL metadata
and then write cellblocks (can do an implementation that writes straight to the WAL -- wrap
the WAL outputstream in a CellOutputStream?).

Writeup is great.

Let me look at the patch now.

> Convert WAL to pb
> -----------------
>                 Key: HBASE-7413
>                 URL: https://issues.apache.org/jira/browse/HBASE-7413
>             Project: HBase
>          Issue Type: Sub-task
>          Components: wal
>            Reporter: stack
>            Assignee: Sergey Shelukhin
>            Priority: Critical
>             Fix For: 0.95.1
>         Attachments: HBASE-7413-v0.patch, HBASE-7413-v1.patch, HBASE-7413-v2.patch
> From HBASE-7201

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message