hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-15509) Avoid copy of block data in HFileBlock$Writer.finishBlock
Date Wed, 30 Mar 2016 16:07:25 GMT

    [ https://issues.apache.org/jira/browse/HBASE-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218218#comment-15218218

stack commented on HBASE-15509:

What is the mysterious 'len'?

84	   * @param len - length

HFileContext is meant to be immutable. You add methods that do get/set.

We add new configs for that non-existent 'expert hbase user' to twiddle: "hbase.hfile.write.buffer-reuse.max.size"
The on/off switch is fine but this new 'sizing' with no means of measuring if an arbitrary
setting is improving things or making them worse.

What is TLS in writeBufferTLS? Transport Layer Security ?

What is expected lifecycle of the thread local byte buffer?  If "Thread local buffer which
is used to compress and encode file block before writing it out to output.", then why are
we keeping ti around?

You introduce white space #1027 and unorthodox bracketing of methods -- see canReuseBufferForCurrentBlock

In other than a synthetic loading, the below looks like it could make things worse. We will
have churn as size fluctuates and we have the added cost of get/set a thread local:

1016	      if (canReuseBufferForCurrentBlock()) {
1017	        byte[] currentBuffer = writeBufferTLS.get();
1018	        if (currentBuffer == null || currentBuffer.length < baosInMemory.size()) {
1019	          currentBuffer = new byte[baosInMemory.size()];
1020	          writeBufferTLS.set(currentBuffer);
1021	        }

Seeing your reset of BBOS to length zero, and considering all the BB work that has been going
on in here abouts and elsewhere to do caching, wouldn't use of BBs be more natural? More natural
than adding methods to BBOS?

How effective is this caching? Block has to be same size or less than incoming one. How often
will this be the case? Any idea? How do we mitigate case where a Cell is 1G? All my threads
now have thread locals of 1G though I've gone back to serving Cells of 16k?

As per [~ram_krish], any perf numbers? Any GC graphs? This on a laptop with a fixed loading?

> Avoid copy of block data in HFileBlock$Writer.finishBlock
> ---------------------------------------------------------
>                 Key: HBASE-15509
>                 URL: https://issues.apache.org/jira/browse/HBASE-15509
>             Project: HBase
>          Issue Type: Sub-task
>          Components: HFile
>            Reporter: Vladimir Rodionov
>            Assignee: Vladimir Rodionov
>         Attachments: HBASE-15509-v1.patch, HBASE-15509-v2.patch
> This calls ByteArrayOutputStream.toByteArray() which creates a copy of data block. 

This message was sent by Atlassian JIRA

View raw message