hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ryan rawson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-3514) Speedup HFile.Writer append
Date Mon, 28 Feb 2011 22:08:36 GMT

    [ https://issues.apache.org/jira/browse/HBASE-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13000559#comment-13000559
] 

ryan rawson commented on HBASE-3514:
------------------------------------

a few comments here:

- why do you use a 256MB buffer in the "direct write" path?  That is a pretty damn big buffer,
the default buffer is 32k.  I'm not sure we should have such a huge buffer (and ram sink).
- finishBlock() timings for writes dont measure anything useful anymore, the long now = ...
is too low, it needs to go up higher.

I'm poking at the unit tests, I'm still not sure why we got that negative number, no unit
tests sets the block size to 1GB as far as i know!

> Speedup HFile.Writer append
> ---------------------------
>
>                 Key: HBASE-3514
>                 URL: https://issues.apache.org/jira/browse/HBASE-3514
>             Project: HBase
>          Issue Type: Improvement
>          Components: io
>    Affects Versions: 0.90.0
>            Reporter: Matteo Bertozzi
>            Priority: Minor
>         Attachments: HBASE-3514-append-0.90-v2.patch, HBASE-3514-append-0.90-v3.patch,
HBASE-3514-append-0.90.patch, HBASE-3514-append-trunk-v2.patch, HBASE-3514-append-trunk-v3.patch,
HBASE-3514-append.patch, HBASE-3514-metaBlock-bsearch.patch
>
>
> Remove double writes when block cache is specified, by using, only, the ByteArrayDataStream.
> baos is flushed with the compress stream on finishBlock.
> On my machines HFilePerformanceEvaluation SequentialWriteBenchmark passes from 4000ms
to 2500ms.
> Running SequentialWriteBenchmark for 1000000 rows took 4247ms.
> Running SequentialWriteBenchmark for 1000000 rows took 4512ms.
> Running SequentialWriteBenchmark for 1000000 rows took 4498ms.
> Running SequentialWriteBenchmark for 1000000 rows took 2697ms.
> Running SequentialWriteBenchmark for 1000000 rows took 2770ms.
> Running SequentialWriteBenchmark for 1000000 rows took 2721ms.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message