hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer
Date Mon, 15 Dec 2014 20:26:17 GMT

    [ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247173#comment-14247173
] 

Sean Busbey commented on HBASE-5699:
------------------------------------

bq. I was also surprised about the boost, since my initial thinking was that we'd be constrained
in the single wal case by writing to one pipeline. However, once I think through things more
it makes a little more sense. For one thing we only use hflush and not hsync, so even for
network flushes we're still largely in memory. That also means those datanodes can keep handling
the write to disk as we get the next pipeline. On the far end of this chart for 128MB blocks,
that should be happening every ~1.5 seconds for the single wal. That allows us to keep more
than just 3 disks busy even in the single wal case.

One other note about the rate at which we are rolling pipelines already. This had me thinking
about the improvements in HBASE-10278. If we're rolling that often under load, I wonder if
we'd be better off just forcing a roll at whatever the "pipeline sync is slow" threshold is
rather than maintain the state to do a switch. A question better exploded on that ticket,
I suppose.

> Run with > 1 WAL in HRegionServer
> ---------------------------------
>
>                 Key: HBASE-5699
>                 URL: https://issues.apache.org/jira/browse/HBASE-5699
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance, wal
>            Reporter: binlijin
>            Assignee: Sean Busbey
>            Priority: Critical
>         Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff,
HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff,
HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff,
HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, hbase-5699_multiwal_400-threads_stats_sync_heavy.txt,
hbase-5699_total_throughput_sync_heavy.txt, results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2,
results-updated-hbase5699-wals-2.txt.bz2, results-updated-hbase5699-wals-4.txt.bz2, results-updated-hbase5699-wals-6.txt.bz2
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message