Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Mon, 15 Dec 2014 20:26:17 +0000 (UTC)
From: "Sean Busbey (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12549194.1333390699000.26898.1418675177168@Atlassian.JIRA>
In-Reply-To: <JIRA.12549194.1333390699000@Atlassian.JIRA>
References: <JIRA.12549194.1333390699000@Atlassian.JIRA>
 <JIRA.12549194.1333390699454@arcas>
Subject: [jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247173#comment-14247173 ] 

Sean Busbey commented on HBASE-5699:
------------------------------------

bq. I was also surprised about the boost, since my initial thinking was that we'd be constrained in the single wal case by writing to one pipeline. However, once I think through things more it makes a little more sense. For one thing we only use hflush and not hsync, so even for network flushes we're still largely in memory. That also means those datanodes can keep handling the write to disk as we get the next pipeline. On the far end of this chart for 128MB blocks, that should be happening every ~1.5 seconds for the single wal. That allows us to keep more than just 3 disks busy even in the single wal case.

One other note about the rate at which we are rolling pipelines already. This had me thinking about the improvements in HBASE-10278. If we're rolling that often under load, I wonder if we'd be better off just forcing a roll at whatever the "pipeline sync is slow" threshold is rather than maintain the state to do a switch. A question better exploded on that ticket, I suppose.

> Run with > 1 WAL in HRegionServer
> ---------------------------------
>
>                 Key: HBASE-5699
>                 URL: https://issues.apache.org/jira/browse/HBASE-5699
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance, wal
>            Reporter: binlijin
>            Assignee: Sean Busbey
>            Priority: Critical
>         Attachments: HBASE-5699.3.patch.txt, HBASE-5699.4.patch.txt, HBASE-5699_#workers_vs_MiB_per_s_1x1col_512Bval_wal_count_1,2,4.tiff, HBASE-5699_write_iops_multiwal-1_1_to_200_threads.tiff, HBASE-5699_write_iops_multiwal-2_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-4_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_multiwal-6_10,50,120,190,260,330,400_threads.tiff, HBASE-5699_write_iops_upstream_1_to_200_threads.tiff, PerfHbase.txt, hbase-5699_multiwal_400-threads_stats_sync_heavy.txt, hbase-5699_total_throughput_sync_heavy.txt, results-hbase5699-upstream.txt.bz2, results-hbase5699-wals-1.txt.bz2, results-updated-hbase5699-wals-2.txt.bz2, results-updated-hbase5699-wals-4.txt.bz2, results-updated-hbase5699-wals-6.txt.bz2
>
>


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)