hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhruba Borthakur <dhr...@gmail.com>
Subject Re: Multiple WALs
Date Sat, 01 Oct 2011 17:56:58 GMT
I have been experimenting with the WAL settings too. It is obvious that
turning off the wal makes ur transactions go faster, HDFS write/sync are not
yet very optimized for high throughput small writes.

However, irrespective of whether I have one wal or two, I have seeing the
same throughput. I have experimented with an HDFS setting that allows
writing/sync to multiple replicas in parallel, and that has increased
performance for my test workload, see
https://issues.apache.org/jira/browse/HDFS-1783.

About using one wal or two, it will be nice if we can separate out the wal
API elegantly and make it pluggable. In that case, we can experiment HBase
with multiple systems. Once we have it pluggable, we can make the habse wal
go to a separate HDFS (pure SSD based maybe?).

-dhruba


On Sat, Oct 1, 2011 at 8:09 AM, Akash Ashok <thehellmaker@gmail.com> wrote:

> Hey,
> I've see that setting writeToWAL(false) boosts up the writes like crazy. I
> was just thinking having MuiltipleWAL on HBase. I understand that this is a
> consideration in BigTable paper that a WAL per region is not used because
> it
> might result in a lot of disk seeks when there are large number of reasons.
> But how about having as many WALs as the number of HardDrives in the
> system.
> I see that the recommended configs for HBase are 4 - 12 hard drives per
> node. This might kick the writes up a notch.
>
> Would like to know the general opinion on this one?
>
> Cheers,
> Akash A
>



-- 
Connect to me at http://www.facebook.com/dhruba

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message