accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Newton (JIRA)" <>
Subject [jira] [Commented] (ACCUMULO-1946) Include dfs.datanode.synconclose in hdfs configuration documentation
Date Mon, 02 Dec 2013 19:40:35 GMT


Eric Newton commented on ACCUMULO-1946:

Let's keep the two issues separate:

1. sync the WAL

Writes to the WAL are sent out to the data nodes, and they (in 2.0) attempt to push it to
persistent media. This happens for every group commit, but each group can be sync'd in parallel.
 By increasing {{tserver.mutation.queue.max}} we can get good performance at the cost of ~50ms
of latency.

If you really don't care about persistence, you can just turn off the WAL for your table.

2. sync-on-close

I'm sure there's some penalty for sync-on-close, but I'm less concerned because we close files
a lot less frequently than we flush the WALog.

As long as these are global settings, we need them set because the METADATA table (and ROOT
table) need these  to keep everything working in the face of an HDFS restart or power loss
event.  Eventually, we may want to have these as table settings, or as mentioned in HBASE-5954,
with each update.

> Include dfs.datanode.synconclose in hdfs configuration documentation
> --------------------------------------------------------------------
>                 Key: ACCUMULO-1946
>                 URL:
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: docs
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.5.1, 1.6.0
> We should be including some writeup about dfs.datanode.synconclose in our documentation
surrounding the HDFS configuration as it better ensures that data is lost in the face of hard
shutdown (power loss) of the datanode process.

This message was sent by Atlassian JIRA

View raw message