accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1946) Include dfs.datanode.synconclose in hdfs configuration documentation
Date Mon, 02 Dec 2013 19:22:35 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836815#comment-13836815
] 

Keith Turner commented on ACCUMULO-1946:
----------------------------------------

I think the dfs sync option should be turned on and that Accumulo should be tuned to compensate
some.  I didn't mention tserver.mutation.queue.max that defaults to 256K, which results in
frequent syncs.  Making the default 1M would reduce the number of flushes 4x, setting it to
4M would decrease the frequency by 16x.   At 4M 100 concurrent writers could use 400M of memory
to buffer writes for the walog.  We need to change how this behaves.  Basically when you have
 a few concurrent writer you want bigger buffers per writer.  With lots of concurrent writers,
small buffers are ok and you make up for it w/ group commit.

> Include dfs.datanode.synconclose in hdfs configuration documentation
> --------------------------------------------------------------------
>
>                 Key: ACCUMULO-1946
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1946
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: docs
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.5.1, 1.6.0
>
>
> We should be including some writeup about dfs.datanode.synconclose in our documentation
surrounding the HDFS configuration as it better ensures that data is lost in the face of hard
shutdown (power loss) of the datanode process.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message