accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-1637) Update HDFS append/sync precondition check for Hadoop 1.2
Date Wed, 16 Oct 2013 04:16:48 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-1637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796401#comment-13796401
] 

Josh Elser commented on ACCUMULO-1637:
--------------------------------------

[~jvines@gmail.com] If you have specifics, that'd be awesome.

[~ecn] Do you know of a means to reliably check that?

Alright, I think I get it now (someone correct me if I'm wrong). Brain-dumping before I lose
it.

0.20.205.0 and 1.0.x need the parameter 'dfs.support.append' as a means to make sure *sync*
actually works (misnomer on the configuration parameter). 1.1.x (and thus 1.2.x as well) remove
this misnomer and introduced 'dfs.durable.sync' as a means for users to turn *off* sync when
they don't want it (irrelevant for us). If you have dfs.support.append defined in 1.1.. Hadoop2
variants (0.23 and 2.x) all support append properly and sync implicitly.

In other words: 0.20.205.0 and 1.0.x *must* have the 'dfs.support.append' parameter provided
to ensure sync occurs when we call it. >=1.1.0 and >=0.23 must *not* have dfs.durable.sync=false
configured (although, realistically, we probably don't ever want to see that).

Personally, if someone is still running 0.20.205.0 or 1.0.x, I'm tempted to just say "shame
on you" through documentation rather than trying to make a runtime check. We could reflect
on DFSConfigKeys (handling the absence of DFS_SUPPORT_APPEND_DEFAULT for the 1.1.x line) and
then check for the presence of dfs.support.append being true in the Hadoop conf when DFS_SUPPORT_APPEND_DEFAULT
is false. We should probably always check that dfs.durable.sync is absent or true when configured.
Thoughts?

> Update HDFS append/sync precondition check for Hadoop 1.2
> ---------------------------------------------------------
>
>                 Key: ACCUMULO-1637
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1637
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.5.0
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Critical
>             Fix For: 1.5.1, 1.6.0
>
>
> Apache Hadoop 1.2.0 ships with the durable sync enabled by default and the support append
option marked as obsolete. Because of this, the check inside of TabletServer, meant to ensure
that the HDFS WAL can function properly, incorrectly fails as it doesn't know that dfs.durable.sync
is on by default.
> This can be worked around by specifying the old durable sync property in hdfs-site.xml:
> {noformat}
> <property>
>   <name>dfs.durable.sync</name>
>   <value>true</value>
> </property>
> {noformat}
> I'm not sure how to best way to address the differences between the newer and older versions
of Hadoop and their differing append support.
> Thanks to Carlos Mundi for pointing this out on user@a.a.o
> Using this table to track the presence of these variables and their default from hdfs/o/a/h/h/DFSConfigKeys
and from the codebase when there is no config parameter for it.
> ||Version||DFSConfigKeys.DFS_SUPPORT_APPEND_KEY||DFSConfigKeys.DFS_SUPPORT_APPEND_DEFAULT||"dfs.durable.sync"||Specific
Configuration Required||
> |0.20.205.0|defined|false|not present|yes|
> |0.23.x|defined|true|not present|no|
> |1.0.x|defined|false|not present|yes|
> |1.1.X|not present|absent|implicit "true"|no|
> |2.0.x|defined|true|not present|no|
> |2.1.x|defined|true|not present|no|
> |2.2.0|defined|true|not present|no|



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message