hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lei (Eddy) Xu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-8582) Reduce failure messages when running datanode reconfiguration
Date Tue, 16 Jun 2015 22:05:00 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lei (Eddy) Xu updated HDFS-8582:
--------------------------------
    Attachment: HDFS-8582.003.patch

Thanks a lot for the reviews, [~cmccabe]. I made the following changes based on your comments:

* {{listAllowed}} to {{properties}} subcommand.
* {{GetReconfigurationAllowedPropertiesResponse}} to {{ListReconfigurablePropertiesResponse}}.
 The reason that I did not change it to {{GetReconfigurablePropertiesResponse}} is that {{ReconfiguableBase}}
has a {{#getReconfigurableProperties()}} function, thus I could not use the same name in {{ClientProtocol}}.
Naming the protobuf to {{List...}} makes it consistent with {{ClientDatanodeProtocol}}.


bq. So it should use the correct defaults for the datanode. 

Currently, the following properties that are different
{noformat}
rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB
rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB
mapreduce.client.genericoptionsparser.used
dfs.datanode.startup
{noformat}

{{rpc.engine.*}} are set in {{DataNode#initIpcServer}}, {{dfs.datanode.setup}} is set in {{DataNode#parseArguments()}}
which is in {{main()}}, and {{mapreduce.client.genericoptionsparser.used}} is set in {{GenericOptionsParser}}.
 It might not clean to simply set default values of these properties in {{DataNode#getNewConf()}},
because the further code changes might break the consistency between them. So for now, I let
{{ReconfigureBase}} ignore non-reconfigurable properties and put them in log.  User should
use {{-reconfig properties}} to check whether a property is supported first. 

[~cmccabe] What would you think? 




> Reduce failure messages when running datanode reconfiguration
> -------------------------------------------------------------
>
>                 Key: HDFS-8582
>                 URL: https://issues.apache.org/jira/browse/HDFS-8582
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: HDFS
>    Affects Versions: 2.7.0
>            Reporter: Lei (Eddy) Xu
>            Assignee: Lei (Eddy) Xu
>            Priority: Minor
>         Attachments: HDFS-8582.000.patch, HDFS-8582.001.patch, HDFS-8582.002.patch, HDFS-8582.003.patch
>
>
> When running a DN reconfig to hotswap some drives, it spits out this output:
> {noformat}
> $ hdfs dfsadmin -reconfig datanode localhost:9023 status
> 15/06/09 14:58:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> Reconfiguring status for DataNode[localhost:9023]: started at Tue Jun 09 14:57:37 PDT
2015 and finished at Tue Jun 09 14:57:56 PDT 2015.
> FAILED: Change property rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
>         From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
>         To: ""
>         Error: Property rpc.engine.org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolPB
is not reconfigurable.
> FAILED: Change property mapreduce.client.genericoptionsparser.used
>         From: "true"
>         To: ""
>         Error: Property mapreduce.client.genericoptionsparser.used is not reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB
>         From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
>         To: ""
>         Error: Property rpc.engine.org.apache.hadoop.ipc.ProtocolMetaInfoPB is not reconfigurable.
> SUCCESS: Change property dfs.datanode.data.dir
>         From: "file:///data/1/user/dfs"
>         To: "file:///data/1/user/dfs,file:///data/2/user/dfs"
> FAILED: Change property dfs.datanode.startup
>         From: "REGULAR"
>         To: ""
>         Error: Property dfs.datanode.startup is not reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
>         From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
>         To: ""
>         Error: Property rpc.engine.org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolPB
is not reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB
>         From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
>         To: ""
>         Error: Property rpc.engine.org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolPB
is not reconfigurable.
> FAILED: Change property rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB
>         From: "org.apache.hadoop.ipc.ProtobufRpcEngine"
>         To: ""
>         Error: Property rpc.engine.org.apache.hadoop.tracing.TraceAdminProtocolPB is
not reconfigurable.
> {noformat}
> These failed messages are spurious and should not be shown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message