reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dhruv Mahajan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (REEF-1081) Improved Logging in HDFSCommandRunner
Date Tue, 15 Dec 2015 08:05:46 GMT

     [ https://issues.apache.org/jira/browse/REEF-1081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dhruv Mahajan updated REEF-1081:
--------------------------------
    Description: 
Currently in {{HdfsCommandRunner}}, the function {{LogCommandOutput(CommandResult result)}}
logs the output generated when the command is executed in {{internal CommandResult Run(string
hdfsCommandLineArguments)}}. However, in the current code we just check whether result is
null or not and then just terminate if result exists. However, we never log or check the exit
variable. If it is zero we should declare operation to be success otherwise keep trying. Infact,
result will never be null. Also after maximum retries, the thrown exception does not show
result logs as part of exception which contain vital information for debugging.  The code
is pasted below for reference also. 

{code}
  for (var attemptNumber = 0; attemptNumber < _numberOfRetries; ++attemptNumber)
            {
                var processName = String.Format("HDFS_Attempt_{0}_of_{1}", attemptNumber,
_numberOfRetries);
                var result = RunAttempt(processStartInfo, _timeOutInMilliSeconds, processName);
                if (null != result)
                {
                    LogCommandOutput(result);
                    return result;
                }
            }

            // If we reached here, we ran out of retries.
            throw new Exception(
                string.Format("HDFS Cmd {0} {1} could not be executed in the specified timeout
& retry settings",
                    _hdfsCommandPath, hdfsCommandLineArguments));
{code}

The main aim of JIRA is to introduce more elaborate logging and checking in the {{Run}} function.
Specifically, we would like to check the exit message and then write appropriate messages.

  was:
Currently in {{HdfsCommandRunner}}, the function {{ LogCommandOutput(CommandResult result)
}} logs the output generated when the command is executed in {{internal CommandResult Run(string
hdfsCommandLineArguments)}}. However, in the current code we just check whether result is
null or not and then just terminate if result exists. However, we never log or check the exit
variable. If it is zero we should declare operation to be success otherwise keep trying. Infact,
result will never be null. Also after maximum retries, the thrown exception does not show
result logs as part of exception which contain vital information for debugging.  The code
is pasted below for reference also. 

{code}
  for (var attemptNumber = 0; attemptNumber < _numberOfRetries; ++attemptNumber)
            {
                var processName = String.Format("HDFS_Attempt_{0}_of_{1}", attemptNumber,
_numberOfRetries);
                var result = RunAttempt(processStartInfo, _timeOutInMilliSeconds, processName);
                if (null != result)
                {
                    LogCommandOutput(result);
                    return result;
                }
            }

            // If we reached here, we ran out of retries.
            throw new Exception(
                string.Format("HDFS Cmd {0} {1} could not be executed in the specified timeout
& retry settings",
                    _hdfsCommandPath, hdfsCommandLineArguments));
{code}

The main aim of JIRA is to introduce more elaborate logging and checking in the {{Run}} function.
Specifically, we would like to check the exit message and then write appropriate messages.


> Improved Logging in HDFSCommandRunner
> -------------------------------------
>
>                 Key: REEF-1081
>                 URL: https://issues.apache.org/jira/browse/REEF-1081
>             Project: REEF
>          Issue Type: Improvement
>          Components: REEF.NET IO
>    Affects Versions: 0.14
>         Environment: C#
>            Reporter: Dhruv Mahajan
>            Assignee: Dhruv Mahajan
>             Fix For: 0.14
>
>
> Currently in {{HdfsCommandRunner}}, the function {{LogCommandOutput(CommandResult result)}}
logs the output generated when the command is executed in {{internal CommandResult Run(string
hdfsCommandLineArguments)}}. However, in the current code we just check whether result is
null or not and then just terminate if result exists. However, we never log or check the exit
variable. If it is zero we should declare operation to be success otherwise keep trying. Infact,
result will never be null. Also after maximum retries, the thrown exception does not show
result logs as part of exception which contain vital information for debugging.  The code
is pasted below for reference also. 
> {code}
>   for (var attemptNumber = 0; attemptNumber < _numberOfRetries; ++attemptNumber)
>             {
>                 var processName = String.Format("HDFS_Attempt_{0}_of_{1}", attemptNumber,
_numberOfRetries);
>                 var result = RunAttempt(processStartInfo, _timeOutInMilliSeconds, processName);
>                 if (null != result)
>                 {
>                     LogCommandOutput(result);
>                     return result;
>                 }
>             }
>             // If we reached here, we ran out of retries.
>             throw new Exception(
>                 string.Format("HDFS Cmd {0} {1} could not be executed in the specified
timeout & retry settings",
>                     _hdfsCommandPath, hdfsCommandLineArguments));
> {code}
> The main aim of JIRA is to introduce more elaborate logging and checking in the {{Run}}
function. Specifically, we would like to check the exit message and then write appropriate
messages.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message