hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jay vyas (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5902) JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up jobs with % characters in the name.
Date Tue, 27 May 2014 12:53:02 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

jay vyas updated MAPREDUCE-5902:
--------------------------------

    Description: 
1) JobHistoryServer sometimes skips over certain history files, and ignores serving them as
completed.

2) In addition to skipping these files, the JobHistoryServer doesnt effectively log which
files are being skipped , and why.  

So In addition to determining why certain types of files are skipped (file name length doesnt
appear to be the reason, rather, it appears to be that % characters throw the JobHistoryServer
filter off), we should log completed .jhist  files which  are available in the mr-history/tmp
directory, yet they are skipped for some reason. 

*Regarding the actual bug : Skipping completed jhist files* 

We will need an author of the JobHistoryServer, I think, to chime in on what types of paths
for jobs are actually valid.  It appears that at least some characters, if in a job name,
will make the jobhistoryserver skip recognition of a completed jhist file.

*Regarding logging*
It would be extremely useful , then, to have a couple of gaurded logs at this level of the
code, so that we can see, in the log folders, why files are being filtered out  , i.e. it
is due to filterint or visibility.

{noformat}

  private static List<FileStatus> scanDirectory(Path path, FileContext fc,
      PathFilter pathFilter) throws IOException {
    path = fc.makeQualified(path);
    List<FileStatus> jhStatusList = new ArrayList<FileStatus>();
    RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path);
    while (fileStatusIter.hasNext()) {
      FileStatus fileStatus = fileStatusIter.next();
      Path filePath = fileStatus.getPath();
      if (fileStatus.isFile() && pathFilter.accept(filePath)) {
        jhStatusList.add(fileStatus);
      }
    }
    return jhStatusList;
  }

{noformat}

*Reproducing* 

I was able to reproduce this bug by writing a custom mapreduce job with a job name, which
contained % characters.  I have also seen this with a version of the Mahout ParallelALSFactorizationJob,
which includes "-" characters in its name, which wind up getting replaced by "%2D" later on
at some stage in the job pipeline.


  was:
1) JobHistoryServer sometimes skips over certain history files, and ignores serving them as
completed.

2) In addition to skipping these files, the JobHistoryServer doesnt effectively log which
files are being skipped , and why.  

So In addition to determining why certain types of files are skipped (file name length doesnt
appear to be the reason, rather, it appears to be that % characters throw the JobHistoryServer
filter off), we should log completed .jhist  files which  are available in the mr-history/tmp
directory, yet they are skipped for some reason. 

** Regarding the actual bug : Skipping completed jhist files ** 

We will need an author of the JobHistoryServer, I think, to chime in on what types of paths
for jobs are actually valid.  It appears that at least some characters, if in a job name,
will make the jobhistoryserver skip recognition of a completed jhist file.

** Regarding logging **
It would be extremely useful , then, to have a couple of gaurded logs at this level of the
code, so that we can see, in the log folders, why files are being filtered out  , i.e. it
is due to filterint or visibility.

{noformat}

  private static List<FileStatus> scanDirectory(Path path, FileContext fc,
      PathFilter pathFilter) throws IOException {
    path = fc.makeQualified(path);
    List<FileStatus> jhStatusList = new ArrayList<FileStatus>();
    RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path);
    while (fileStatusIter.hasNext()) {
      FileStatus fileStatus = fileStatusIter.next();
      Path filePath = fileStatus.getPath();
      if (fileStatus.isFile() && pathFilter.accept(filePath)) {
        jhStatusList.add(fileStatus);
      }
    }
    return jhStatusList;
  }

{noformat}

** Reproducing ** 

I was able to reproduce this bug by writing a custom mapreduce job with a job name, which
contained % characters.  I have also seen this with a version of the Mahout ParallelALSFactorizationJob,
which includes "-" characters in its name, which wind up getting replaced by "%2D" later on
at some stage in the job pipeline.



> JobHistoryServer (HistoryFileManager) needs more debug logs, fails to pick up jobs with
% characters in the name.
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5902
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5902
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobhistoryserver
>            Reporter: jay vyas
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> 1) JobHistoryServer sometimes skips over certain history files, and ignores serving them
as completed.
> 2) In addition to skipping these files, the JobHistoryServer doesnt effectively log which
files are being skipped , and why.  
> So In addition to determining why certain types of files are skipped (file name length
doesnt appear to be the reason, rather, it appears to be that % characters throw the JobHistoryServer
filter off), we should log completed .jhist  files which  are available in the mr-history/tmp
directory, yet they are skipped for some reason. 
> *Regarding the actual bug : Skipping completed jhist files* 
> We will need an author of the JobHistoryServer, I think, to chime in on what types of
paths for jobs are actually valid.  It appears that at least some characters, if in a job
name, will make the jobhistoryserver skip recognition of a completed jhist file.
> *Regarding logging*
> It would be extremely useful , then, to have a couple of gaurded logs at this level of
the code, so that we can see, in the log folders, why files are being filtered out  , i.e.
it is due to filterint or visibility.
> {noformat}
>   private static List<FileStatus> scanDirectory(Path path, FileContext fc,
>       PathFilter pathFilter) throws IOException {
>     path = fc.makeQualified(path);
>     List<FileStatus> jhStatusList = new ArrayList<FileStatus>();
>     RemoteIterator<FileStatus> fileStatusIter = fc.listStatus(path);
>     while (fileStatusIter.hasNext()) {
>       FileStatus fileStatus = fileStatusIter.next();
>       Path filePath = fileStatus.getPath();
>       if (fileStatus.isFile() && pathFilter.accept(filePath)) {
>         jhStatusList.add(fileStatus);
>       }
>     }
>     return jhStatusList;
>   }
> {noformat}
> *Reproducing* 
> I was able to reproduce this bug by writing a custom mapreduce job with a job name, which
contained % characters.  I have also seen this with a version of the Mahout ParallelALSFactorizationJob,
which includes "-" characters in its name, which wind up getting replaced by "%2D" later on
at some stage in the job pipeline.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message