hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2748) Upload logs in the sub-folders under the local log dir when aggregating logs
Date Tue, 25 Nov 2014 21:00:13 GMT

    [ https://issues.apache.org/jira/browse/YARN-2748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225194#comment-14225194
] 

Varun Saxena commented on YARN-2748:
------------------------------------

[~zjshen], kindly review this initial patch.

Few points :
* Sub-folders may contain log files which have the same name as parent folder. 
*# We write file name while aggregating logs. To differentiate log files which may have same
file names(due to subfolders), I think we can write file path relative to container log directory
 instead. Your views on this. Do you envisage any problem elsewhere because of this ?
{code:title=AggregatedLogFormat.java|borderStyle=solid}
public void write(DataOutputStream out, Set<File> pendingUploadFiles)
        throws IOException {
      ...
      for (File logFile : fileList) {
        ...
        // Write the logFile Type
        out.writeUTF(logFile.getName());
        ....
      }
    }
{code}
*# In case of file meta data used for differentiating between log files which have already
been aggregated and which are yet to be, I think file modification time may avoid issues which
may arise due to clash of names. We can probably include subfolder name here as well. Your
views on this ?
{code:title=AggregatedLogFormat.java|borderStyle=solid}
private String getLogFileMetaData(File file) {
      return containerId.toString() + "_" + file.getName() + "_"
          + file.lastModified();
    }
{code}

* I have checked for subfolders in both getPendingLogFilesToUpload and write methods. Although
NM isnt calling the write method directly, this method still has public visibility. Hence,
a client application may call it.

> Upload logs in the sub-folders under the local log dir when aggregating logs
> ----------------------------------------------------------------------------
>
>                 Key: YARN-2748
>                 URL: https://issues.apache.org/jira/browse/YARN-2748
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation
>    Affects Versions: 2.6.0
>            Reporter: Zhijie Shen
>            Assignee: Varun Saxena
>         Attachments: YARN-2748.001.patch
>
>
> YARN-2734 has a temporal fix to skip sub folders to avoid exception. Ideally, if the
app is creating a sub folder and putting its rolling logs there, we need to upload these logs
as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message