hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
Date Mon, 23 Apr 2012 14:47:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259638#comment-13259638
] 

Robert Joseph Evans commented on MAPREDUCE-4169:
------------------------------------------------

I also like the patch for the most part, but while reviewing it I saw one disturbing preexisting
problem.  

The LogValue constructor is called from AppLogAggregatorImpl.uploadLogsFromContainer(), which
gets its value from LocalDirsHandlerService.getLogDirs() which calls DirectoryCollection.getGoodDirs(),
which returns an internal ArrayList.  When we sort this we are sorting an internal data structre
of DirectoryCollection, that appears to be something that can be called from multiple threads
and that it too modifies internally, possibly on a separate thread.

I think the correct thing to do is to modify DirectoryCollection to not return internal values,
as we could get concurrent modification exceptions as it is.  We either need to clone the
Lists before returning them, or we need to make them concurrent in some way.

I am fine if we do this on a different JIRA, because it is a preexisting problem.  It is just
that this JIRA will potentially exacerbate the problem.
                
> Container Logs appear in unsorted order
> ---------------------------------------
>
>                 Key: MAPREDUCE-4169
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.3, 2.0.0
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>            Priority: Minor
>         Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch
>
>
> container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory ui appear
in unsorted order where the order displayed is based on what file was created first. This
jira will have the results be displayed in a consistent order.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message