chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-267) Collect shuffling information from userlogs/syslog
Date Thu, 28 May 2009 23:27:45 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714219#action_12714219
] 

Eric Yang commented on CHUKWA-267:
----------------------------------

We do not want synchronize to occur on every log statement, the performance will be severely
impacted, if we lock on every log statement.  Hence, we check for condition first, if there
is already one of thread pass through, then rest of the reads don't need to go through the
same subroutine.  There would be only one thread pass in this subroutine to creates the background
thread to register the file.  Second, we can't do this in the constructor because we do not
know the task attempt id until there is actually something to log, hence this was done at
subAppend level.

> Collect shuffling information from userlogs/syslog
> --------------------------------------------------
>
>                 Key: CHUKWA-267
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-267
>             Project: Hadoop Chukwa
>          Issue Type: New Feature
>          Components: data collection
>    Affects Versions: 0.2.0
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>             Fix For: 0.1.2, 0.2.0
>
>         Attachments: CHUKWA-267.patch
>
>
> Hadoop uses a customized version of log4j appender (org.apache.hadoop.mapred.TaskLogAppender)
for tasks.  Shuffling information is written to userlogs/syslog by TaskLogAppender.  For chukwa
to collect shuffling information, we need to write a new TaskLogAppender to do this.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message