hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-1716) Truncate logs of finished tasks to prevent node thrash due to excessive logging
Date Tue, 01 Feb 2011 01:56:29 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated MAPREDUCE-1716:

    Attachment: mapreduce-1716-testcase-race.txt

This test case has a bit of a race, since it assumes that the logs will be truncated immediately
upon job completion.

This isn't the case, since the logs aren't added to the truncation manager until the JVM finished,
which can be several seconds after the last task finishes (eg when sleepTimeBeforeSigKill
is 5 seconds). So, I found that this test was flaky as is.

You can show this by adding a 10 second sleep before truncating the logs, for example.

This delta patch has the tests loop for 20 seconds while checking the logs for truncation.
It only fails if the logs aren't truncated after 20 seconds.

> Truncate logs of finished tasks to prevent node thrash due to excessive logging
> -------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1716
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1716
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: task, tasktracker
>    Affects Versions: 0.20.3, 0.21.0, 0.21.1, 0.22.0
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>            Priority: Blocker
>             Fix For: 0.22.0
>         Attachments: MAPREDUCE-1100-20091216.2.txt, mapreduce-1716-testcase-race.txt,
patch-1100-fix-ydist.2.txt, patch-log-truncation-bugs-20100514.txt

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message