hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2415) Distribute TaskTracker userlogs onto multiple disks
Date Mon, 04 Apr 2011 16:20:06 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015483#comment-13015483

Owen O'Malley commented on MAPREDUCE-2415:

It looks like you'll have undefined references when you goto cleanup. In general you should
free the memory when you are done with it rather than moving the frees to a cleanup.

Part of getting this code (or any important C/C++ code) correct requires testing and running
in valgrind. Run both the unit tests and manual tests with valgrind, in particular looking
at the failure cases and fixing all of the undefined memory references and unfreed memory.

Use stdbool.h instead of defining your own.

You have tabs instead of all spaces.

I think it would be clearer to replace the code that traverses the link with code that does:
rm link
foreach dir: goodir
  rm dir

The reinitialization code needs to delete old user logs that are no longer pointed to by a
symlink. This is easier given the change above.

You should create a new method createLogDir in TaskController rather than testing the type
in an if statement.

You need to update the unit tests to test the new functionality.

> Distribute TaskTracker userlogs onto multiple disks
> ---------------------------------------------------
>                 Key: MAPREDUCE-2415
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2415
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: task-controller, tasktracker
>    Affects Versions:
>            Reporter: Bharath Mundlapudi
>            Assignee: Bharath Mundlapudi
>             Fix For:
>         Attachments: MAPREDUCE-2415-1.patch, TaskTracker Userlogs Design.pdf
> Currently, userlogs directory in TaskTracker is placed under hadoop.log.dir like <hadoop.log.dir>/userlogs.
I am proposing to spread these userlogs onto multiple configured mapred.local.dirs to strengthen
TaskTracker reliability w.r.t disk failures.  

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message