hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1553) Extensive logging of C++ application can slow down task by an order of magnitude
Date Mon, 06 Aug 2007 05:19:59 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517824
] 

Arun C Murthy commented on HADOOP-1553:
---------------------------------------

+1 (assuming the shell shenanigans work on Windows and cgywin+cygpath... I only tested the
patch on Linux. *smile*)

Personally I'm glad we only have 3 files: ${HADOOP_LOG_DIR}/userlogs/${taskid}/[stdout|stderr|syslog]
rather than subdirs... much easier to debug stuff.


Super-minor nit: Given that {{mapred.userlog.retain.hours}} defaults to 24hrs in hadoop-default.xml...

{noformat}
Index: src/java/org/apache/hadoop/mapred/TaskTracker.java
===================================================================
--- src/java/org/apache/hadoop/mapred/TaskTracker.java	(revision 561588)
+++ src/java/org/apache/hadoop/mapred/TaskTracker.java	(working copy)
@@ -1758,6 +1764,7 @@
             
       Task task = umbilical.getTask(taskid);
       JobConf job = new JobConf(task.getJobFile());
+      TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 12));
       task.setConf(job);
           
       defaultConf.addFinalResource(new Path(task.getJobFile()));
{noformat}

should be:
{noformat}
+      TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 24));
{noformat}

> Extensive logging of C++ application can slow down task by an order of magnitude
> --------------------------------------------------------------------------------
>
>                 Key: HADOOP-1553
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1553
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Christian Kunz
>            Assignee: Owen O'Malley
>            Priority: Blocker
>             Fix For: 0.14.0
>
>         Attachments: new-log-2.patch, new-log-3.patch
>
>
> We observed that extensive logging (due to some configuration mistake) of a c++ application
using the pipes interface can slow down the task by an order of magnitude. During that time
disk usage was not high, with no abnormal memory usage, and basically idle CPU.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message