From hadoop-dev-return-16048-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org Mon Aug 06 05:20:26 2007 Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 17063 invoked from network); 6 Aug 2007 05:20:23 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Aug 2007 05:20:23 -0000 Received: (qmail 9512 invoked by uid 500); 6 Aug 2007 05:20:22 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 9474 invoked by uid 500); 6 Aug 2007 05:20:22 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 9446 invoked by uid 99); 6 Aug 2007 05:20:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 05 Aug 2007 22:20:22 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Aug 2007 05:20:11 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 551637141F4 for ; Sun, 5 Aug 2007 22:19:59 -0700 (PDT) Message-ID: <25060491.1186377599344.JavaMail.jira@brutus> Date: Sun, 5 Aug 2007 22:19:59 -0700 (PDT) From: "Arun C Murthy (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Commented: (HADOOP-1553) Extensive logging of C++ application can slow down task by an order of magnitude In-Reply-To: <17740825.1183175224446.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-1553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517824 ] Arun C Murthy commented on HADOOP-1553: --------------------------------------- +1 (assuming the shell shenanigans work on Windows and cgywin+cygpath... I only tested the patch on Linux. *smile*) Personally I'm glad we only have 3 files: ${HADOOP_LOG_DIR}/userlogs/${taskid}/[stdout|stderr|syslog] rather than subdirs... much easier to debug stuff. Super-minor nit: Given that {{mapred.userlog.retain.hours}} defaults to 24hrs in hadoop-default.xml... {noformat} Index: src/java/org/apache/hadoop/mapred/TaskTracker.java =================================================================== --- src/java/org/apache/hadoop/mapred/TaskTracker.java (revision 561588) +++ src/java/org/apache/hadoop/mapred/TaskTracker.java (working copy) @@ -1758,6 +1764,7 @@ Task task = umbilical.getTask(taskid); JobConf job = new JobConf(task.getJobFile()); + TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 12)); task.setConf(job); defaultConf.addFinalResource(new Path(task.getJobFile())); {noformat} should be: {noformat} + TaskLog.cleanup(job.getInt("mapred.userlog.retain.hours", 24)); {noformat} > Extensive logging of C++ application can slow down task by an order of magnitude > -------------------------------------------------------------------------------- > > Key: HADOOP-1553 > URL: https://issues.apache.org/jira/browse/HADOOP-1553 > Project: Hadoop > Issue Type: Bug > Components: mapred > Affects Versions: 0.13.0 > Reporter: Christian Kunz > Assignee: Owen O'Malley > Priority: Blocker > Fix For: 0.14.0 > > Attachments: new-log-2.patch, new-log-3.patch > > > We observed that extensive logging (due to some configuration mistake) of a c++ application using the pipes interface can slow down the task by an order of magnitude. During that time disk usage was not high, with no abnormal memory usage, and basically idle CPU. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.