Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 05DD9613C for ; Mon, 6 Jun 2011 00:17:41 +0000 (UTC) Received: (qmail 42218 invoked by uid 500); 6 Jun 2011 00:17:38 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 42178 invoked by uid 500); 6 Jun 2011 00:17:38 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 42101 invoked by uid 99); 6 Jun 2011 00:17:37 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:17:37 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [128.135.12.77] (HELO smtp01.uchicago.edu) (128.135.12.77) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jun 2011 00:17:30 +0000 Received: from m4500-03.uchicago.edu (md00.uchicago.edu [128.135.249.212]) by smtp01.uchicago.edu (8.13.8/8.13.8) with ESMTP id p560H9OW020710 for ; Sun, 5 Jun 2011 19:17:09 -0500 Received: (from mstore03.uchicago.edu [128.135.249.212]) by mstore03.uchicago.edu (MOS 4.1.9-GA) with HTTP/1.1 id AQQ94822 (AUTH shiyu@uchicago.edu); Sun, 5 Jun 2011 19:17:09 -0500 (CDT) From: Shi Yu Subject: Re: Verbose screen logging on hadoop-0.20.203.0 To: common-user@hadoop.apache.org X-Mailer: Mirapoint Webmail Direct 4.1.9-GA MIME-Version: 1.0 In-Reply-To: References: <20110605120414.AQQ73113@mstore03.uchicago.edu> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <20110605191709.AQQ94822@mstore03.uchicago.edu> Date: Sun, 5 Jun 2011 19:17:09 -0500 (CDT) I still didn't get it. To make sure I am not using any old version, I downloaded two versions 0.20.2 and 0.20.203.0 again and had a fresh install separately on two independent clusters. I tried with a very simple toy program. I didn't change anything in the API so it probably calls the old API. Now both them have massive log output on the screen (I didn't have this problem deploying 0.20.2 months ago unless I forgot some important steps)! The output is really distracting (I would not say it is useless) but there should be an easy option to turn it off without changing and recompiling the code, right? Is there any option to turn off in the command line? Another question is how to write logs in /log/userlogs/? Why in the new versions nothing is written there? I remember in old version it is automatically done by log4j-1.2.15.jar right? The output log is as follows: 11/06/05 19:08:59 INFO mapred.LocalJobRunner: hdfs://server1:54320/input/medline/medline10n0240.apd:0+204511 52 11/06/05 19:09:02 INFO mapred.LocalJobRunner: hdfs://server1:54320/input/medline/medline10n0240.apd:0+204511 52 11/06/05 19:09:05 INFO mapred.LocalJobRunner: hdfs://server1:54320/input/medline/medline10n0240.apd:0+204511 52 11/06/05 19:09:08 INFO mapred.LocalJobRunner: hdfs://server1:54320/input/medline/medline10n0240.apd:0+204511 52 11/06/05 19:09:08 INFO mapred.TaskRunner: Task:attempt_local_0001_m_000000_0 is done. And is in the process of commiting 11/06/05 19:09:08 INFO mapred.LocalJobRunner: hdfs://server1:54320/input/medline/medline10n0240.apd:0+204511 52 11/06/05 19:09:08 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done. 11/06/05 19:09:08 INFO mapred.LocalJobRunner: 11/06/05 19:09:08 INFO mapred.Merger: Merging 1 sorted segments 11/06/05 19:09:08 INFO mapred.Merger: Down to the last merge- pass, with 1 segments left of total size: 258896359 bytes 11/06/05 19:09:08 INFO mapred.LocalJobRunner: 11/06/05 19:09:15 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:15 INFO mapred.JobClient: map 100% reduce 74% 11/06/05 19:09:18 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:18 INFO mapred.JobClient: map 100% reduce 78% 11/06/05 19:09:21 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:21 INFO mapred.JobClient: map 100% reduce 82% 11/06/05 19:09:24 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:24 INFO mapred.JobClient: map 100% reduce 86% 11/06/05 19:09:27 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:27 INFO mapred.JobClient: map 100% reduce 90% 11/06/05 19:09:30 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:30 INFO mapred.JobClient: map 100% reduce 94% 11/06/05 19:09:33 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:33 INFO mapred.TaskRunner: Task:attempt_local_0001_r_000000_0 is done. And is in the process of commiting 11/06/05 19:09:33 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:33 INFO mapred.TaskRunner: Task attempt_local_0001_r_000000_0 is allowed to commit now 11/06/05 19:09:33 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_000000_0' to hdfs://server1:54320/output/5grams 11/06/05 19:09:33 INFO mapred.LocalJobRunner: reduce > reduce 11/06/05 19:09:33 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_000000_0' done. 11/06/05 19:09:34 INFO mapred.JobClient: map 100% reduce 100% 11/06/05 19:09:34 INFO mapred.JobClient: Job complete: job_local_0001