Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 568AE186B1 for ; Fri, 25 Sep 2015 09:32:12 +0000 (UTC) Received: (qmail 65788 invoked by uid 500); 25 Sep 2015 09:32:06 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 65658 invoked by uid 500); 25 Sep 2015 09:32:06 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 65648 invoked by uid 99); 25 Sep 2015 09:32:06 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Sep 2015 09:32:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id E5B411809C3 for ; Fri, 25 Sep 2015 09:32:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.65 X-Spam-Level: *** X-Spam-Status: No, score=3.65 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, KAM_INFOUSMEBIZ=0.75, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id kKO2fwXOGvPS for ; Fri, 25 Sep 2015 09:31:54 +0000 (UTC) Received: from mail-wi0-f194.google.com (mail-wi0-f194.google.com [209.85.212.194]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id 854F72306D for ; Fri, 25 Sep 2015 09:31:53 +0000 (UTC) Received: by wicmn1 with SMTP id mn1so1845719wic.3 for ; Fri, 25 Sep 2015 02:31:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:subject:references:message-id:date:user-agent:mime-version :in-reply-to:content-type; bh=UPmyeqU2sIxGjDecaIKEJRekrbKTu4ZL1DfiQKXfCM0=; b=ouZefsUq0DR9A/KV19uavJVgQC4WBARaUtXOJ5PbIYsGqetvgnMg5ETp1HpmwPGsk9 Dx0A1iI0yNEgJnCYpsz7s5AkLqNAU1Rh+KFHmbLgxXgpnq5mgqo6SgUtcDAx/LI5Z+xD rv+Bm6LnkI4cZ3XXIHs8d4IRQjtCXvrBq/TtsI+EKFiCEH7ER39xthPNzJ8ZY/W9Sy5r 2k2Kf4c82ak6CTpMMfcNm5/HeIVS9IkYcU5HMDTG0nh3J8sl4rb0gsdiO92m1rxJlqLG KsUfgboE+DdGNmDDKc+TweVaGuOm3IN68X+FiWd/5eafS7wE+NClg1pm/fiTfr7YRdqF H2LA== X-Received: by 10.180.8.36 with SMTP id o4mr2388686wia.82.1443173512164; Fri, 25 Sep 2015 02:31:52 -0700 (PDT) Received: from [10.10.5.115] ([194.117.22.6]) by smtp.gmail.com with ESMTPSA id mx19sm2342040wic.0.2015.09.25.02.31.50 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 25 Sep 2015 02:31:51 -0700 (PDT) From: xeonmailinglist To: user@hadoop.apache.org Subject: Re: println in MapReduce job References: <5604146D.5050606@gmail.com> <56042AB9.7010909@gmail.com> <56044541.5090201@gmail.com> Message-ID: <56051486.7080104@gmail.com> Date: Fri, 25 Sep 2015 10:31:50 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <56044541.5090201@gmail.com> Content-Type: multipart/alternative; boundary="------------070107070807090703040707" This is a multi-part message in MIME format. --------------070107070807090703040707 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit I have found the error. It was a bug in my code. On 09/24/2015 07:47 PM, xeonmailinglist wrote: > No, I am looking to the right place. Here is the output of a maptask. > I also thought that the print would come up, but it isn't showing. > That is why I am asking. > > > [1] Output from from one map task. > > ``` > > Log Type: stderr > > Log Upload Time: 24-Sep-2015 12:45:19 > > Log Length: 317 > > Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now. > It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. > > Log Type: stdout > > Log Upload Time: 24-Sep-2015 12:45:19 > > Log Length: 0 > > Log Type: syslog > > Log Upload Time: 24-Sep-2015 12:45:19 > > Log Length: 2604 > > 2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable > 2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties > 2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). > 2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started > 2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: > 2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082) > 2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. > 2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001 > 2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id > 2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] > 2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21 > 2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing > 2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now > 2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000 > 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done. > 2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... > 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped. > 2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete. > > ``` > > On 09/24/2015 06:42 PM, Peyman Mohajerian wrote: >> You log statement should show up in the container log for that Map >> class within the data node. I'm guessing you aren't looking in the >> right place. >> >> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist >> > wrote: >> >> Does anyone know this question about logging in MapReduce. I >> can't find an example or an explanation on how to print something >> inside user map and reduce functions. >> >> On 09/24/2015 04:19 PM, xeonmailinglist wrote: >>> >>> Hi, >>> >>> 1. >>> >>> I have this example of MapReduce [1], and I want to print >>> info in the stdout and in a log file. It seems that the logs >>> isn’t print anything. How can I make my class print these words? >>> >>> 2. >>> >>> I also have set in the |yarn-site.xml| to retain log. >>> Although the logs are retained in the |/app-logs| dir, the >>> |userlogs| dir is deleted at the end of the job execution. >>> How can I make MapReduce to not delete files in the >>> |userlogs| dir? >>> >>> I am using Yarn. >>> >>> Thanks, >>> >>> [1] Wordcount exampla with just the map part. >>> >>> |public class MyWordCount { public static class MyMap extends >>> Mapper { Log log = LogFactory.getLog(MyWordCount.class); private >>> final static IntWritable one = new IntWritable(1); private Text >>> word = new Text(); public void map(LongWritable key, Text value, >>> OutputCollector output, Reporter reporter) >>> throws IOException { StringTokenizer itr = new >>> StringTokenizer(value.toString()); System.out.println("HERRE"); >>> log.info ("HERRRRRE"); while >>> (itr.hasMoreTokens()) { word.set(itr.nextToken()); >>> output.collect(word, one); } } public void run(Context context) >>> throws IOException, InterruptedException { setup(context); try { >>> while (context.nextKeyValue()) { System.out.println("Key: " + >>> context.getCurrentKey() + " Value: " + >>> context.getCurrentValue()); map(context.getCurrentKey(), >>> context.getCurrentValue(), context); } } finally { >>> cleanup(context); } } public void cleanup(Mapper.Context >>> context) {} } | >>> >>> [2] yarn-site.xml >>> >>> | >>> yarn.log-aggregation-enable true >>> >>> yarn.nodemanager.log.retain-seconds >>> 900000 >>> yarn.nodemanager.remote-app-log-dir >>> /app-logs | >>> ​ >> >> > --------------070107070807090703040707 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit I have found the error. It was a bug in my code.

On 09/24/2015 07:47 PM, xeonmailinglist wrote:
No, I am looking to the right place. Here is the output of a maptask. I also thought that the print would come up, but it isn't showing. That is why I am asking.


[1] Output from from one map task.

```

Log Type: stderr

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 317 

Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

Log Type: stdout

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 0

Log Type: syslog

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 2604 

2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0 is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0 is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0' done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.

```

On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
You log statement should show up in the container log for that Map class within the data node. I'm guessing you aren't looking in the right place.

On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist <xeonmailinglist@gmail.com> wrote:
Does anyone know this question about logging in MapReduce. I can't find an example or an explanation on how to print something inside user map and reduce functions.

On 09/24/2015 04:19 PM, xeonmailinglist wrote:

Hi,

  1. I have this example of MapReduce [1], and I want to print info in the stdout and in a log file. It seems that the logs isn’t print anything. How can I make my class print these words?

  2. I also have set in the yarn-site.xml to retain log. Although the logs are retained in the /app-logs dir, the userlogs dir is deleted at the end of the job execution. How can I make MapReduce to not delete files in the userlogs dir?

I am using Yarn.

Thanks,

[1] Wordcount exampla with just the map part.


public class MyWordCount {
    public static class MyMap extends Mapper {
        Log log = LogFactory.getLog(MyWordCount.class);
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
            StringTokenizer itr = new StringTokenizer(value.toString());
            System.out.println("HERRE");
            log.info("HERRRRRE");
            while (itr.hasMoreTokens()) {
                word.set(itr.nextToken());
                output.collect(word, one);
            }
        }

        public void run(Context context) throws IOException, InterruptedException {
            setup(context);
            try {
                while (context.nextKeyValue()) {
                    System.out.println("Key: " + context.getCurrentKey() + " Value: " + context.getCurrentValue());
                    map(context.getCurrentKey(), context.getCurrentValue(), context);
                }
            } finally {
                cleanup(context);
            }
        }

        public void cleanup(Mapper.Context context) {}
    }

[2] yarn-site.xml

        <!-- job history -->
        <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property>
        <property> <name>yarn.nodemanager.log.retain-seconds</name> <value>900000</value> </property>
        <property> <name>yarn.nodemanager.remote-app-log-dir</name> <value>/app-logs</value> </property>
​




--------------070107070807090703040707--