hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeonmailinglist <xeonmailingl...@gmail.com>
Subject Re: println in MapReduce job
Date Thu, 24 Sep 2015 18:47:29 GMT
No, I am looking to the right place. Here is the output of a maptask. I 
also thought that the print would come up, but it isn't showing. That is 
why I am asking.


[1] Output from from one map task.

```

Log Type: stderr

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 317

Java HotSpot(TM) Client VM warning: You have loaded library /home/xubuntu/Programs/hadoop-2.6.0/lib/native/libhadoop.so
which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link
it with '-z noexecstack'.

Log Type: stdout

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 0

Log Type: syslog

Log Upload Time: 24-Sep-2015 12:45:19

Log Length: 2604

2015-09-24 12:45:04,569 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where applicable
2015-09-24 12:45:05,139 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded
properties from hadoop-metrics2.properties
2015-09-24 12:45:05,412 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2015-09-24 12:45:05,413 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system started
2015-09-24 12:45:05,462 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens:
2015-09-24 12:45:05,463 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job,
Service: job_1443113036547_0001, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@1b5a082)
2015-09-24 12:45:05,847 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now.
2015-09-24 12:45:06,915 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir
for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1443113036547_0001
2015-09-24 12:45:07,604 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id
is deprecated. Instead, use dfs.metrics.session-id
2015-09-24 12:45:09,402 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree
: [ ]
2015-09-24 12:45:10,187 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/input1/b.txt:0+21
2015-09-24 12:45:10,812 INFO [main] org.apache.hadoop.mapred.Task: Task:attempt_1443113036547_0001_m_000000_0
is done. And is in the process of committing
2015-09-24 12:45:10,969 INFO [main] org.apache.hadoop.mapred.Task: Task attempt_1443113036547_0001_m_000000_0
is allowed to commit now
2015-09-24 12:45:10,993 INFO [main] org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter:
Saved output of task 'attempt_1443113036547_0001_m_000000_0' to hdfs://192.168.10.110:9000/output1-1442847968/_temporary/1/task_1443113036547_0001_m_000000
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.mapred.Task: Task 'attempt_1443113036547_0001_m_000000_0'
done.
2015-09-24 12:45:11,135 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping
MapTask metrics system...
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system stopped.
2015-09-24 12:45:11,136 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask
metrics system shutdown complete.


```

On 09/24/2015 06:42 PM, Peyman Mohajerian wrote:
> You log statement should show up in the container log for that Map 
> class within the data node. I'm guessing you aren't looking in the 
> right place.
>
> On Thu, Sep 24, 2015 at 9:54 AM, xeonmailinglist 
> <xeonmailinglist@gmail.com <mailto:xeonmailinglist@gmail.com>> wrote:
>
>     Does anyone know this question about logging in MapReduce. I can't
>     find an example or an explanation on how to print something inside
>     user map and reduce functions.
>
>     On 09/24/2015 04:19 PM, xeonmailinglist wrote:
>>
>>     Hi,
>>
>>     1.
>>
>>         I have this example of MapReduce [1], and I want to print
>>         info in the stdout and in a log file. It seems that the logs
>>         isn’t print anything. How can I make my class print these words?
>>
>>     2.
>>
>>         I also have set in the |yarn-site.xml| to retain log.
>>         Although the logs are retained in the |/app-logs| dir, the
>>         |userlogs| dir is deleted at the end of the job execution.
>>         How can I make MapReduce to not delete files in the
>>         |userlogs| dir?
>>
>>     I am using Yarn.
>>
>>     Thanks,
>>
>>     [1] Wordcount exampla with just the map part.
>>
>>     |public class MyWordCount { public static class MyMap extends
>>     Mapper { Log log = LogFactory.getLog(MyWordCount.class); private
>>     final static IntWritable one = new IntWritable(1); private Text
>>     word = new Text(); public void map(LongWritable key, Text value,
>>     OutputCollector<Text, IntWritable> output, Reporter reporter)
>>     throws IOException { StringTokenizer itr = new
>>     StringTokenizer(value.toString()); System.out.println("HERRE");
>>     log.info <http://log.info>("HERRRRRE"); while
>>     (itr.hasMoreTokens()) { word.set(itr.nextToken());
>>     output.collect(word, one); } } public void run(Context context)
>>     throws IOException, InterruptedException { setup(context); try {
>>     while (context.nextKeyValue()) { System.out.println("Key: " +
>>     context.getCurrentKey() + " Value: " +
>>     context.getCurrentValue()); map(context.getCurrentKey(),
>>     context.getCurrentValue(), context); } } finally {
>>     cleanup(context); } } public void cleanup(Mapper.Context context)
>>     {} } |
>>
>>     [2] yarn-site.xml
>>
>>     |<!-- job history --> <property>
>>     <name>yarn.log-aggregation-enable</name> <value>true</value>
>>     </property> <property>
>>     <name>yarn.nodemanager.log.retain-seconds</name>
>>     <value>900000</value> </property> <property>
>>     <name>yarn.nodemanager.remote-app-log-dir</name>
>>     <value>/app-logs</value> </property> |
>>     ​
>
>


Mime
View raw message