hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From xeonmailinglist-gmail <xeonmailingl...@gmail.com>
Subject Invalid key type in the map task
Date Fri, 26 Jun 2015 16:13:46 GMT
Hi,

I have this map class that is accepting input files with a key as 
LongWritable and a value of Text.

The input file is in [1]. Here we can see that it contains a key as a 
Long (I think) and bytes as value.
In [2], it is my map class. The goal of the map class is to read the 
input data, and sent it out as it is.
I was expecting that the key was the index number, and the value was the 
gibberish. But, I am having a strange error [3], as the key is getting 
there a s Text, and not the index.

Can anyone tell me why I get this error, and how I can solve it?

[1] Input file

|xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your
platform... using builtin-java classes where applicable
0    SEQ"org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ��n��tXeX����zZi��ndFw�{>�H�\���Bi�'�gb�Y�a�*4�7;���uL_��gs���`��*
187    ����tAj#�5*nX�b"��>�˥k�c2�)�1z�1���6�l���Q�be3'$�G��!{��u����z@�t����v���r�gj8fq�P��.�m���y��n��!��֌�_w�U8e�����v0���S���2c�_�ջ�R\x*��
                                                                                         
                                                                    �܁���|x
                                                                                         
                                                                              U�3���l���qaq�)�N��k�L��a�2�ն���k
   NL
404    ��)i�g    A�2)L��W,���ܺ�V�w��M7ڭCϣ����ZI��be��$�f[���W�7��5�6.}�ѽb�
   ����C��%|�h�lj�D}@F�"�(K��xh�'6
|

[2] My map class

|
     /** Identity mapper set by the user. */
     public static class MyFullyIndentityMapperWebdataScan
     extends Mapper<LongWritable, Text, LongWritable, Text>{

         //        private LongWritable word = new LongWritable();
         private LongWritable word = new LongWritable();
         private Text rvalue = new Text();

         public void map(LongWritable key, Text value, Context context
                 ) throws IOException, InterruptedException {

             System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
             System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
             context.write(key, value);
         }
     }
|

[3] Output of execution

|
              Log Type: stdout

              Log Upload Time: 26-Jun-2015 11:58:53

              Log Length: 138

1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable
class org.apache.hadoop.io.Text



              Log Type: syslog

              Log Upload Time: 26-Jun-2015 11:58:53

              Log Length: 4367

              Showing 4096 bytes of 4367 total. Click              here              for the
full log.

operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657)
2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009
2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id
is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task:
 Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26
11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb:
100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid
= 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart
= 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask:
Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:58:33,269
INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276
WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException:
Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)     at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
    at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144)
    at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)     at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
    at java.security.AccessController.doPrivileged(Native Method)     at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
|

​

-- 
--
Thanks,


Mime
View raw message