hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dieter De Witte <drdwi...@gmail.com>
Subject Re: Invalid key type in the map task
Date Fri, 26 Jun 2015 16:42:51 GMT
You need to show the driver class as well. Are you using textinputformat?
Are you aware that this standard inputformat will take as a value the
complete line (until newline separator), the key in that case is the
bitoffset in the file and definitely not the number you assume it will be.



2015-06-26 18:13 GMT+02:00 xeonmailinglist-gmail <xeonmailinglist@gmail.com>
:

>  Hi,
>
> I have this map class that is accepting input files with a key as
> LongWritable and a value of Text.
>
> The input file is in [1]. Here we can see that it contains a key as a Long
> (I think) and bytes as value.
> In [2], it is my map class. The goal of the map class is to read the input
> data, and sent it out as it is.
> I was expecting that the key was the index number, and the value was the
> gibberish. But, I am having a strange error [3], as the key is getting
> there a s Text, and not the index.
>
> Can anyone tell me why I get this error, and how I can solve it?
>
> [1] Input file
>
> xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
> 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> 0    SEQ "org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ
� � n�� tX eX����zZi��n dFw�{>�H� \� ��Bi�'�g b�Y�a
�*4�7;���uL_��gs�� �`�� *
> 187    ���� tAj#�5*nX�b " ��>�˥k�c2�)�1z�1� ��6�l���
Q  �be3'$ �G��!{� �  u����z@�t����v���r�gj8fq�P��.�m
��� y ��n��!��֌�_w�U8e�����v0���S���2c�_�ջ�R\x
*��
>                                                                                     
                                                                           �܁���|x
>                                                                                     
                                                                                  U�3���l���qa
q� )�N ��k�L��a�2�ն���k    NL
> 404    ��)i �g    A�2 )L��W,���ܺ�V�w� �M7ڭCϣ����ZI��
be��$�f [�� � W� 7��5�6.}�ѽ  b�     ����C��%|�h�lj�D
} @ F�"�(K ��xh�'6
>
> [2] My map class
>
>
>     /** Identity mapper set by the user. */
>     public static class MyFullyIndentityMapperWebdataScan
>     extends Mapper<LongWritable, Text, LongWritable, Text>{
>
>         //        private LongWritable word = new LongWritable();
>         private LongWritable word = new LongWritable();
>         private Text rvalue = new Text();
>
>         public void map(LongWritable key, Text value, Context context
>                 ) throws IOException, InterruptedException {
>
>             System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
>             System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
>             context.write(key, value);
>         }
>     }
>
> [3] Output of execution
>
>
>              Log Type: stdout
>
>              Log Upload Time: 26-Jun-2015 11:58:53
>
>              Log Length: 138
>
> 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable
class org.apache.hadoop.io.Text
>
>
>
>              Log Type: syslog
>
>              Log Upload Time: 26-Jun-2015 11:58:53
>
>              Log Length: 4367
>
>              Showing 4096 bytes of 4367 total. Click              here              for
the full log.
>
> operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild:
Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild:
Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657)
2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before
retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild:
mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009
2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
>   session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038
INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ] 2015-06-26
11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853
2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb:
100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080
2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid
= 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart
= 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask:
Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
>  5
> 8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26
11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException:
Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)     at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
    at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144)
    at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)     at org.apache.
>  h
> adoop.mapred.MapTask.runNewMapper(MapTask.java:784)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)     at java.security.AccessController.doPrivileged(Native
Method)     at javax.security.auth.Subject.doAs(Subject.java:422)     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>
> ​
>
> --
> --
> Thanks,
>
>

Mime
View raw message