From user-return-20108-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org Fri Jun 26 16:14:05 2015 Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 810611817F for ; Fri, 26 Jun 2015 16:14:05 +0000 (UTC) Received: (qmail 21488 invoked by uid 500); 26 Jun 2015 16:13:59 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 21376 invoked by uid 500); 26 Jun 2015 16:13:59 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 21358 invoked by uid 99); 26 Jun 2015 16:13:59 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 26 Jun 2015 16:13:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 14FF7D079D for ; Fri, 26 Jun 2015 16:13:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.292 X-Spam-Level: **** X-Spam-Status: No, score=4.292 tagged_above=-999 required=6.31 tests=[DKIM_ADSP_CUSTOM_MED=0.001, DKIM_SIGNED=0.1, HTML_MESSAGE=3, NML_ADSP_CUSTOM_MED=1.2, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_DKIM_INVALID=0.01, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=fail (2048-bit key) reason="fail (body has been altered)" header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id p_6fKPsJoewC for ; Fri, 26 Jun 2015 16:13:49 +0000 (UTC) Received: from mail-wi0-f181.google.com (mail-wi0-f181.google.com [209.85.212.181]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id EBD432095A for ; Fri, 26 Jun 2015 16:13:48 +0000 (UTC) Received: by wicnd19 with SMTP id nd19so49236464wic.1 for ; Fri, 26 Jun 2015 09:13:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject :content-type; bh=+lHHzrrmYjTKug+keoSY3b3eEewW7sV/Ru+2V7PZ1fM=; b=DrRfB0f0lIPkf1YN1ZRQ3rXpqjzvq4sqcJYrZXrk2/b4K2nnFoZo11gGo3aQ1wVFgC NtK2RAJzdYxAH/HdT+sO5Nc4EUVjfJzDHdRTMw/dpzdiiTuq4UvsTHiu1lVCGEGBxWW4 nnppRtzGFztKctIKfUqi2qFggQNUPnToZCvsla4crH8ce19LMOfIn7H2aGOKCjlYlz69 DVOQeFLeeb0xySSguPsyYxQskFVMvrg2JnyJxC48WgHJOfQvjZmBtWfbw9xvVVw98IjB qTZV0xIpNbVr6YtDlNF1qrcWBVtsP6EjazYKkuJhHHu0G77/64Lm09A1uUkGNxFyeKhW UfHA== X-Received: by 10.180.83.135 with SMTP id q7mr6149800wiy.17.1435335227645; Fri, 26 Jun 2015 09:13:47 -0700 (PDT) Received: from [10.10.5.115] ([194.117.22.6]) by mx.google.com with ESMTPSA id ev8sm34729786wjb.8.2015.06.26.09.13.46 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Jun 2015 09:13:46 -0700 (PDT) Message-ID: <558D7A3A.9020100@gmail.com> Date: Fri, 26 Jun 2015 17:13:46 +0100 From: xeonmailinglist-gmail User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "user@hadoop.apache.org" Subject: Invalid key type in the map task Content-Type: multipart/alternative; boundary="------------060506040306090304060507" This is a multi-part message in MIME format. --------------060506040306090304060507 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable Hi, I have this map class that is accepting input files with a key as=20 LongWritable and a value of Text. The input file is in [1]. Here we can see that it contains a key as a=20 Long (I think) and bytes as value. In [2], it is my map class. The goal of the map class is to read the=20 input data, and sent it out as it is. I was expecting that the key was the index number, and the value was the = gibberish. But, I am having a strange error [3], as the key is getting=20 there a s Text, and not the index. Can anyone tell me why I get this error, and how I can solve it? [1] Input file |xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-143497070= 7/part-m-00000 15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoo= p library for your platform... using builtin-java classes where applicabl= e 0 SEQ=06"org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.Bytes= WritableI=EF=BF=BD=EF=BF=BD=EF=BF=BD#HA=EF=BF=BD=EF=BF=BDu=EF=BF=BD=EF=BF= =BDBZ=03=EF=BF=BD=03=EF=BF=BD=02n=EF=BF=BD=EF=BF=BD=04tX=1CeX=EF=BF=BD=EF= =BF=BD=EF=BF=BD=EF=BF=BDzZi=EF=BF=BD=EF=BF=BDn=03dFw=EF=BF=BD{>=EF=BF=BDH= =EF=BF=BD=14\=EF=BF=BD=1E=EF=BF=BD=EF=BF=BDBi=EF=BF=BD'=EF=BF=BDg=06b=EF=BF= =BDY=EF=BF=BDa=10=EF=BF=BD*4=EF=BF=BD7;=EF=BF=BD=EF=BF=BD=EF=BF=BDuL_=EF=BF= =BD=EF=BF=BDgs=EF=BF=BD=EF=BF=BD=02=EF=BF=BD`=EF=BF=BD=EF=BF=BD=03* 187 =EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=1AtAj#=EF=BF=BD5*nX=EF=BF=BDb= =1A"=03=EF=BF=BD=EF=BF=BD>=EF=BF=BD=CB=A5k=EF=BF=BDc2=EF=BF=BD)=EF=BF=BD1= z=EF=BF=BD1=EF=BF=BD=11=EF=BF=BD=EF=BF=BD6=EF=BF=BDl=EF=BF=BD=EF=BF=BD=EF= =BF=BD=10Q=13=19=EF=BF=BDbe3'$=14=EF=BF=BDG=EF=BF=BD=EF=BF=BD!{=EF=BF=BD=03= =EF=BF=BD=1F=18u=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDz@=EF=BF=BDt=EF=BF=BD= =EF=BF=BD=EF=BF=BD=EF=BF=BDv=EF=BF=BD=EF=BF=BD=EF=BF=BDr=EF=BF=BDgj8fq=EF= =BF=BDP=EF=BF=BD=EF=BF=BD.=EF=BF=BDm=1D=EF=BF=BD=EF=BF=BD=EF=BF=BD=11y=03= =EF=BF=BD=EF=BF=BDn=EF=BF=BD=EF=BF=BD!=EF=BF=BD=EF=BF=BD=D6=8C=EF=BF=BD_w= =EF=BF=BDU8e=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDv0=EF=BF=BD=EF=BF= =BD=EF=BF=BDS=EF=BF=BD=EF=BF=BD=EF=BF=BD2c=EF=BF=BD_=EF=BF=BD=D5=BB=EF=BF= =BDR\x=10*=EF=BF=BD=EF=BF=BD = = =04=1D=1C=EF=BF=BD=DC=81=EF=BF=BD=EF=BF=BD=EF=BF=BD|x = = U=EF=BF=BD3=EF=BF=BD=EF=BF=BD=EF=BF=BDl=EF=BF=BD=EF= =BF=BD=EF=BF=BDqa=02q=EF=BF=BD=1D)=EF=BF=BDN=16=EF=BF=BD=EF=BF=BDk=EF=BF=BD= L=EF=BF=BD=EF=BF=BDa=EF=BF=BD2=EF=BF=BD=D5=B6=EF=BF=BD=EF=BF=BD=EF=BF=BDk= NL 404 =EF=BF=BD=EF=BF=BD)i=01=EF=BF=BDg A=EF=BF=BD2=01)L=EF=BF=BD=EF=BF= =BDW,=EF=BF=BD=EF=BF=BD=EF=BF=BD=DC=BA=EF=BF=BDV=EF=BF=BDw=EF=BF=BD=02=EF= =BF=BD=EE=B4=81M7=DA=ADC=CF=A3=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDZI=EF=BF= =BD=EF=BF=BD=1Dbe=EF=BF=BD=EF=BF=BD$=EF=BF=BDf=1A[=EF=BF=BD=EF=BF=BD=06=EF= =BF=BD=04W=EF=BF=BD=067=EF=BF=BD=EF=BF=BD5=EF=BF=BD6.}=EF=BF=BD=D1=BD=18=11= b=EF=BF=BD =01=EF=BF=BD=EF=BF=BD=EF=BF=BD=EF=BF=BDC=EF=BF=BD=EF=BF=BD%= |=EF=BF=BDh=EF=BF=BD=C7=89=EF=BF=BDD=1A}=12@=02F=EF=BF=BD"=EF=BF=BD(K=14=EF= =BF=BD=EF=BF=BDxh=EF=BF=BD'6 | [2] My map class | /** Identity mapper set by the user. */ public static class MyFullyIndentityMapperWebdataScan extends Mapper{ // private LongWritable word =3D new LongWritable(); private LongWritable word =3D new LongWritable(); private Text rvalue =3D new Text(); public void map(LongWritable key, Text value, Context context ) throws IOException, InterruptedException { System.out.println("1: " + key.getClass().getName() + " " + = value.getClass().getName()); System.out.println("2: " + context.getCurrentKey().getClass(= ) + " " + context.getCurrentValue().getClass()); context.write(key, value); } } | [3] Output of execution | Log Type: stdout Log Upload Time: 26-Jun-2015 11:58:53 Log Length: 138 1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class o= rg.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text Log Type: syslog Log Upload Time: 26-Jun-2015 11:58:53 Log Length: 4367 Showing 4096 bytes of 4367 total. Click here = for the full log. operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [ma= in] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot= period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.h= adoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 201= 5-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Exec= uting with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.= mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, I= dent: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d76= 57) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChil= d: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58= :32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster= =2Elocal.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/a= ppcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main= ] org.apache.hadoop.conf.Configuration.deprecation: session.id is depreca= ted. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [ma= in] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree := [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask= : Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00= 000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.= MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO= [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 = 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: sof= t limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop= =2Emapred.MapTask: bufstart =3D 0; bufvoid =3D 104857600 2015-06-26 11:58= :33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart =3D 2621439= 6; length =3D 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hado= op.mapred.MapTask: Map output collector class =3D org.apache.hadoop.mapre= d.MapTask$MapOutputBuffer 2015-06-26 11:58:33,269 INFO [main] org.apache.= hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,2= 76 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running chil= d : java.io.IOException: Type mismatch in key from map: expected org.apac= he.hadoop.io.Text, received org.apache.hadoop.io.LongWritable at org.= apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069) = at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.j= ava:712) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextIm= pl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.map= reduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at= org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWe= bdataScan.map(MyWebDataScan.java:144) at org.apache.hadoop.mapred.exa= mples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.j= ava:131) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) = at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784) = at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apa= che.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.securit= y.AccessController.doPrivileged(Native Method) at javax.security.auth= =2ESubject.doAs(Subject.java:422) at org.apache.hadoop.security.UserG= roupInformation.doAs(UserGroupInformation.java:1628) at org.apache.ha= doop.mapred.YarnChild.main(YarnChild.java:158) | =E2=80=8B --=20 -- Thanks, --------------060506040306090304060507 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit

Hi,

I have this map class that is accepting input files with a key as LongWritable and a value of Text.

The input file is in [1]. Here we can see that it contains a key as a Long (I think) and bytes as value.
In [2], it is my map class. The goal of the map class is to read the input data, and sent it out as it is.
I was expecting that the key was the index number, and the value was the gibberish. But, I am having a strange error [3], as the key is getting there a s Text, and not the index.

Can anyone tell me why I get this error, and how I can solve it?

[1] Input file

xubuntu@hadoop-coc-1:~/Programs/hadoop$ hdfs dfs -cat /output1-1434970707/part-m-00000
15/06/26 12:01:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
0    SEQ"org.apache.hadoop.io.BytesWritable"org.apache.hadoop.io.BytesWritableI���#HA��u��BZ��n��tXeX����zZi��ndFw�{>�H�\���Bi�'�gb�Y�a�*4�7;���uL_��gs���`��*
187    ����tAj#�5*nX�b"��>�˥k�c2�)�1z�1���6�l���Q�be3'$�G��!{��u����z@�t����v���r�gj8fq�P��.�m���y��n��!��֌�_w�U8e�����v0���S���2c�_�ջ�R\x*��
                                                                                                                                                             �܁���|x
                                                                                                                                                                       U�3���l���qaq�)�N��k�L��a�2�ն���k    NL
404    ��)i�g    A�2)L��W,���ܺ�V�w��M7ڭCϣ����ZI��be��$�f[���W�7��5�6.}�ѽb�    ����C��%|�h�lj�D}@F�"�(K��xh�'6

[2] My map class


    /** Identity mapper set by the user. */
    public static class MyFullyIndentityMapperWebdataScan 
    extends Mapper<LongWritable, Text, LongWritable, Text>{

        //        private LongWritable word = new LongWritable();
        private LongWritable word = new LongWritable();
        private Text rvalue = new Text();

        public void map(LongWritable key, Text value, Context context
                ) throws IOException, InterruptedException {

            System.out.println("1: " + key.getClass().getName() + " " + value.getClass().getName());
            System.out.println("2: " + context.getCurrentKey().getClass() + " " + context.getCurrentValue().getClass());
            context.write(key, value);
        }
    }

[3] Output of execution


             Log Type: stdout           

             Log Upload Time: 26-Jun-2015 11:58:53           

             Log Length: 138           

1: org.apache.hadoop.io.LongWritable org.apache.hadoop.io.Text 2: class org.apache.hadoop.io.LongWritable class org.apache.hadoop.io.Text 



             Log Type: syslog           

             Log Upload Time: 26-Jun-2015 11:58:53           

             Log Length: 4367           

             Showing 4096 bytes of 4367 total. Click              here              for the full log.           

operties from hadoop-metrics2.properties 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2015-06-26 11:58:32,118 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system started 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Executing with tokens: 2015-06-26 11:58:32,128 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1435332416394_0009, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@5d7657) 2015-06-26 11:58:32,208 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again. Got null now. 2015-06-26 11:58:32,439 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-temp/nm-local-dir/usercache/xubuntu/appcache/application_1435332416394_0009 2015-06-26 11:58:32,611 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
  
session.id is deprecated. Instead, use dfs.metrics.session-id 2015-06-26 11:58:33,038 INFO [main] org.apache.hadoop.mapred.Task:  Using ResourceCalculatorProcessTree : [ ] 2015-06-26 11:58:33,180 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: hdfs://hadoop-coc-1:9000/output1-1434970707/part-m-00000:0+17853 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-06-26 11:58:33,252 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600 2015-06-26 11:58:33,255 INFO [main] org.apache.hadoop.mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2015-06-26 11:
 5
8:33,269 INFO [main] org.apache.hadoop.mapred.MapTask: Starting flush of map output 2015-06-26 11:58:33,276 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, received org.apache.hadoop.io.LongWritable     at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1069)     at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:712)     at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)     at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)     at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:144)     at org.apache.hadoop.mapred.examples.MyWebDataScan$MyFullyIndentityMapperWebdataScan.map(MyWebDataScan.java:131)     at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)     at org.apache.
 h
adoop.mapred.MapTask.runNewMapper(MapTask.java:784)     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)     at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)     at java.security.AccessController.doPrivileged(Native Method)     at javax.security.auth.Subject.doAs(Subject.java:422)     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)     at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
​
-- 
--
Thanks,
--------------060506040306090304060507--