hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tahir Hashmi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1001) the output of the map is not type checked against the specified types
Date Mon, 02 Apr 2007 18:52:32 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tahir Hashmi updated HADOOP-1001:
---------------------------------

    Attachment: TestMapOutputType.java

The problem with this test is that the exception thrown by MapTask doesn't get propagated
to the client. The best effort here is to just make sure that the test cases fail for whatever
reason when wrong key or value types are specified and pass with correct types. Hopefully
the earlier runs would've failed due to type mismatch exceptions rather than something else.
Attaching the revised version of the test now. Here's the kind of output it produces in the
logs:

$ cat build/test/TEST-org.apache.hadoop.mapred.TestMapOutputType.txt
Testsuite: org.apache.hadoop.mapred.TestMapOutputType
Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 5.585 sec
------------- Standard Output ---------------
2007-04-03 00:18:02,162 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(51))
- Unable to load native-hadoop library for your platform... using builtin-java classes where
applicable
2007-04-03 00:18:02,508 INFO  mapred.InputFormatBase (InputFormatBase.java:validateInput(141))
- Total input paths to process : 1
2007-04-03 00:18:03,238 WARN  mapred.LocalJobRunner (LocalJobRunner.java:run(166)) - job_bzpgha
java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.IntWritable,
recieved org.apache.hadoop.io.Text
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:312)
        at org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:04,121 INFO  mapred.InputFormatBase (InputFormatBase.java:validateInput(141))
- Total input paths to process : 1
2007-04-03 00:18:04,249 WARN  mapred.LocalJobRunner (LocalJobRunner.java:run(166)) - job_4f7xh3
java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.IntWritable,
recieved org.apache.hadoop.io.Text
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:317)
        at org.apache.hadoop.mapred.TestMapOutputType$TextGen.map(TestMapOutputType.java:50)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:175)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:126)
2007-04-03 00:18:05,324 INFO  mapred.InputFormatBase (InputFormatBase.java:validateInput(141))
- Total input paths to process : 1
2007-04-03 00:18:05,604 INFO  mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) -
file:/home/tahir/Desktop/Hadoop/build/test/test.mapred.spill/in/part0:0+150
2007-04-03 00:18:05,752 INFO  mapred.LocalJobRunner (LocalJobRunner.java:progress(188)) -
reduce > reduce
------------- ---------------- ---------------

Testcase: testKeyMismatch took 3.144 sec
Testcase: testValueMismatch took 1.128 sec
Testcase: testNoMismatch took 1.302 sec


> the output of the map is not type checked against the specified types
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-1001
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1001
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.11.0
>            Reporter: Owen O'Malley
>         Assigned To: Tahir Hashmi
>             Fix For: 0.13.0
>
>         Attachments: 1001.patch, 1001.patch, 1001.patch, TestMapOutputType.java, TestMapOutputType.java
>
>
> The output of the map is not checked against the types specified in the JobConf leading
to hard to diagnose bugs.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message