hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Navis Ryu" <navis....@nexr.com>
Subject Review Request 23971: HCatInputFormat in hive-hcatalog-core 0.13 does not compatible with hadoop2
Date Mon, 28 Jul 2014 01:25:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23971/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-6970
    https://issues.apache.org/jira/browse/HIVE-6970


Repository: hive-git


Description
-------

I'm developing HCatalog support for Crunch (CRUNCH-340). Crunch support both hadoop 1 and
2, by specify profile -Phadoop-1 or -Phadoop-2. When I run my program with hadoop2, I got
the following error. I think this was intended to be fixed by HIVE-4460.

{code}
Exception in thread "Thread-4" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext,
but class was expected
        at org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:102)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491)
        at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchControlledJob.submit(CrunchControlledJob.java:331)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.startReadyJobs(CrunchJobControl.java:201)
        at org.apache.crunch.hadoop.mapreduce.lib.jobcontrol.CrunchJobControl.pollJobStatusAndStartNewOnes(CrunchJobControl.java:235)
        at org.apache.crunch.impl.mr.exec.MRExecutor.monitorLoop(MRExecutor.java:112)
        at org.apache.crunch.impl.mr.exec.MRExecutor.access$000(MRExecutor.java:55)
        at org.apache.crunch.impl.mr.exec.MRExecutor$1.run(MRExecutor.java:83)
        at java.lang.Thread.run(Thread.java:744)

{code}


Diffs
-----

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 93a03ad 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/transfer/impl/HCatInputFormatReader.java
8669ded 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DefaultOutputCommitterContainer.java
cead40d 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DefaultOutputFormatContainer.java
3a07b0c 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DefaultRecordWriterContainer.java
209d7bc 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/DynamicPartitionFileRecordWriterContainer.java
4df912a 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputCommitterContainer.java
491da14 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileOutputFormatContainer.java
1a7595f 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FileRecordWriterContainer.java
2a883d6 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java
33807f5 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseOutputFormat.java
4f7a74a 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatMapRedUtil.java b651cb3

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatRecordReader.java 3ee6157

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InternalUtil.java 9b97939

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/MultiOutputFormat.java 69a7345

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/ProgressReporter.java 40b9bb1

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/Security.java 39ef86e 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/rcfile/RCFileMapReduceInputFormat.java
0c3d707 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/rcfile/RCFileMapReduceOutputFormat.java
64699f2 
  hcatalog/core/src/main/java/org/apache/hive/hcatalog/rcfile/RCFileMapReduceRecordReader.java
fd42b5a 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/rcfile/TestRCFileMapReduceInputFormat.java
02a13b3 
  shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java d03f270 
  shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java b85a69c 
  shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java e98b54d 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java eefd5e5 

Diff: https://reviews.apache.org/r/23971/diff/


Testing
-------


Thanks,

Navis Ryu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message