Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1AC0D10B6D for ; Thu, 6 Mar 2014 10:30:12 +0000 (UTC) Received: (qmail 86941 invoked by uid 500); 6 Mar 2014 10:30:05 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 85484 invoked by uid 500); 6 Mar 2014 10:30:02 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 85471 invoked by uid 99); 6 Mar 2014 10:30:01 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Mar 2014 10:30:01 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kingdavies@gmail.com designates 209.85.219.67 as permitted sender) Received: from [209.85.219.67] (HELO mail-oa0-f67.google.com) (209.85.219.67) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Mar 2014 10:29:56 +0000 Received: by mail-oa0-f67.google.com with SMTP id j17so825695oag.10 for ; Thu, 06 Mar 2014 02:29:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=SURq8I5Tya/jZUQ+X0u7JdB1EzmgqmTMPF8qCpmVE6I=; b=z7QiuedTPVt79dQ8NHepl6PAmhysklykKRANElg4ikcfJhBz+kc4Y57dwMZqXkFoaZ GBWFUl0jSr48xkrMYwnlMpVsANhZBZhNmlYd1mwiElYDLdWZVRrkUzI1eFQMHOUPhcq8 3LrsYWq8ewYXwRk36SEx4IG4TQ8WyLNRnzIO2hhxyDIhH7sIN7ZuY5MHhIBAaRjQ+vxg W85vLG+3OR2ymNog34yYnp6h5kniJ2tD2LXPFKAFk0QgJER/ak2CM41aWELzPgxk28o8 dXMrcj8k5n7d+RU0dLiYzWlMmBxxUP+1P6OTnpMoJzyCUbnfgyPwGYlQ8Qw5XOOHFqyl 49NQ== MIME-Version: 1.0 X-Received: by 10.60.233.138 with SMTP id tw10mr955896oec.56.1394101775863; Thu, 06 Mar 2014 02:29:35 -0800 (PST) Received: by 10.60.147.134 with HTTP; Thu, 6 Mar 2014 02:29:35 -0800 (PST) Date: Thu, 6 Mar 2014 10:29:35 +0000 Message-ID: Subject: MR2 Job over LZO data From: KingDavies To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a1136b66c2196fe04f3ed9dfd X-Virus-Checked: Checked by ClamAV on apache.org --001a1136b66c2196fe04f3ed9dfd Content-Type: text/plain; charset=ISO-8859-1 Running on Hadoop 2.2.0 The Java MR2 job works as expected on an uncompressed data source using the TextInputFormat.class. But when using the LZO format the job fails: import com.hadoop.mapreduce.LzoTextInputFormat; job.setInputFormatClass(LzoTextInputFormat.class); Dependencies from the maven repository: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/ Also tried with elephant-bird-core 4.4 The same data can be queried fine from within Hive(0.12) on the same cluster. The exception: Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected at com.hadoop.mapreduce.LzoTextInputFormat.listStatus(LzoTextInputFormat.java:62) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:340) at com.hadoop.mapreduce.LzoTextInputFormat.getSplits(LzoTextInputFormat.java:101) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:491) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:508) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:392) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at com.cloudreach.DataQuality.Main.main(Main.java:42) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) I believe the issue is related to the changes in Hadoop 2, but where can I find a H2 compatible version? Thanks --001a1136b66c2196fe04f3ed9dfd Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Running on Hadoop 2.2.0

The = Java MR2 job works as expected on an uncompressed data source using the Tex= tInputFormat.class.
B= ut when using the LZO format the job fails:
import com.hadoop.mapreduce.LzoT= extInputFormat;
job.s= etInputFormatClass(LzoTextInputFormat.class);

Dep= endencies from the maven repository:
http://maven.twttr.com/com/= hadoop/gplcompression/hadoop-lzo/0.4.19/
Also tried with=A0e= lephant-bird-core 4.4

The same data can be queried fine from within Hive(0.12= ) on the same cluster.


The exceptio= n:
Exceptio= n in thread "main" java.lang.IncompatibleClassChangeError: Found = interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at com.hadoop.mapreduce.LzoTextInputFormat.li= stStatus(LzoTextInputFormat.java:62)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(File= InputFormat.java:340)
at com.hadoop.mapreduce.LzoTextInputFormat.ge= tSplits(LzoTextInputFormat.java:101)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter= .java:491)
at org.apache.hadoop.mapreduce.JobSubmitter.w= riteSplits(JobSubmitter.java:508)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter= .java:392)
at org.apache.hadoop.mapreduce.Job$10.run(Job= .java:1268)
at org.apache.hadoop.= mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivilege= d(Native Method)
at javax.securit= y.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInform= ation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at com.cloudreach.DataQuality.Main.main(Main.= java:42)
= at sun.reflect.NativeMet= hodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invok= e(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.jav= a:606)
at org.apache.hadoop.util.= RunJar.main(RunJar.java:212)

I=A0believe the issue is= related to the=A0changes=A0in Hadoop 2, but where can I find a H2 compatib= le version?

Thanks
--001a1136b66c2196fe04f3ed9dfd--