Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of jwills@cloudera.com
 designates 209.85.216.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHCsPn_ZeDOupF8iqnRA0nmcV+xqOk9huPWvkdSzSLzGA01sTw@mail.gmail.com>
References: 
 <CAHCsPn_ZeDOupF8iqnRA0nmcV+xqOk9huPWvkdSzSLzGA01sTw@mail.gmail.com>
From: Josh Wills <jwills@cloudera.com>
Date: Tue, 1 Oct 2013 13:10:08 -0700
Message-ID: 
 <CAH29n6PR92cSYA2QysZcvJNHxU0DNATvJb1iR3OdsNk9a+dp9g@mail.gmail.com>
Subject: Re: Crunch on EMR
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=047d7bdc1b864ee91204e7b38b83

--047d7bdc1b864ee91204e7b38b83
Content-Type: text/plain; charset=ISO-8859-1

Hey Som,

You should be able to use any of the non-hadoop2 jars for Crunch on EMR,
like the regular 0.7.0:

http://mvnrepository.com/artifact/org.apache.crunch/crunch-core/0.7.0

Those are compiled against the MR1 APIs, which is why you're getting the
TaskInputOutputContext exception (the API changed from MR1 to MR2, which
CDH4.3.0 and hadoop2 use.)

Josh


On Tue, Oct 1, 2013 at 12:00 PM, Som Satpathy <somsatpathy@gmail.com> wrote:

> Hi All,
>
> I have been trying to run crunch jobs on amazon EMR and faced a problem
> while job execution -
>
> "found class org.apache.hadoop.mapreduce.taskinputoutputcontext but
> interface was expected"
>
> This is happening because of hadoop incompatibilities between APIs used
> while implementing the hadoop job, and the hadoop-code that runs in the
> cluster.
>
> My crunch fat jar is based on crunch version 0.7 (CDH 4.3.0) while EMR
> runs hadoop 1.0.3 (where TaskInputOutputContext is implemented as an
> abstract class)
>
> Has any one been able to successfully execute their crunch jobs on EMR?
>
> If yes, what are the best practices to make custom crunch fat jars work on
> EMR?
>
>
> Look forward to hearing your thoughts.
>
> Thanks,
>
> Som
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

--047d7bdc1b864ee91204e7b38b83
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey Som,<div><br></div><div>You should be able to use any =
of the non-hadoop2 jars for Crunch on EMR, like the regular 0.7.0:</div><di=
v><br></div><div><a href=3D"http://mvnrepository.com/artifact/org.apache.cr=
unch/crunch-core/0.7.0">http://mvnrepository.com/artifact/org.apache.crunch=
/crunch-core/0.7.0</a><br>

</div><div><br></div><div>Those are compiled against the MR1 APIs, which is=
 why you&#39;re getting the TaskInputOutputContext exception (the API chang=
ed from MR1 to MR2, which CDH4.3.0 and hadoop2 use.)</div><div><br></div>

<div>Josh</div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">On Tue, Oct 1, 2013 at 12:00 PM, Som Satpathy <span dir=3D"ltr">&lt=
;<a href=3D"mailto:somsatpathy@gmail.com" target=3D"_blank">somsatpathy@gma=
il.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi All,<div><br></div><div>=
I have been trying to run crunch jobs on amazon EMR and faced a problem whi=
le job execution -=A0</div>

<div>


<p>&quot;found class org.apache.hadoop.mapreduce.taskinputoutputcontext but=
 interface was expected&quot;</p>
<p>This is happening because of hadoop incompatibilities between APIs used =
while implementing the hadoop job, and the hadoop-code that runs in the clu=
ster.</p><p>My crunch fat jar is based on crunch version 0.7 (CDH 4.3.0) wh=
ile EMR runs hadoop 1.0.3 (where TaskInputOutputContext is implemented as a=
n abstract class)</p>


<p>Has any one been able to successfully execute their crunch jobs on EMR?<=
/p><p>If yes, what are the best practices to make custom crunch fat jars wo=
rk on EMR?</p><p><br></p><p>Look forward to hearing your thoughts.</p>


<p>Thanks,</p><p>Som</p>


</div></div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div>Directo=
r of Data Science</div><div><a href=3D"http://www.cloudera.com" target=3D"_=
blank">Cloudera</a></div><div>Twitter: <a href=3D"http://twitter.com/josh_w=
ills" target=3D"_blank">@josh_wills</a></div>


</div>

--047d7bdc1b864ee91204e7b38b83--