Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 66045DFA3 for ; Mon, 16 Jul 2012 06:34:37 +0000 (UTC) Received: (qmail 32726 invoked by uid 500); 16 Jul 2012 06:34:36 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 32082 invoked by uid 500); 16 Jul 2012 06:34:29 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 32054 invoked by uid 99); 16 Jul 2012 06:34:28 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Jul 2012 06:34:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: 209.85.213.171 is neither permitted nor denied by domain of jeremy@lewi.us) Received: from [209.85.213.171] (HELO mail-yx0-f171.google.com) (209.85.213.171) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 16 Jul 2012 06:34:23 +0000 Received: by yenq11 with SMTP id q11so6538749yen.30 for ; Sun, 15 Jul 2012 23:34:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=Zsil3sKo/Nb//wMqNGiNnWuD/eGKhce0LZRoTg9jMyo=; b=dB0OyR5y2o+uxkBErbQkN5AOvIg+zhiIcQtOoI340x7zTyrwznZ7hYmlh7TrE3Yaay RwFrStLs4h0Vy4NlMN73i5uEJsYiwkoHNzT0RC+i+YvdUcrM36UUcPs70Lif+fxQuslz kQOj1BXekwBTFunulK39Mx+V+8kUI+mrlmWnD6JuvaEz1gt7ntLhxAFXt4mOSl9NKrBh Row4PFMUbhRHG/b2ec47YQ53MZwC57cC91DzfbfNlgIbkzN4VgnlyXbW1JS7o7LKHjqK jal0pa8+PqHsb7wi59D+Rgz/qN9OH7ZtcGw/mEk5YNT7RZBwJ4ED5Ev8xY63WWC5idq9 2WfA== MIME-Version: 1.0 Received: by 10.50.208.34 with SMTP id mb2mr4563960igc.5.1342420441438; Sun, 15 Jul 2012 23:34:01 -0700 (PDT) Received: by 10.64.101.38 with HTTP; Sun, 15 Jul 2012 23:34:01 -0700 (PDT) X-Originating-IP: [74.95.13.126] In-Reply-To: <3A8076BA-6FD2-4172-86A0-3BE66310E451@synopsys.com> References: <3A8076BA-6FD2-4172-86A0-3BE66310E451@synopsys.com> Date: Sun, 15 Jul 2012 23:34:01 -0700 Message-ID: Subject: Re: SpecificData.deepCopy exception with CDH4 From: Jeremy Lewi To: user@avro.apache.org Cc: Alan.Miller@synopsys.com Content-Type: multipart/alternative; boundary=14dae9340cad8d19e004c4ec9dd6 X-Gm-Message-State: ALoCoQkWuC5YkDdrc1rSp/Ddz96D11z+t7uRWPXlE5Ixz3AM4b9qsPD4pqkB92x0fi35fNSf7k6B X-Virus-Checked: Checked by ClamAV on apache.org --14dae9340cad8d19e004c4ec9dd6 Content-Type: text/plain; charset=ISO-8859-1 Thanks Alan. I'm still getting the same error as before. Here's how I'm running the job * - * hadoop jar ./target/contrail-1.0-SNAPSHOT-job.jar contrail.avro.QuickMergeAvro -D mapreduce.task.classpath.first=true -libjars=/users/jlewi/svn_avro_1.6.1/lang/java/avro/target/avro-1.6.1.jar,/users/jlewi/svn_avro_1.6.1/lang/java/mapred/target/avro-mapred-1.6.1.jar --inputpath=/users/jlewi/staph/assembly/BuildGraph --outputpath=/users/jlewi/staph/assembly/QuickMerge --K=45 I verified via the job tracker that the property "mapreduce.task.classpath.first" is getting picked up. It looks like the problem I'm dealing with is related to https://issues.apache.org/jira/browse/AVRO-1103. Any ideas? Thanks J On Sun, Jul 15, 2012 at 2:00 AM, Alan Miller wrote: > Hi Just a quick idea. > Also check ALL directories returned by > hadoop classpath > for any Avro related classes. > > I was struggling trying to use > avro-1.7.0 with CDH4 but made it work > by using the -libjars option and making sure my classes are used BEFORE > the standard classes. There's a config > property (dont remember) to set for > that. Note the above setting is for the > task's classpath, to control the > classpath of your driver class set > HADOOOP_CLASSPATH=... and > HADOOOP_USER_CLASSPATH_FIRST=true > > Alan > > Sent from my iPhone > > On Jul 15, 2012, at 3:59, "Jeremy Lewi" wrote: > > > hi avro-users, > > > > I'm getting the following exception when using avro 1.6.1 with CDH4. > > java.lang.NoSuchMethodError: > org.apache.avro.specific.SpecificData.deepCopy(Lorg/apache/avro/Schema;Ljava/lang/Object;)Ljava/lang/Object; > > > > The offending code is > > GraphNodeData copy = (GraphNodeData) > SpecificData.get().deepCopy(data.getSchema(), data); > > > > where GraphNodeData is a class generated from my AVRO record. > > > > The code runs just fine on CDH3. I tried rebuilding AVRO from source and > installing it my local repo because of a previous post that said Avro 1.6.1 > in maven had been built against CDH3. I also deleted all the avro jar files > I found in > > /usr/lib/hadoop > > > > Any ideas? Thanks? > > Jeremy > > > > > > > --14dae9340cad8d19e004c4ec9dd6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks Alan.

I'm still getting the same error as bef= ore. Here's how I'm running the job

=
hadoop jar ./target/contrail-1.0-SNAPSHOT-job.jar contrail.avro.QuickMergeA= vro -D mapreduce.task.classpath.first=3Dtrue -libjars=3D/users/jlewi/svn_av= ro_1.6.1/lang/java/avro/target/avro-1.6.1.jar,/users/jlewi/svn_avro_1.6.1/l= ang/java/mapred/target/avro-mapred-1.6.1.jar --inputpath=3D/users/jlewi/sta= ph/assembly/BuildGraph --outputpath=3D/users/jlewi/staph/assembly/QuickMerg= e --K=3D45

I verified via the job tracker that the property "mapre= duce.task.classpath.first" =A0is getting picked up.

It looks like the problem I'm dealing with is related to=A0https://issues.apach= e.org/jira/browse/AVRO-1103.

Any ideas?=A0

Thanks
J
On Sun, Jul 15, 2012 at 2:00 AM, Alan Mill= er <Alan.Miller@synopsys.com> wrote:
Hi Just a quick idea.
Also check ALL directories returned by
=A0 hadoop classpath
for any Avro related classes.

I was struggling trying to use
avro-1.7.0 with CDH4 but made it work
by using the -libjars option and making sure my classes are used BEFORE the= standard classes. There's a config
property (dont remember) to set for
that. Note the above setting is for the
task's classpath, to control the
classpath of your driver class set
HADOOOP_CLASSPATH=3D... and
HADOOOP_USER_CLASSPATH_FIRST=3Dtrue

Alan

Sent from my iPhone

On Jul 15, 2012, at 3:59, "Jeremy Lewi" <jeremy@lewi.us> wrote:

> hi avro-users,
>
> I'm getting the following exception when using avro 1.6.1 with CDH= 4.
> java.lang.NoSuchMethodError: org.apache.avro.specific.SpecificData.dee= pCopy(Lorg/apache/avro/Schema;Ljava/lang/Object;)Ljava/lang/Object;
>
> The offending code is
> GraphNodeData copy =3D (GraphNodeData) SpecificData.get().deepCopy(dat= a.getSchema(), data);
>
> where GraphNodeData is a class generated from my AVRO record.
>
> The code runs just fine on CDH3. I tried rebuilding AVRO from source a= nd installing it my local repo because of a previous post that said Avro 1.= 6.1 in maven had been built against CDH3. I also deleted all the avro jar f= iles I found in
> /usr/lib/hadoop
>
> Any ideas? Thanks?
> Jeremy
>
>
>

--14dae9340cad8d19e004c4ec9dd6--