Mailing-List: contact user-help@avro.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@avro.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=darose.net; h=message-id:date
	:from:mime-version:to:subject:content-type:
	content-transfer-encoding; q=dns; s=darose.net; b=lF5MPwWASWnykU
	9bP85ZGgc6uOVEqlcAjtcNDo1ML4VlVF74+9y6rUJzwe4ChxBZPMUbFReIicyv78
	LnWVphskRcokk4wr7N9rIEsI5+nXfgxN8QhKRZpxtbv6mWZFFmuc5673wxD7MCv9
	oMQgEVffr+NMXaqU0Crf0NqZeaqf4=
Message-ID: <4C647857.10001@darose.net>
Date: Thu, 12 Aug 2010 18:40:23 -0400
From: David Rosenstrauch <darose@darose.net>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.8) Gecko/20100806 Lanikai/3.1.2
MIME-Version: 1.0
To: mapreduce-user@hadoop.apache.org, user@avro.apache.org
Subject: How to work around MAPREDUCE-1700
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Anyone have any ideas how I might be able to work around 
https://issues.apache.org/jira/browse/MAPREDUCE-1700 ?  It's quite a 
thorny issue!

I have a M/R job that's using Avro (v1.3.3).  Avro, in turn, has a 
dependency on Jackson (of which I'm using v1.5.4).  I'm able to add the 
jars to the distributed cache fine, and my Mapper starts to run and load 
Avro ... and then blammo:  "Error: 
org.codehaus.jackson.JsonFactory.enable(Lorg/codehaus/jackson/JsonParser$Feature;)Lorg/codehaus/jackson/JsonFactory;"

The problem is that there's already an older (and obviously 
incompatible) version of Jackson (v1.0.1) that's already included in the 
Hadoop distribution.  And since that appears earlier on the classpath 
than my Jackson jars, I get the error.

There doesn't seem to be any elegant solution to this.  I can't 
downgrade to an earlier version of Avro, as my code relies on features 
in the newer version.  And there doesn't seem to be any way 
configuration-wise to solve this either (i.e., tell Hadoop to use the 
newer Jackson jars for my M/R job, or to add those jars earlier on the 
classpath).

Near as I can tell, the only solutions involve doing a hack on each of 
my slave nodes.  I.e., either:

a) removing the existing jackson jars on each slave.  (Since I have no 
need for the Hadoop feature that requires that Jackson.)

b) putting my newer jackson jars onto each slave in a place where it 
will be loaded before the older one (e.g., 
/usr/lib/hadoop-0.20/lib/aaaaa_jackson-core-asl-1.5.4.jar)

Either of these options is a bit of a hack - and error prone as well, 
since my job tasks will fail on any node that doesn't have this hack 
applied.


Is there any cleaner way to resolve this issue that I'm not seeing?

Thanks,

DR