Return-Path: 
 <user-return-14227-apmail-hadoop-hdfs-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 5F32E10308
	for <apmail-hadoop-hdfs-user-archive@minotaur.apache.org>;
 Mon, 10 Mar 2014 03:33:57 +0000 (UTC)
Received: (qmail 53446 invoked by uid 500); 10 Mar 2014 03:33:46 -0000
Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org
Received: (qmail 52478 invoked by uid 500); 10 Mar 2014 03:33:43 -0000
Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:user@hadoop.apache.org>
List-Id: <user.hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Delivered-To: mailing list user@hadoop.apache.org
Received: (qmail 52459 invoked by uid 99); 10 Mar 2014 03:33:42 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Mar 2014 03:33:42 +0000
X-ASF-Spam-Status: No, hits=1.5 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of raofengyun@gmail.com
 designates 209.85.128.181 as permitted sender)
Received: from [209.85.128.181] (HELO mail-ve0-f181.google.com)
 (209.85.128.181)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Mar 2014 03:33:37 +0000
Received: by mail-ve0-f181.google.com with SMTP id oy12so6420535veb.40
        for <user@hadoop.apache.org>; Sun, 09 Mar 2014 20:33:16 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=20120113;
        h=mime-version:date:message-id:subject:from:to:content-type;
        bh=wFS/smJLC8mQq0hUxCrmAhuYCRJP7uUOHgwoWqAk42w=;
        b=XPAo81dAdbUH5Dj8/vpnBL9QfyH7FjjROUps9HwAHfktJ9zNeRvF65pfrTN1/24P2h
         oG4SpxIQs74LF27mGbKoNNePpop4dKJzxhkuZ2H9aXk7u6D46WVVIfrYnvtN78uXwYLL
         3IsjND+leoB4AVQXsejlmWBsFE6saOcjA0D8srVFmPhjowWVlGIVSw191O5Kg/SSjJ5j
         Jyb1rU3EkkF6QWcnLLSfEExXaqHFLqVAwT0YN80SpZQonncrShrUM/9wEb1jMZimfzU5
         TQp1DbZvy9y2jd2zW3imaJKdMykOR80HFqAv3xHpWUyfv2KTqNHh067KlKeteNjQlZKr
         fN9w==
MIME-Version: 1.0
X-Received: by 10.220.114.135 with SMTP id e7mr1174429vcq.23.1394422396573;
 Sun, 09 Mar 2014 20:33:16 -0700 (PDT)
Received: by 10.220.232.68 with HTTP; Sun, 9 Mar 2014 20:33:16 -0700 (PDT)
Date: Mon, 10 Mar 2014 11:33:16 +0800
Message-ID: 
 <CAGSyEuBfw2=OyDeC=QB6QVuBJQ+p7aYWkozanLbi=WSESoKj-Q@mail.gmail.com>
Subject: What's the best practice for managing Hadoop dependencie?
From: Fengyun RAO <raofengyun@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b342d8a9d669704f4384300
X-Virus-Checked: Checked by ClamAV on apache.org

--047d7b342d8a9d669704f4384300
Content-Type: text/plain; charset=ISO-8859-1

First of all, I want to claim that I used CDH5 beta, and managed project
using maven, and I googled and read a lot, e.g.
https://issues.apache.org/jira/browse/MAPREDUCE-1700
http://www.datasalt.com/2011/05/handling-dependencies-and-configuration-in-java-hadoop-projects-efficiently/

I believe the problem is quite common, when we write an MR job, we need
lots of dependencies,
which may not exist in or conflict with HADDOP_CLASSPATH.
There are several options, e.g.
1. add all libraries to my own JAR, and set HADOOP_USER_CLASSPATH_FIRST=true
   This is what I do, which makes the jar very big, and still it doesn't
work.
   e.g. I already packaged guava-16.0.jar in my package, but it still use
guava-11.0.2.jar in the HADDOP_CLASSPATH.
   below is my build configuration.
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <mainClass>xxx.xxx.xxx.Runner</mainClass>
                        </manifest>
                    </archive>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

2. distinguish which library is not present in HADDOP_CLASSPATH, and put it
into DistributedCache
    I think it's hard to distinguish, and still if it conflicts, which
dependency would be precedent?


*What's the best practice, especially using maven?*

--047d7b342d8a9d669704f4384300
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">First of all, I want to claim that I used CDH5 beta, and m=
anaged project using maven, and I googled and read a lot, e.g.=A0<div><a hr=
ef=3D"https://issues.apache.org/jira/browse/MAPREDUCE-1700">https://issues.=
apache.org/jira/browse/MAPREDUCE-1700</a><br>
</div><div><a href=3D"http://www.datasalt.com/2011/05/handling-dependencies=
-and-configuration-in-java-hadoop-projects-efficiently/">http://www.datasal=
t.com/2011/05/handling-dependencies-and-configuration-in-java-hadoop-projec=
ts-efficiently/</a><br>
</div><div><br></div><div>I believe the problem is quite common, when we wr=
ite an MR job, we need lots of dependencies,</div><div>which may not exist =
in or conflict with HADDOP_CLASSPATH.=A0</div><div>There are several option=
s, e.g.</div>
<div>1. add all libraries to my own JAR, and set HADOOP_USER_CLASSPATH_FIRS=
T=3Dtrue</div><div>=A0 =A0This is what I do, which makes the jar very big, =
and still it doesn&#39;t work.=A0</div><div>=A0 =A0e.g. I already packaged =
guava-16.0.jar in my package, but it still use guava-11.0.2.jar in the HADD=
OP_CLASSPATH.</div>
<div>=A0 =A0below is my build configuration.</div><div><div>=A0 =A0 =A0 =A0=
 =A0 =A0 &lt;plugin&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;artif=
actId&gt;maven-assembly-plugin&lt;/artifactId&gt;</div><div>=A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0 &lt;configuration&gt;</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;archive&gt;</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;manifest&gt;</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;mainClass&gt;xxx.xx=
x.xxx.Runner&lt;/mainClass&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 &lt;/manifest&gt;</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;/archive&gt;</div><div>=A0=
 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;descriptorRefs&gt;</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;descriptorRef&gt;jar-with-d=
ependencies&lt;/descriptorRef&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 &lt;/descriptorRefs&gt;</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;/configuration&gt;</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;executions&gt;</div><div>=A0 =A0 =A0 =A0 =
=A0 =A0 =A0 =A0 =A0 =A0 &lt;execution&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0=
 =A0 =A0 =A0 =A0 =A0 =A0 &lt;id&gt;make-assembly&lt;/id&gt;</div><div>=A0 =
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;phase&gt;package&lt;/phase&=
gt;</div>
<div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;goals&gt;</div><di=
v>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;goal&gt;singl=
e&lt;/goal&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &l=
t;/goals&gt;</div><div>=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;/executi=
on&gt;</div><div>
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 &lt;/executions&gt;</div><div>=A0 =A0 =A0 =
=A0 =A0 =A0 &lt;/plugin&gt;</div></div><div><br></div><div>2. distinguish w=
hich library is not present in HADDOP_CLASSPATH, and put it into Distribute=
dCache</div><div>=A0 =A0 I think it&#39;s hard to distinguish, and still if=
 it conflicts, which dependency would be precedent?</div>
<div><br></div><div><br></div><div><b>What&#39;s the best practice, especia=
lly using maven?</b></div><div><br></div><div><br></div></div>

--047d7b342d8a9d669704f4384300--