incubator-giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From André Kelpe <efeshundert...@googlemail.com>
Subject Re: Problem deploying Giraph job to hadoop cluster: onlineZooKeeperServers connection failure
Date Sat, 24 Mar 2012 09:33:47 GMT
2012/3/22 Benjamin Heitmann <benjamin.heitmann@deri.org>:
Hi Benjamin!

> This was the deciding issue. My jar file contained the dependencies as jar files in the
lib dir inside of the job jar.
> While (almost) all Google search results for assembling a hadoop job as a jar suggest
that this is the right way to do it,
> it seems that Giraph or a dependency introduces some changes to the process in which
the job jar is loaded.
>
> After checking out the giraph-*-jar-with-dependencies.jar (with jar -tf), I saw that
all dependency jars are unpacked in there.
> I copied and modified the relevant invocation of the maven assembly plugin to my project
pom.xml and built that jar (with mvn clean assembly:assembly).

Side remark: Please don't use mvn assembly:assembly anymore, all
goals, except mvn assembly:single are deprecated since a few releases
of the asssembly plugin [0]. I am using the assembly descriptor
explained in [1] to build my jobs and I trigger it automatically
during the package phase like shown in [2]. (We only do it in a
release profile, so that we don't pollute our internal maven repo with
huge assembly SNAPSHOT jars)

HTH

--André

[0] https://maven.apache.org/plugins/maven-assembly-plugin/usage.html
[1] https://exported.wordpress.com/2010/01/30/building-hadoop-job-jar-with-maven/
[2] http://pastebin.com/2cg72V6V

Mime
View raw message