giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: Packaging a Giraph application in a jar
Date Thu, 10 Nov 2011 15:18:26 GMT
Ok,

quite basically this is my pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>org.acaro.graffiti</groupId>
	<artifactId>graffiti</artifactId>
	<packaging>jar</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>graffiti</name>
	<url>http://maven.apache.org</url>
	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-assembly-plugin</artifactId>
				<version>2.2</version>
				<configuration>
					<outputDirectory>target</outputDirectory>
					<descriptorRefs>
						<descriptorRef>jar-with-dependencies</descriptorRef>
					</descriptorRefs>
				</configuration>
				<executions>
					<execution>
						<id>make-assembly</id>
						<phase>compile</phase>
						<goals>
							<goal>single</goal>
						</goals>
					</execution>
				</executions>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-eclipse-plugin</artifactId>
				<configuration>
					<downloadSources>true</downloadSources>
					<downloadJavadocs>true</downloadJavadocs>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-shade-plugin</artifactId>
				<version>1.5</version>
				<executions>
					<execution>
						<phase>package</phase>
						<goals>
							<goal>shade</goal>
						</goals>
						<configuration>
						</configuration>
					</execution>
				</executions>
			</plugin>
		</plugins>
	</build>
	<dependencies>
		<dependency>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
			<version>4.8.2</version>
			<scope>test</scope>
		</dependency>
		<dependency>
			<groupId>ch.qos.logback</groupId>
			<artifactId>logback-classic</artifactId>
			<version>0.9.27</version>
		</dependency>
		<dependency>
			<groupId>com.google.guava</groupId>
			<artifactId>guava</artifactId>
			<version>r09</version>
		</dependency>
		<dependency>
			<groupId>org.antlr</groupId>
			<artifactId>antlr</artifactId>
			<version>3.3</version>
		</dependency>
		<dependency>
			<groupId>org.apache.giraph</groupId>
			<artifactId>giraph</artifactId>
			<version>0.70</version>
		</dependency>
		<dependency>
			<groupId>org.apache.hadoop</groupId>
			<artifactId>hadoop-core</artifactId>
			<version>0.20.204.0</version>
			<scope>provided</scope>
		</dependency>
	</dependencies>
</project>


You set your project dependecies as usual, set hadoop-core (you don't
really need it in the job jar) as <provided> and then use the
maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
package goal it will upack your dependencies and re-pack them along
with your project classes in an uberjar.

maven-shade allows you to set excludes in case of conflicts.

Easy, automatic and clean.


Hope this helps,
Claudio

On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching <aching@apache.org> wrote:
> Would be great if you can document what you did. =)
>
> Thanks,
>
> Avery
>
> On 11/8/11 3:13 PM, Claudio Martella wrote:
>>
>> Sorry guys, may bad.
>>
>> Was calling job.waitForCompletion() directly. I've been coding
>> standard mapreduce whole weekend...
>>
>> Anyway I got a solution for clean packaging of your own application
>> over giraph, and that is exactly using maven-shade-plugin. it will
>> prepare the uberjar for you.
>>
>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
>> <claudio.martella@gmail.com>  wrote:
>>>
>>> Hello list,
>>>
>>> I'm actually having troubles as well to get my application running.
>>>
>>> I've give a shot to maven-shade plugin which unpacks my dependencies
>>> and packs them all together with my classes in a new jar.
>>>
>>> I attach the hierarchy of the jar so that somebody can maybe spot
>>> what's missing, because i can't get it working. I get an identity
>>> map-reduce job with jobconf complaining about no job jar being set.
>>>
>>> Any idea?
>>>
>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<aching@apache.org>  wrote:
>>>>
>>>> Hi Gianmarco,
>>>>
>>>> You're right, most of us (to my knowledge) have been using Giraph with
>>>> an
>>>> uberjar as you've put it.  However, Jakob has been doing some work to
>>>> make
>>>> this easier.  See the below issue:
>>>>
>>>> https://issues.apache.org/jira/browse/GIRAPH-64
>>>>
>>>> If you can suggest a better approach, please add to the issue or create
>>>> a
>>>> new one if appropriate.
>>>>
>>>> Thanks,
>>>>
>>>> Avery
>>>>
>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
>>>>>
>>>>> Hi community,
>>>>>
>>>>> I was wondering what is the current best practice to package an
>>>>> application in a jar for deployment.
>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib directory
>>>>> of
>>>>> my jar, and using the -libjars option but none of them worked. It looks
>>>>> like
>>>>> the backend classloader is doing some mess and it doesn't find my own
>>>>> classes in the jar.
>>>>>
>>>>> I resorted to uncompressing the giraph-*.jar and repackaging my classes
>>>>> with it, all at the same level (an uber-fat jar), but even though it
>>>>> works
>>>>> it doesn't sound like the right approach.
>>>>>
>>>>> Any suggestions?
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Gianmarco
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>>     Claudio Martella
>>>     claudio.martella@gmail.com
>>>
>>
>>
>
>



-- 
    Claudio Martella
    claudio.martella@gmail.com

Mime
View raw message