giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <ach...@apache.org>
Subject Re: Packaging a Giraph application in a jar
Date Fri, 23 Dec 2011 17:19:37 GMT
Hi Christoph,

Is your svn up to date?  I see bin/giraph in my svn checkout.

I am not sure that we have totally figured out the uber-fat-jar 
discussion.  It's still evolving.  Here's some quick possibilities you 
can try out though.

1)  Add your application to the Giraph code and compile the uber fat jar

2)  mvn install your giraph jar into your local maven repo and then 
include it as a dependency for your applications.  Then you can either 
include the application jar (uber jar approach) or you can put the jar 
in the library path for your hadoop cluster to reduce your application 
jar size.

Hope that helps!

Avery

On 12/23/11 5:11 AM, "Christoph Böhm" wrote:
> Hi all,
>
> first, I cannot find this node runner "bin/giraph" mentioned in GIRAPH-64 in the 0.7
trunk and, second, I'm not quite sure about the essence from that ueber-fat-jar discussion.
> Is there a better/best practice way to run my implementation (vertex, io, of, etc) other
then the ueber jar?
>
> What am I missing?
> Thanks.
> Christoph
>
>
>
>
> --- original msg ---
> Gianmarco-
>     GIRAPH-64 has been committed, so it should be much easier to treat
> Giraph as a library and link your vertices against it.  Try it out and
> see if it meets your needs.  If not, be sure to open a JIRA so we can
> improve it.
>
> Thanks,
> Jakob
>
>
> On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
> <gdfm@apache.org>  wrote:
>> Nice!
>> It looks cleaner than my ant+ivy hack :)
>> Thanks for sharing,
>> --
>> Gianmarco
>>
>>
>> On Thu, Nov 10, 2011 at 16:18, Claudio Martella<claudio.martella@gmail.com>
>> wrote:
>>> Ok,
>>>
>>> quite basically this is my pom.xml:
>>>
>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>> http://maven.apache.org/maven-v4_0_0.xsd">
>>> � � � �<modelVersion>4.0.0</modelVersion>
>>> � � � �<groupId>org.acaro.graffiti</groupId>
>>> � � � �<artifactId>graffiti</artifactId>
>>> � � � �<packaging>jar</packaging>
>>> � � � �<version>1.0-SNAPSHOT</version>
>>> � � � �<name>graffiti</name>
>>> � � � �<url>http://maven.apache.org</url>
>>> � � � �<build>
>>> � � � � � � � �<plugins>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>>
>>> �<artifactId>maven-assembly-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<version>2.2</version>
>>> � � � � � � � � � � � � � � � �<configuration>
>>>
>>> �<outputDirectory>target</outputDirectory>
>>> � � � � � � � � � � � � � � � � � � � �<descriptorRefs>
>>>
>>> �<descriptorRef>jar-with-dependencies</descriptorRef>
>>> � � � � � � � � � � � � � � � � � � � �</descriptorRefs>
>>> � � � � � � � � � � � � � � � �</configuration>
>>> � � � � � � � � � � � � � � � �<executions>
>>> � � � � � � � � � � � � � � � � � � � �<execution>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<id>make-assembly</id>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<phase>compile</phase>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<goals>
>>> � � � � � � � � � � � � � � � � � � � �
� � � � � � � �<goal>single</goal>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �</goals>
>>> � � � � � � � � � � � � � � � � � � � �</execution>
>>> � � � � � � � � � � � � � � � �</executions>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>>
>>> �<artifactId>maven-eclipse-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<configuration>
>>>
>>> �<downloadSources>true</downloadSources>
>>>
>>> �<downloadJavadocs>true</downloadJavadocs>
>>> � � � � � � � � � � � � � � � �</configuration>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>> � � � � � � � � � � � � � � � �<artifactId>maven-shade-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<version>1.5</version>
>>> � � � � � � � � � � � � � � � �<executions>
>>> � � � � � � � � � � � � � � � � � � � �<execution>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<phase>package</phase>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<goals>
>>> � � � � � � � � � � � � � � � � � � � �
� � � � � � � �<goal>shade</goal>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �</goals>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �<configuration>
>>> � � � � � � � � � � � � � � � � � � � �
� � � �</configuration>
>>> � � � � � � � � � � � � � � � � � � � �</execution>
>>> � � � � � � � � � � � � � � � �</executions>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � �</plugins>
>>> � � � �</build>
>>> � � � �<dependencies>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>junit</groupId>
>>> � � � � � � � � � � � �<artifactId>junit</artifactId>
>>> � � � � � � � � � � � �<version>4.8.2</version>
>>> � � � � � � � � � � � �<scope>test</scope>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>ch.qos.logback</groupId>
>>> � � � � � � � � � � � �<artifactId>logback-classic</artifactId>
>>> � � � � � � � � � � � �<version>0.9.27</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>com.google.guava</groupId>
>>> � � � � � � � � � � � �<artifactId>guava</artifactId>
>>> � � � � � � � � � � � �<version>r09</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.antlr</groupId>
>>> � � � � � � � � � � � �<artifactId>antlr</artifactId>
>>> � � � � � � � � � � � �<version>3.3</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.apache.giraph</groupId>
>>> � � � � � � � � � � � �<artifactId>giraph</artifactId>
>>> � � � � � � � � � � � �<version>0.70</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.apache.hadoop</groupId>
>>> � � � � � � � � � � � �<artifactId>hadoop-core</artifactId>
>>> � � � � � � � � � � � �<version>0.20.204.0</version>
>>> � � � � � � � � � � � �<scope>provided</scope>
>>> � � � � � � � �</dependency>
>>> � � � �</dependencies>
>>> </project>
>>>
>>>
>>> You set your project dependecies as usual, set hadoop-core (you don't
>>> really need it in the job jar) as<provided>  and then use the
>>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
>>> package goal it will upack your dependencies and re-pack them along
>>> with your project classes in an uberjar.
>>>
>>> maven-shade allows you to set excludes in case of conflicts.
>>>
>>> Easy, automatic and clean.
>>>
>>>
>>> Hope this helps,
>>> Claudio
>>>
>>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<aching@apache.org>  wrote:
>>>> Would be great if you can document what you did. =)
>>>>
>>>> Thanks,
>>>>
>>>> Avery
>>>>
>>>> On 11/8/11 3:13 PM, Claudio Martella wrote:
>>>>> Sorry guys, may bad.
>>>>>
>>>>> Was calling job.waitForCompletion() directly. I've been coding
>>>>> standard mapreduce whole weekend...
>>>>>
>>>>> Anyway I got a solution for clean packaging of your own application
>>>>> over giraph, and that is exactly using maven-shade-plugin. it will
>>>>> prepare the uberjar for you.
>>>>>
>>>>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
>>>>> <claudio.martella@gmail.com>  �wrote:
>>>>>> Hello list,
>>>>>>
>>>>>> I'm actually having troubles as well to get my application running.
>>>>>>
>>>>>> I've give a shot to maven-shade plugin which unpacks my dependencies
>>>>>> and packs them all together with my classes in a new jar.
>>>>>>
>>>>>> I attach the hierarchy of the jar so that somebody can maybe spot
>>>>>> what's missing, because i can't get it working. I get an identity
>>>>>> map-reduce job with jobconf complaining about no job jar being set.
>>>>>>
>>>>>> Any idea?
>>>>>>
>>>>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<aching@apache.org>
 �wrote:
>>>>>>> Hi Gianmarco,
>>>>>>>
>>>>>>> You're right, most of us (to my knowledge) have been using Giraph
>>>>>>> with
>>>>>>> an
>>>>>>> uberjar as you've put it. �However, Jakob has been doing some
work to
>>>>>>> make
>>>>>>> this easier. �See the below issue:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-64
>>>>>>>
>>>>>>> If you can suggest a better approach, please add to the issue
or
>>>>>>> create
>>>>>>> a
>>>>>>> new one if appropriate.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Avery
>>>>>>>
>>>>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
>>>>>>>> Hi community,
>>>>>>>>
>>>>>>>> I was wondering what is the current best practice to package
an
>>>>>>>> application in a jar for deployment.
>>>>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib
>>>>>>>> directory
>>>>>>>> of
>>>>>>>> my jar, and using the -libjars option but none of them worked.
It
>>>>>>>> looks
>>>>>>>> like
>>>>>>>> the backend classloader is doing some mess and it doesn't
find my
>>>>>>>> own
>>>>>>>> classes in the jar.
>>>>>>>>
>>>>>>>> I resorted to uncompressing the giraph-*.jar and repackaging
my
>>>>>>>> classes
>>>>>>>> with it, all at the same level (an uber-fat jar), but even
though it
>>>>>>>> works
>>>>>>>> it doesn't sound like the right approach.
>>>>>>>>
>>>>>>>> Any suggestions?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> --
>>>>>>>> Gianmarco
>>>>>>>>
>>>>>>>>


Mime
View raw message