giraph-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matthew Laird <lai...@sfu.ca>
Subject Re: Giraph & yarn build
Date Mon, 07 Oct 2013 02:53:37 GMT
Hmm, I might do that, thanks.  But I'm a little unclear, is that 
actually a different version or do I just give a different profile? (ie. 
-Dhadoop.version=0.20.203.0 or such)  And will that work against 2.0.5 
or do I need to install a different hadoop version as well?

I originally was following the quick start guide but ran in to a problem 
with LocalJobRunner, I'm running a hadoop cluster with split master and 
worker nodes, so trying to run the examples -w 1 errored about this 
setup, but -w 4 errored that LocalJobRunner can't run multiple jobs or 
some wording like that (it was about 8 hours ago, I've forgotten :)

So my next query, I couldn't find any answers about running the examples 
on a split master-slave cluster (I hope that's the correct terminology, 
I'm still getting the hang of hadoop, I'm trying to cluster a large set 
of orthologous genes across about 2000 bacterial species).

Thanks.

On 06/10/13 19:16, Milinda Pathirage wrote:
> I got the build working just by following above steps I mentioned.
>
> May be best option is to try giraph version which uses map-reduce API
> instead of pure YARN. Also there aren't any scripts which can use to
> submit jobs to YARN cluster (for pure YARN based implementation). So
> going with giraph version which is based on map-reduce API will be
> easy for you.
>
> Thanks
> Milinda
>
> On Sun, Oct 6, 2013 at 10:04 PM, Matthew Laird <lairdm@sfu.ca> wrote:
>> Hmm, is there a recommendation on a particular version which might be good
>> for building/testing against Hadoop 2.0.5?  I realize I'm playing with the
>> bleeding edge here...
>>
>> I threw a try/catch around   public static void printHaltInfo() just like
>> private static boolean waitForHaltInfo() has.  However next I had issues
>> with the tests, I'm guessing because I'm running the hadoop head node on
>> this machine, it was a socket bind error, so I threw in a -DskipTests.
>>
>> Next I ran in to a org.apache.rat issue, I added an exclude for *.txt,
>> *.xml, and *.py.  Unfortunately next was:
>>
>> [INFO] --- findbugs-maven-plugin:2.5.1:check (default) @ giraph-core ---
>> [INFO] BugInstance size is 2
>> [INFO] Error size is 0
>> [INFO] Total bugs: 2
>> [INFO]
>> org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler.uncaughtException(Thread,
>> Throwable) invokes System.exit(...), which shuts down the entire virtual
>> machine ["org.apache.giraph.yarn.GiraphYarnTask$OverrideExceptionHandler"]
>> At GiraphYarnTask.java:[lines 171-178]
>> [INFO] exceptional return value of java.io.File.delete() ignored in
>> org.apache.giraph.yarn.YarnUtils.exportGiraphConfiguration(GiraphConfiguration,
>> ApplicationId) ["org.apache.giraph.yarn.YarnUtils"] At YarnUtils.java:[lines
>> 47-240]
>>
>> I guess this is what I get for playing with the trunk. :)  But, what version
>> was it that is mentioned in the previous mailing list threads that you can
>> apply the patch to and get it compiled?
>>
>> Thanks!
>>
>>
>> On 06/10/13 18:19, Matthew Laird wrote:
>>> Thanks for the reply, so I just grabbed that I assume is the latest
>>> version, the snapshot link from git:
>>>
>>>
>>> https://git-wip-us.apache.org/repos/asf?p=giraph.git;a=tree;h=refs/heads/trunk;hb=trunk
>>>
>>> And ran the command:
>>>
>>> mvn -Dhadoop.version=2.0.5-alpha -Phadoop_yarn clean install
>>>
>>> The error I receive in this situation is:
>>>
>>> [INFO] 2 errors
>>> [INFO] -------------------------------------------------------------
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] Apache Giraph Parent .............................. SUCCESS
>>> [1.593s]
>>> [INFO] Apache Giraph Core ................................ FAILURE
>>> [23.522s]
>>> [INFO] Apache Giraph Examples ............................ SKIPPED
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] BUILD FAILURE
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [INFO] Total time: 25.574s
>>> [INFO] Finished at: Sun Oct 06 18:17:21 PDT 2013
>>> [INFO] Final Memory: 37M/692M
>>> [INFO]
>>> ------------------------------------------------------------------------
>>> [ERROR] Failed to execute goal
>>> org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile)
>>> on project giraph-core: Compilation failure: Compilation failure:
>>> [ERROR]
>>> /opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[78,49]
>>> unreported exception java.lang.InterruptedException; must be caught or
>>> declared to be thrown
>>> [ERROR]
>>> /opt/giraph/giraph-core/target/munged/main/org/apache/giraph/job/HaltApplicationUtils.java:[81,50]
>>> unreported exception java.lang.InterruptedException; must be caught or
>>> declared to be thrown
>>>
>>> That looks like a bug in the code?
>>>
>>> Thanks.
>>>
>>> On 06/10/13 18:13, Milinda Pathirage wrote:
>>>> AFAIK, you need to specify hadoop.version system property. So command
>>>> to build giraph with YARN (2.0.5) should be something like following.
>>>>
>>>> mvn -Phadoop_yarn -Dhadoop.version=2.0.5-alpha clean install
>>>>
>>>> Hope this helps.
>>>>
>>>> Thanks
>>>> Milinda
>>>>
>>>>
>>>> On Sun, Oct 6, 2013 at 9:04 PM, Matthew Laird <lairdm@sfu.ca> wrote:
>>>>> Further, I let it go and after two hours I see the file:
>>>>>
>>>>>
>>>>> /giraph-core/target/munged/main/org/apache/giraph/yarn/GiraphYarnTask.java.rej
>>>>>
>>>>> is now 295GB in size.  I'm not sure what it's stuck doing after applying
>>>>> this patch...
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On 06/10/13 16:13, Matthew Laird wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I'm trying to build giraph against Hadoop 2.0.5, and I've been browsing
>>>>>> the mailing list archives but am still having an issue. I found the
>>>>>> previous
>>>>>> thread where Eli points to
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/GIRAPH-688
>>>>>>
>>>>>> I've grabbed the patch and applied it against a clone from git
>>>>>> (http://git-wip-us.apache.org/repos/asf/giraph.git, is that the correct
>>>>>> place to get the latest trunk?)  Just like the other fellow asking
a
>>>>>> few
>>>>>> months ago, patch complains the patch has been applied before.
>>>>>>
>>>>>> Once applied I run the suggested build command:
>>>>>>
>>>>>> mvn -U -Phadoop_yarn clean install
>>>>>>
>>>>>> It reaches
>>>>>>
>>>>>> [INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
>>>>>>
>>>>>> And so far it's been sitting there about 20 minutes at this step,
in
>>>>>> another window I can see it's working hard doing something, but I'm
>>>>>> assuming
>>>>>> it's stuck doing... I'm not sure what.
>>>>>>
>>>>>> Without the patch I receive the same error the other fellow was having
>>>>>> about dependancies for hadoop not being satisfied.
>>>>>>
>>>>>> Any advice on what might be going on, or is there a better place
to get
>>>>>> a
>>>>>> snapshot that has the needed patches applied? (I see a lot of commits
>>>>>> since
>>>>>> the original thread)
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>
>
>


Mime
View raw message