Return-Path: X-Original-To: apmail-giraph-user-archive@www.apache.org Delivered-To: apmail-giraph-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 16B8B10792 for ; Sat, 1 Mar 2014 16:57:43 +0000 (UTC) Received: (qmail 13295 invoked by uid 500); 1 Mar 2014 16:57:42 -0000 Delivered-To: apmail-giraph-user-archive@giraph.apache.org Received: (qmail 13241 invoked by uid 500); 1 Mar 2014 16:57:41 -0000 Mailing-List: contact user-help@giraph.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@giraph.apache.org Delivered-To: mailing list user@giraph.apache.org Received: (qmail 13233 invoked by uid 99); 1 Mar 2014 16:57:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Mar 2014 16:57:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rao.jyoti26yadav@gmail.com designates 209.85.212.182 as permitted sender) Received: from [209.85.212.182] (HELO mail-wi0-f182.google.com) (209.85.212.182) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 01 Mar 2014 16:57:35 +0000 Received: by mail-wi0-f182.google.com with SMTP id f8so1710815wiw.15 for ; Sat, 01 Mar 2014 08:57:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Sx8rHCqm27RyVFaN8A80rCl89oMO5Vgz8oNdSF1ONK8=; b=eKNSQNF/wQ4wqkDMXvw95FpPGUGJFtBEqyFoxWePtcuzHxlJnlC7TFlwf39wvd8odi zzf8a8n9bgahMw0b2XGVDwltW/zl6kVQfM9M4UgXleawNZxbBYDuPck4j3yXcF/1f7Uw pkivu51dpVKYicxiTLk4YRTLENXyW1+a7Y8Wl6MeH+jOfyCLumNMLbMPzfp/zacCmocN H6rmjZCn9kOIlGWaa0YQAVwSc8a5PECq/ZlFFVKSG1+PX4eDpI9rubcLjovwPtj1MV9j HXsODotzggllvioCKRkcpu5xwjD8/XwTPu6sHZcqBSsnTLDLrp7uczOlyDH/r1nsRMKc bD7w== MIME-Version: 1.0 X-Received: by 10.180.205.130 with SMTP id lg2mr8105253wic.59.1393693034777; Sat, 01 Mar 2014 08:57:14 -0800 (PST) Received: by 10.194.3.35 with HTTP; Sat, 1 Mar 2014 08:57:14 -0800 (PST) In-Reply-To: References: <531194BF.70002@apache.org> Date: Sat, 1 Mar 2014 22:27:14 +0530 Message-ID: Subject: Re: Sample data for Single Source shortest path From: Jyoti Yadav To: user@giraph.apache.org Content-Type: multipart/alternative; boundary=001a11c26ac043958d04f38e7267 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c26ac043958d04f38e7267 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Mirko.. Thanks for your reply.. All MapReduce programs are running fine on this system. And it is yarn setup. Please guide me how to bulid giraph with this hadoop version..Should I need to install external zookeeper also.? Thanks in advance.. Jyoti On Sat, Mar 1, 2014 at 6:31 PM, Mirko K=E4mpf wr= ote: > Hello, > > if you build Giraph for hadoop 0.20.... the same jars will not work for > hadoop version 2.2.0. > Right now I build the profile -Phadoop_2 from curren the 1.1. branch in > the git repo. > > How many nodes (physical servers or VMs) do you run on your 64 core syste= m? > What distro of Hadoop are working with? and is it a MRv1 or MRV2 (YARN) > setup? > > Is your MapReduce system working properly ... can you run TerraSort for > example? > > Cheers, > Mirko > > > > On Sat, Mar 1, 2014 at 4:15 AM, Jyoti Yadav w= rote: > >> Anyone please reply ..Is it portability problem??.. Does giraph has any >> issues with Hadoop 2.2.0?? >> >> Do I need to build Giraph on the new system ?? >> >> Thanks >> >> >> >> On Sat, Mar 1, 2014 at 2:28 PM, Jyoti Yadav = wrote: >> >>> Hi Sebastian.. >>> Thanks for the links given for big graphs.. >>> >>> Actually I want to tell you something about problem i am facing. >>> >>> Initially I was working with *hadoop 0.20.203* . I build Giraph there.. >>> it was running fine. >>> >>> Now to test very big graph related problem and to compare the >>> performance , I moved to new system which is of 64 cores and 512 GB me= mory >>> and 3 TB storage. Instead to building Giraph in the new system, I jus= t >>> copied Giraph folder from my previous system to this new system. In thi= s >>> new system *hadoop version 2.2..0 * . I tried to execute >>> SimpleSourceShortestPath algo on sample data set. It is throwing follow= ing >>> exception. >>> >>> I gave following command to execute the job. >>> >>> hadoop jar >>> /home/abcd2014/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAP= SHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar >>> org.apache.giraph.GiraphRunner -Dgiraph.SplitMasterWorker=3Dtrue >>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>> -vip /user/abcd2014/giraph_input/tiny_graph.txt -vof >>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>> /user/abcd2014/output2/shortestpaths -w 1 >>> >>> >>> >>> 14/03/01 12:44:46 INFO utils.ConfigurationUtils: No edge input format >>> specified. Ensure your InputFormat does not require one. >>> 14/03/01 12:44:46 INFO utils.ConfigurationUtils: No edge output format >>> specified. Ensure your OutputFormat does not require one. >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapreduce.job.counters.limit is deprecated. Instead, use >>> mapreduce.job.counters.max >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memo= ry.mb >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapred.job.reduce.memory.mb is deprecated. Instead, use >>> mapreduce.reduce.memory.mb >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapred.map.tasks.speculative.execution is deprecated. Instead, use >>> mapreduce.map.speculative >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapreduce.user.classpath.first is deprecated. Instead, use >>> mapreduce.job.user.classpath.first >>> 14/03/01 12:44:46 INFO Configuration.deprecation: >>> mapred.map.max.attempts is deprecated. Instead, use >>> mapreduce.map.maxattempts >>> 14/03/01 12:44:46 INFO job.GiraphJob: run: Since checkpointing is >>> disabled (default), do not allow any task retries (setting >>> mapred.map.max.attempts =3D 0, old value =3D 4) >>> 14/03/01 12:44:46 INFO Configuration.deprecation: mapred.job.tracker is >>> deprecated. Instead, use mapreduce.jobtracker.address >>> >>> *Exception in thread "main" java.lang.IllegalArgumentException: >>> checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot= run >>> in split master / worker mode since there is only 1 task at a time! * >>> at >>> org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfiguration(Giraph= Job.java:165) >>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:233) >>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja= va:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso= rImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212) >>> >>> >>> >>> Would you suggest me something to fix this...If you need any details >>> further,please let me know... >>> >>> Thanks & Regards >>> >>> Jyoti >>> >>> >>> >>> >>> On Sat, Mar 1, 2014 at 1:35 PM, Sebastian Schelter wrot= e: >>> >>>> Hi Jyoti, >>>> >>>> You can find a couple of very large graphs in KONECT [1] and on the >>>> website of the laboratory for web algorithmics from the University of = Milan >>>> [2]. You will probably have to convert them to an appropriate format f= or >>>> Giraph. >>>> >>>> Best, >>>> Sebastian >>>> >>>> [1] http://konect.uni-koblenz.de/ >>>> [2] http://law.di.unimi.it/datasets.php >>>> >>>> >>>> On 03/01/2014 05:22 AM, Jyoti Yadav wrote: >>>> >>>>> Hi folks.. >>>>> >>>>> I got new system which is of 64 cores and 512 GB memory and 3 TB >>>>> storage.I want to test the performance of Giraph on this system. >>>>> Would anyone provide me the link for very large graph so that I ca= n >>>>> execute Single Source Shortest Path Example. For this algo to run gra= ph >>>>> should be weighted graph. and to feed it into the Giraph -input >>>>> format is >>>>> JsonLongDoubleFloatDouble >>>>> >>>>> Thanks in advance... >>>>> With Regards >>>>> >>>>> Jyoti >>>>> >>>>> >>>> >>> >> > > > -- > -- > Mirko K=E4mpf > > *Trainer* @ Cloudera > > tel: +49 *176 20 63 51 99* > skype: *kamir1604* > mirko@cloudera.com > > --001a11c26ac043958d04f38e7267 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi Mirko..

Thanks fo= r your reply.. All MapReduce programs are running fine on this system.
<= /div>=A0And it=A0 is yarn setup.

Please guide me how to bulid = giraph with this hadoop version..Should I need to install external zookeepe= r also.?

Thanks in advance..

Jyoti


On Sat, Mar 1, 2014 at 6:31 PM= , Mirko K=E4mpf <mirko.kaempf@cloudera.com> wrote:
Hello,

if you build Giraph for hadoop 0.20.... the same jar= s will not work for hadoop version 2.2.0.
Right now I build the profile -Phadoop_2 from curren the 1.1. branch = in the git repo.

How many nodes (physical servers or VMs) do you run on your 64 co= re system?
What distro of Hadoop are working with? and is it a MRv= 1 or MRV2 (YARN) setup?

Is your MapReduce system working prope= rly ... can you run TerraSort for example?

Cheers,
Mirko
=A0


On Sat, Mar= 1, 2014 at 4:15 AM, Jyoti Yadav <rao.jyoti26yadav@gmail.com&= gt; wrote:
Anyone pleas= e reply ..Is it portability problem??.. Does giraph has any issues with Had= oop 2.2.0??

Do I need to build Giraph on the new system ??

Thanks



On Sat, Mar 1, 2014 at 2:28 PM, Jyoti Yadav <rao.jyoti26yadav@= gmail.com> wrote:
Hi Sebastian..
Thanks for the links given=A0 for big graph= s..

Actually I want to tell you something about problem i am facing.<= br>
Initially I was working with hadoop 0.20.203 . I build = Giraph there.. it was running fine.

Now=A0 to test very big graph related problem and to compare the = performance , I moved to new system which is=A0 of 64 cores and 512 GB memo= ry and=A0 3 TB storage.=A0 Instead to building Giraph in the new system, I = just copied Giraph folder from my previous system to this new system. In th= is new system hadoop version 2.2..0=A0 . I tried to execute SimpleSo= urceShortestPath algo on sample data set. It is throwing following exceptio= n.

I gave following command to execute the job.

hadoop jar /home/ab= cd2014/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-had= oop-0.20.203.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner -Dg= iraph.SplitMasterWorker=3Dtrue=A0 org.apache.giraph.examples.SimpleShortest= PathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubl= eVertexInputFormat -vip /user/abcd2014/giraph_input/tiny_graph.txt -vof org= .apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/abcd2014/ou= tput2/shortestpaths -w 1



14/03/01 12:44:46 INFO utils.ConfigurationUtils: No edge input = format specified. Ensure your InputFormat does not require one.
14/03/01= 12:44:46 INFO utils.ConfigurationUtils: No edge output format specified. E= nsure your OutputFormat does not require one.
14/03/01 12:44:46 INFO Configuration.deprecation: mapreduce.job.counters.li= mit is deprecated. Instead, use mapreduce.job.counters.max
14/03/01 12:4= 4:46 INFO Configuration.deprecation: mapred.job.map.memory.mb is deprecated= . Instead, use mapreduce.map.memory.mb
14/03/01 12:44:46 INFO Configuration.deprecation: mapred.job.reduce.memory.= mb is deprecated. Instead, use mapreduce.reduce.memory.mb
14/03/01 12:44= :46 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution = is deprecated. Instead, use mapreduce.map.speculative
14/03/01 12:44:46 INFO Configuration.deprecation: mapreduce.user.classpath.= first is deprecated. Instead, use mapreduce.job.user.classpath.first
14/= 03/01 12:44:46 INFO Configuration.deprecation: mapred.map.max.attempts is d= eprecated. Instead, use mapreduce.map.maxattempts
14/03/01 12:44:46 INFO job.GiraphJob: run: Since checkpointing is disabled = (default), do not allow any task retries (setting mapred.map.max.attempts = =3D 0, old value =3D 4)
14/03/01 12:44:46 INFO Configuration.deprecation= : mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.addre= ss
Exception in thread "main" java.lang.IllegalArgumentException:= checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot ru= n in split master / worker mode since there is only 1 task at a time!
=A0=A0=A0 at org.apache.giraph.job.GiraphJob.checkLocalJobRunnerConfigu= ration(GiraphJob.java:165)
=A0=A0=A0 at org.apache.giraph.job.GiraphJob.= run(GiraphJob.java:233)
=A0=A0=A0 at org.apache.giraph.GiraphRunner.run(= GiraphRunner.java:94)
=A0=A0=A0 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
= =A0=A0=A0 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
= =A0=A0=A0 at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
= =A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) =A0=A0=A0 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess= orImpl.java:57)
=A0=A0=A0 at sun.reflect.DelegatingMethodAccessorImpl.in= voke(DelegatingMethodAccessorImpl.java:43)
=A0=A0=A0 at java.lang.reflec= t.Method.invoke(Method.java:606)
=A0=A0=A0 at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


Would you suggest me something to fix this...If you need any det= ails further,please let me know...

Thanks & Regards<= font color=3D"#888888">

Jyoti


=


On Sat, = Mar 1, 2014 at 1:35 PM, Sebastian Schelter <ssc@apache.org> wro= te:
Hi Jyoti,

You can find a couple of very large graphs in KONECT [1] and on the website= of the laboratory for web algorithmics from the University of Milan [2]. Y= ou will probably have to convert them to an appropriate format for Giraph.<= br>
Best,
Sebastian

[1] http://kone= ct.uni-koblenz.de/
[2] http:= //law.di.unimi.it/datasets.php


On 03/01/2014 05:22 AM, Jyoti Yadav wrote:
Hi folks..

I got new system which is =A0of 64 cores and 512 GB memory and =A03 TB
storage.I want to test the performance of Giraph on this system.
=A0 Would anyone provide me the link for very large graph =A0so that I can<= br> execute Single Source Shortest Path Example. For this algo to run graph
should be weighted graph. and =A0to feed it into the Giraph -input format i= s
JsonLongDoubleFloatDouble

Thanks in advance...
With Regards

Jyoti







--
--=A0
Mirko K=E4mpf=

Trainer=A0@ Cloudera

tel: +49=A0= 176 20 63 51 99
skype:=A0kamir1604
--001a11c26ac043958d04f38e7267--