giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mirko Kämpf <mirko.kae...@cloudera.com>
Subject Re: Giraph and Hadoop Cloudera CDH4.1.1
Date Sat, 10 Nov 2012 13:24:01 GMT
Hello all,

I created a now pom.xml file, which allows compiling Giraph with the new
CDH 4.1.2 repository.

I attached it to this email.



But there are still some problems. I can not start the examples, because I
do not have
demo data and I do not jet know, what formats have to be used. Has anybody
some test data files to share for a public tutorial?

Currently I prepare a tutorial on working with Giraph, but until now I had
to find out, that
the wiki content is quite outdated. The ShortestPathExample looks nice, but
without
some hints I can not start. Did I miss some documentation?
If not, I will create it, but therefor I would like to ask for some help.

My plan is, to create some demo data files and some Demo-Apps comparable to
the benchmark classes but based
on real data. I have some correlation obtained from wikipedia user data,
but until now I can not analuse that data with Giraph.

I get the following error ( at the end of this mail ) if I want to start
the PageRankBenchmark.

Has anybody an idea, what I can do to solve this problem?

Thanks a lot!

Best wishes
Mirko


[kamir@master2 trunk]$ *hadoop jar
giraph/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-cdh4.2.1-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -s 5 -v -w 5 -V 3 -e
80                                      *

12/11/10 14:12:35 INFO benchmark.PageRankBenchmark: Using class
org.apache.giraph.benchmark.EdgeListVertexPageRankBenchmark

12/11/10 14:12:36 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/11/10 14:12:36 WARN bsp.BspOutputFormat: checkOutputSpecs:
ImmutableOutputCommiter will not check
anything
12/11/10 14:12:37 INFO mapred.JobClient: Running job:
job_201211101312_0007

12/11/10 14:12:38 INFO mapred.JobClient:  map 0% reduce
0%

12/11/10 14:12:49 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_0, Status :
FAILED
*Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
*
attempt_201211101312_0007_m_000001_0: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000001_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000001_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000001_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:49 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_0, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;

attempt_201211101312_0007_m_000003_0: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000003_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000003_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000003_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:52 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_0, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;

attempt_201211101312_0007_m_000002_0: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000002_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000002_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000002_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:53 INFO mapred.JobClient:  map 16% reduce
0%

12/11/10 14:12:53 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;

attempt_201211101312_0007_m_000003_1: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000003_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000003_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000003_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:54 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;

attempt_201211101312_0007_m_000001_1: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000001_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000001_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]

attempt_201211101312_0007_m_000001_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:58 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;

attempt_201211101312_0007_m_000002_1: SLF4J: Class path contains multiple
SLF4J
bindings.

attempt_201211101312_0007_m_000002_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:12:59 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000003_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000003_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:12:59 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000001_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000001_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:13:04 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000002_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000002_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:13:07 INFO mapred.JobClient: Job complete: job_201211101312_0007
12/11/10 14:13:07 INFO mapred.JobClient: Counters: 6
12/11/10 14:13:07 INFO mapred.JobClient:   Job Counters
12/11/10 14:13:07 INFO mapred.JobClient:     Failed map tasks=1
12/11/10 14:13:07 INFO mapred.JobClient:     Launched map tasks=13
12/11/10 14:13:07 INFO mapred.JobClient:     Total time spent by all maps
in occupied slots (ms)=51718
12/11/10 14:13:07 INFO mapred.JobClient:     Total time spent by all
reduces in occupied slots (ms)=0
12/11/10 14:13:07 INFO mapred.JobClient:     Total time spent by all maps
waiting after reserving slots (ms)=0
12/11/10 14:13:07 INFO mapred.JobClient:     Total time spent by all
reduces waiting after reserving slots (ms)=0
[kamir@master2 trunk]$ hadoop fs -ls _bsp/
Found 1 items
drwxr-xr-x   - kamir kamir          0 2012-11-10 14:12
_bsp/_defaultZkManagerDir
[kamir@master2 trunk]$ hadoop fs -ls _bsp//_defaultZkManagerDir

Found 1 items
drwxr-xr-x   - kamir kamir          0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007
[kamir@master2 trunk]$
[kamir@master2 trunk]$ hadoop fs -ls
_bsp/_defaultZkManagerDir/job_201211101312_0007
Found 3 items
drwxr-xr-x   - kamir kamir          0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/_task
drwxr-xr-x   - kamir kamir          0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/_zkServer
-rw-r--r--   3 kamir kamir          0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/zkServerList_master2 0
[kamir@master2 trunk]$




On Wed, Nov 7, 2012 at 8:31 PM, Avery Ching <aching@apache.org> wrote:

> Hi Franco,
>
> I haven't tried.  Please let us know your experiences.
>
> Avery
>
>
> On 11/7/12 2:30 AM, Franco Maria Nardini wrote:
>
>> Dear all,
>>
>> I am trying to run Giraph on Hadoop Cloudera CDH4.1.1. I know that it is
>> not one of the Hadoop distributions reported to work with Giraph. However,
>> I was wondering if you guys try to do that? What is your experience on that?
>>
>> Thanks a lot in advance.
>>
>> Best,
>>
>> Franco Maria
>>
>
>


-- 
-- 
Mirko Kämpf

*Trainer* @ Cloudera

tel: +49 *176 20 63 51 99*
skype: *kamir1604*
mirko@cloudera.com

Mime
View raw message