giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arghya Kusum Das <arghyakusumdas2...@gmail.com>
Subject Re: Problem in running SSSP on hadoop-0.23.10 (insisting use LocalJob)
Date Wed, 24 Sep 2014 15:58:03 GMT
Hi,

I figured out that we need to use mapreduce.jobtracker.address instead of
mapred.job.tracker in order to get rid of this error (with
giraph1.1.0/hadoop-0.23.10). So the job is starting now.
But, Now I am getting another error right after starting the job (Something
related to zookeeper).... I tried with my own code also (which worked in
previous hadoop platform like giraph1.1.0/hadoop-0.20.203.0). But got the
same error.
2014-09-24 10:41:21,943 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir;  Ignoring.
2014-09-24 10:41:21,944 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-09-24 10:41:21,945 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-09-24 10:41:21,946 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.temp.dir;
 Ignoring.
2014-09-24 10:41:21,947 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: Log level remains at info
2014-09-24 10:41:21,981 INFO [main]
org.apache.giraph.graph.GraphTaskManager: Distributed cache is empty.
Assuming fatjar.
2014-09-24 10:41:21,981 INFO [main]
org.apache.giraph.graph.GraphTaskManager: setup: classpath @
/tmp/hadoop-yarn/staging/hadoop/.staging/job_1411572537241_0002/job.jar for
job MySimpleShortestPathsExample
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
mapred.job.id is deprecated. Instead, use mapreduce.job.id
2014-09-24 10:41:21,983 WARN [main] org.apache.hadoop.conf.Configuration:
job.local.dir is deprecated. Instead, use mapreduce.job.local.dir
2014-09-24 10:41:21,986 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Made the directory
_bsp/_defaultZkManagerDir/job_1411572537241_0002
2014-09-24 10:41:21,992 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
createCandidateStamp: Creating my filestamp
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_task/
bicstorage2.cct.lsu.edu 3
2014-09-24 10:41:22,023 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: For task 3, got file '
zkServerList_bicstorage5.cct.lsu.edu 0 ' (polling period is 3000)
2014-09-24 10:41:22,023 INFO [main] org.apache.giraph.zk.ZooKeeperManager:
getZooKeeperServerList: Found [bicstorage5.cct.lsu.edu, 0] 2 hosts in
filename 'zkServerList_bicstorage5.cct.lsu.edu 0 '
2014-09-24 10:41:22,032 WARN [main] org.apache.hadoop.mapred.YarnChild:
Exception running child : java.lang.IllegalStateException: run: Caught an
unrecoverable exception java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
        at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:101)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1264)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
        at
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:792)
        at
org.apache.giraph.graph.GraphTaskManager.startZooKeeperManager(GraphTaskManager.java:371)
        at
org.apache.giraph.graph.GraphTaskManager.setup(GraphTaskManager.java:204)
        at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:59)
        at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:89)
        ... 7 more
Caused by: java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_1411572537241_0002/_zkServer does not exist.
        at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:362)
        at
org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:751)
        ... 11 more

2014-09-24 10:41:22,037 INFO [main] org.apache.hadoop.mapred.Task: Runnning
cleanup for the task
2014-09-24 10:41:22,063 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir;  Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.cluster.local.dir;  Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2014-09-24 10:41:22,064 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
2014-09-24 10:41:22,065 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: mapreduce.cluster.temp.dir;
 Ignoring.
2014-09-24 10:41:22,068 WARN [main]
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Could not
delete hdfs://
bicstorage1.cct.lsu.edu:9010/output/_temporary/1/_temporary/attempt_1411572537241_0002_m_000003_0
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
system...
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
stopped.
2014-09-24 10:41:22,071 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
shutdown complete.


On Wed, Sep 24, 2014 at 10:24 AM, Arghya Kusum Das <
arghyakusumdas2266@gmail.com> wrote:

> Yes...it is set properly.... Still this error....
>
> On Wed, Sep 24, 2014 at 9:33 AM, xuhong zhang <xuhongnever@gmail.com>
> wrote:
>
>> Make sure set this value in your mapred-site.xml
>>
>> <property>
>>     <name>mapred.job.tracker</name>
>>     <value>***********************</value>
>>     <description>jobtracker's address</description>
>>   </property>
>>
>> On Wed, Sep 24, 2014 at 12:07 AM, Arghya Kusum Das <
>> arghyakusumdas2266@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I was trying to run the giraph  simple shortest path example (that comes
>> > with giraph package) on 3 nodes (1 namenode and 2 datanodes)
>> hadoop-0.23.10
>> > cluster. I used 7 mappers and 7 reducers run simultaneously at each
>> node in
>> > mapred-site.xml.
>> >
>> > The command I typed:
>> > $HADOOP_HOME/bin/hadoop jar
>> >
>> >
>> /usr/local/giraph-1.1.0/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-0.23.1-jar-with-dependencies.jar
>> > org.apache.giraph.GiraphRunner
>> > org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> > -vip /user/hduser/input/tiny_graph.txt -of
>> > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> > /output/shortestpaths *-w 12*
>> >
>> > When I ran with  12 workers (-w 12) the error was :
>> > 'Exception in thread "main" java.lang.IllegalArgumentException:
>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, must have
>> only
>> > one worker since only 1 task at a time!'
>> >
>> > When I change it to 1 (-w 1) the error was:
>> > 'Exception in thread "main" java.lang.IllegalArgumentException:
>> > checkLocalJobRunnerConfiguration: When using LocalJobRunner, you cannot
>> run
>> > in split master / worker mode since there is only 1 task at a time!'
>> >
>> > What am I missing?
>> >
>> > --
>> > Thanks and regards,
>> > Arghya Kusum Das
>> > (225-362-4031)
>> >
>>
>>
>>
>> --
>> Xuhong Zhang
>>
>
>
>
> --
> Thanks and regards,
> Arghya Kusum Das
> (225-362-4031)
>



-- 
Thanks and regards,
Arghya Kusum Das
(225-362-4031)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message