hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yingyi Bu <buyin...@gmail.com>
Subject Re: Error partitioning the input path
Date Fri, 27 Sep 2013 10:19:56 GMT
BTW, in this job,  I set the number of tasks to be 240,  since I have 60
groomservers and each server has 4 cores.
I also changed bsp.tasks.maximum to be 4 and bsp.max.tasks.per.job to be
300 in hama-default.xml.
Why there are only 133 tasks, instead of 240?

Thanks!

Yingyi



On Fri, Sep 27, 2013 at 3:06 AM, Yingyi Bu <buyingyi@gmail.com> wrote:

> It fails again:
>
> 13/09/27 09:49:12 INFO mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 13/09/27 09:49:12 INFO bsp.FileInputFormat: Total input paths to process :
> 2
> 13/09/27 09:49:12 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 13/09/27 09:49:12 WARN snappy.LoadSnappy: Snappy native library not loaded
> 13/09/27 09:49:12 INFO bsp.FileInputFormat: Total input paths to process :
> 2
> 13/09/27 09:49:13 INFO bsp.BSPJobClient: Running job: job_201309270937_0004
> 13/09/27 09:49:16 INFO bsp.BSPJobClient: Current supersteps number: 0
> 13/09/27 09:53:09 INFO bsp.BSPJobClient: Job failed.
> 13/09/27 09:53:09 ERROR bsp.BSPJobClient: Error partitioning the input
> path.
>
> java.io.IOException: Runtime partition failed for the job.
>         at
> org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:465)
>         at
> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:333)
>         at
> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
>         at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:229)
>         at org.apache.hama.graph.GraphJob.submit(GraphJob.java:203)
>         at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:236)
>         at org.apache.hama.examples.PageRank2.main(PageRank2.java:162)
>
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>         at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>         at
> org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:43)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.hama.util.RunJar.main(RunJar.java:146)
>
> I set the HAMA_LOG_DIR in conf/hama-env.sh to avoid logs going to NFS.
> Just write a script to dump all tasklogs for the job.
> Attached is the logs.  But I cannot find any clue from the logs.
>
> Any help is appreciated. Thanks!!
>
> Yingyi
>
>
>
> On Fri, Sep 27, 2013 at 2:44 AM, Edward J. Yoon <edwardyoon@apache.org>wrote:
>
>> Hi,
>>
>> The task logs are in ${HAMA_HOME}/logs/tasklogs/.
>>
>>
>>
>> On Fri, Sep 27, 2013 at 6:12 PM, Yingyi Bu <buyingyi@gmail.com> wrote:
>> > All the groomsevers' tasklogs?
>> >
>> > Yingyi
>> >
>> >
>> > On Fri, Sep 27, 2013 at 2:11 AM, Edward J. Yoon <edwardyoon@apache.org
>> >wrote:
>> >
>> >> Could you please check the tasklogs?
>> >>
>> >> On Fri, Sep 27, 2013 at 6:03 PM, Yingyi Bu <buyingyi@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I got a runtime partition exception as follows:
>> >> >
>> >> > 13/09/27 08:52:41 INFO mortbay.log: Logging to
>> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>> >> > org.mortbay.log.Slf4jLog
>> >> > 13/09/27 08:52:42 INFO bsp.FileInputFormat: Total input paths to
>> process
>> >> : 2
>> >> > 13/09/27 08:52:42 WARN util.NativeCodeLoader: Unable to load
>> >> native-hadoop
>> >> > library for your platform... using builtin-java classes where
>> applicable
>> >> > 13/09/27 08:52:42 WARN snappy.LoadSnappy: Snappy native library not
>> >> loaded
>> >> > 13/09/27 08:52:42 INFO bsp.FileInputFormat: Total input paths to
>> process
>> >> : 2
>> >> > 13/09/27 08:52:42 INFO bsp.BSPJobClient: Running job:
>> >> job_201309270852_0002
>> >> > 13/09/27 08:52:45 INFO bsp.BSPJobClient: Current supersteps number:
0
>> >> > 13/09/27 08:58:10 INFO bsp.BSPJobClient: Current supersteps number:
1
>> >> > 13/09/27 08:58:16 INFO bsp.BSPJobClient: Current supersteps number:
2
>> >> > 13/09/27 09:00:04 INFO bsp.BSPJobClient: Job failed.
>> >> > 13/09/27 09:00:04 ERROR bsp.BSPJobClient: Error partitioning the
>> input
>> >> path.
>> >> > java.io.IOException: Runtime partition failed for the job.
>> >> >         at
>> >> org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:465)
>> >> >         at
>> >> >
>> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:333)
>> >> >         at
>> >> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
>> >> >         at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:229)
>> >> >         at org.apache.hama.graph.GraphJob.submit(GraphJob.java:203)
>> >> >         at
>> org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:236)
>> >> >         at
>> org.apache.hama.examples.PageRank2.main(PageRank2.java:164)
>> >> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >> >         at
>> >> >
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >> >         at
>> >> >
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> >         at java.lang.reflect.Method.invoke(Method.java:601)
>> >> >         at
>> >> >
>> >>
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> >> >         at
>> >> > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> >> >         at
>> >> > org.apache.hama.examples.ExampleDriver.main(ExampleDriver.java:43)
>> >> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> >> >         at
>> >> >
>> >>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >> >         at
>> >> >
>> >>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >> >         at java.lang.reflect.Method.invoke(Method.java:601)
>> >> >         at org.apache.hama.util.RunJar.main(RunJar.java:146)
>> >> >
>> >> > Does anyone know what's the possible reason of this exception?
>> >> > Thanks!
>> >> >
>> >> > Yingyi
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >> @eddieyoon
>> >>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>> @eddieyoon
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message