hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: The number of splits has exceeded the number of max tasks
Date Sun, 09 Mar 2014 12:05:54 GMT
What's the value of "bsp.max.tasks.per.job"?

Please try to run Pi example and then how many tasks are launched?

On Sun, Mar 9, 2014 at 8:24 PM, Ammar Sahib <ammar.sahib@yahoo.com> wrote:
> I was sure to restart HAMA. According to the web UI, I have the following in my cluster:
>
>
> master:40000 Hama BSP AdministrationState: RUNNING
> Started: Sun Mar 09 12:17:01 CET 2014
> Version: 0.6.4
> Compiled By: edward
> Compiled At Time: Mon Mar  3 19:14:32 KST 2014
> Identifier: 201403091217
> ________________________________
>
> Groom Servers
> BSP Task Capacity
> Avg. Tasks/Node
> Blacklisted Nodes
> 4 120 30.00 0
> ________________________________
>
> Running Jobs
> No jobs found!
> ________________________________
>
> All Jobs History
> No jobs found!
> ________________________________
>  Hama, 2014.
>
>
> But When running my job I will something different:
>
> 14/03/09 12:19:21 INFO bsp.FileInputFormat: Total input paths to process : 1
> 14/03/09 12:19:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
> 14/03/09 12:19:22 WARN snappy.LoadSnappy: Snappy native library not loaded
> 14/03/09 12:19:22 INFO bsp.FileInputFormat: Total input paths to process : 1
> Exception in thread "main" java.io.IOException: Job failed! The number of splits has
exceeded the number of max tasks. The number of splits: 52, The number of max tasks: 20
>         at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:349)
>         at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296)
>         at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219)
>         at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226)
>         at org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:460)
>         at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:341)
>         at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296)
>         at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219)
>         at org.apache.hama.graph.GraphJob.submit(GraphJob.java:208)
>         at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226)
>         at de.rwthaachen.dbis.i5cloudmatch.controller.Matcher.main(Matcher.java:479)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
>
>
>
>
> On Sunday, March 9, 2014 12:23 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
>
> Please use the web UI to check the cluster capacity.
>
> I think your cluster is not working correctly now, or you didn't
> restart your cluster.
>
> On Sun, Mar 9, 2014 at 2:00 AM, Ammar Sahib <ammar.sahib@yahoo.com> wrote:
>> Hi
>>
>> I tried to increase the bsp.tasks.maximum in hama-default.xml from 3 to 30
>> but I still get the same error.
>> I am thinking of of reducing the number of blocks of the input file by
>> controlling the parameter dfs.namenode.fs-limits.min-block-size in
>> hdfs-default.xml. Do you think this might be a good approach that might
>> solve the problem?
>>
>>
>>
>> On Friday, March 7, 2014 11:32 PM, Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>> If the number of blocks of input file is 52 (see [1]), you should
>> increase number of task slots by adding new machine or increasing max
>> tasks number per node "bsp.tasks.maximum".
>>
>> 1.
>> http://stackoverflow.com/questions/11168427/viewing-the-number-of-blocks-for-a-file-in-hadoop
>>
>> On Sat, Mar 8, 2014 at 12:22 AM, Ammar Sahib <ammar.sahib@yahoo.com> wrote:
>>> Hi
>>>
>>>
>>> I am using HAMA 0.6.4 and I am running my custom program using a cluster
>>> of 4 machines. My input is a single file and I am setting the number of
>>> BSP tasks to the number of Groom servers by using
>>> (JOB.setNumBspTask(cluster.getGroomServers()). I am using the
>>> HashPartitioner.class to partition the data.
>>>
>>>
>>> I have a problem when I load my data. When I run my custom program I get
>>> the following error messages:
>>>
>>> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process :
>>> 1
>>> 14/03/07 16:02:34 INFO util.NativeCodeLoader: Loaded the
>>>  native-hadoop library
>>> 14/03/07 16:02:34 WARN snappy.LoadSnappy: Snappy native library not loaded
>>> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process :
>>> 1
>>> Exception
>>>  in thread "main" java.io.IOException: Job failed! The number of splits
>>> has exceeded the number of max tasks. The number of splits: 52, The
>>> number of max tasks: 20
>>>        at
>>> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:349)
>>>        at
>>> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296)
>>>        at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219)
>>>        at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226)
>>>        at
>>> org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:460)
>>>
>>>  at
>>> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:341)
>>>        at
>>> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296)
>>>        at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219)
>>>        at org.apache.hama.graph.GraphJob.submit(GraphJob.java:208)
>>>        at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226)
>>>        at
>>> de.rwthaachen.dbis.i5cloudmatch.controller.Matcher.main(Matcher.java:479)
>>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>        at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>        at
>>>
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>        at java.lang.reflect.Method.invoke(Method.java:606)
>>>        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>
>>> Any advice of how to solve this problem?
>>>
>>> RegardsAmmar
>>
>>
>>
>> --
>> Edward J. Yoon (@eddieyoon)
>> Chief Executive Officer
>> DataSayer, Inc.
>>
>>
>
>
>
> --
> Edward J. Yoon (@eddieyoon)
> Chief Executive Officer
> DataSayer, Inc.



-- 
Edward J. Yoon (@eddieyoon)
Chief Executive Officer
DataSayer, Inc.

Mime
View raw message