hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Groomserer BSPPeerChild limit
Date Wed, 24 Jun 2015 22:14:44 GMT
Hi,

You can get the maximum number of available tasks like following code:

    BSPJobClient jobClient = new BSPJobClient(conf);
    ClusterStatus cluster = jobClient.getClusterStatus(true);

    // Set to maximum
    bsp.setNumBspTask(cluster.getMaxTasks());


On Wed, Jun 24, 2015 at 11:20 PM, Behroz Sikander <behroz89@gmail.com> wrote:
> Hi,
> 1) Thank you for this.
> 2) Here are the images. I will look into the log files of PI example
>
> *Result of JPS command on slave*
> http://s17.postimg.org/gpwe2bbfj/Screen_Shot_2015_06_22_at_7_23_31_PM.png
>
> *Result of JPS command on Master*
> http://s14.postimg.org/s9922em5p/Screen_Shot_2015_06_22_at_7_23_42_PM.png
>
> 3) In my current case, I do not have any input submitted to the job. During
> run time, I directly fetch data from HDFS. So, I am looking for something
> like BSPJob.set*Max*NumBspTask().
>
> Regards,
> Behroz
>
>
>
> On Tue, Jun 23, 2015 at 12:57 AM, Edward J. Yoon <edwardyoon@apache.org>
> wrote:
>
>> Hello,
>>
>> 1) You can get the filesystem URI from a configuration using
>> "FileSystem fs = FileSystem.get(conf);". Of course, the fs.defaultFS
>> property should be in hama-site.xml
>>
>>   <property>
>>     <name>fs.defaultFS</name>
>>     <value>hdfs://host1.mydomain.com:9000/</value>
>>     <description>
>>       The name of the default file system. Either the literal string
>>       "local" or a host:port for HDFS.
>>     </description>
>>   </property>
>>
>> 2) The 'bsp.tasks.maximum' is the number of tasks per node. It looks
>> cluster configuration issue. Please run Pi example and look at the
>> logs for more details. NOTE: you can not attach the images to mailing
>> list so I can't see it.
>>
>> 3) You can use the BSPJob.setNumBspTask(int) method. If input is
>> provided, the number of BSP tasks is basically driven by the number of
>> DFS blocks. I'll fix it to be more flexible on HAMA-956.
>>
>> Thanks!
>>
>>
>> On Tue, Jun 23, 2015 at 2:33 AM, Behroz Sikander <behroz89@gmail.com>
>> wrote:
>> > Hi,
>> > Recently, I moved from a single machine setup to a 2 machine setup. I was
>> > successfully able to run my job that uses the HDFS to get data. I have 3
>> > trivial questions
>> >
>> > 1- To access HDFS, I have to manually give the IP address of server
>> running
>> > HDFS. I thought that Hama will automatically pick from the configurations
>> > but it does not. I am probably doing something wrong. Right now my code
>> work
>> > by using the following.
>> >
>> > FileSystem fs = FileSystem.get(new URI("hdfs://server_ip:port/"), conf);
>> >
>> > 2- On my master server, when I start hama it automatically starts hama in
>> > the slave machine (all good). Both master and slave are set as
>> groomservers.
>> > This means that I have 2 servers to run my job which means that I can
>> open
>> > more BSPPeerChild processes. And if I submit my jar with 3 bsp tasks then
>> > everything works fine. But when I move to 4 tasks, Hama freezes. Here is
>> the
>> > result of JPS command on slave.
>> >
>> >
>> > Result of JPS command on Master
>> >
>> >
>> >
>> > You can see that it is only opening tasks on slaves but not on master.
>> >
>> > Note: I tried to change the bsp.tasks.maximum property in
>> hama-default.xml
>> > to 4 but still same result.
>> >
>> > 3- I want my cluster to open as many BSPPeerChild processes as possible.
>> Is
>> > there any setting that can I do to achieve that ? Or hama picks up the
>> > values from hama-default.xml to open tasks ?
>> >
>> >
>> > Regards,
>> >
>> > Behroz Sikander
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>>



-- 
Best Regards, Edward J. Yoon

Mime
View raw message