Mailing-List: contact user-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hama.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAAp_xXHPv_O1=JSRAMJ=YFELt6khRXUH2vbwfb=LJymDKedmpg@mail.gmail.com>
References: 
 <CAAp_xXHcgOqh1gqNDngsjbFQYnx1p6um94ukMUQcEaGc1UB5HA@mail.gmail.com>
 <CAGQgZQQd2HWmy7qDBFZeq-1HcsoZ2Gs7zQvQ_yeC-XMrKfo-cg@mail.gmail.com>
 <CAAp_xXG8un8BuxxoW1TJ3oOmXKn1HBL7wBG+SHmJa6zoKdv-MA@mail.gmail.com>
 <CAGQgZQRo=qBNfq9V6VgFqxvRGM_qApmYMRqEoiyXVCN=6OFJaA@mail.gmail.com>
 <CAAp_xXHPv_O1=JSRAMJ=YFELt6khRXUH2vbwfb=LJymDKedmpg@mail.gmail.com>
From: Behroz Sikander <behroz89@gmail.com>
Date: Sat, 27 Jun 2015 01:03:24 +0200
Message-ID: 
 <CAAp_xXHF0E7t2baOPdU1aYyviCdsosfmUyePQZtXtYRTV=VwwQ@mail.gmail.com>
Subject: Re: Groomserer BSPPeerChild limit
To: user@hama.apache.org
Content-Type: multipart/alternative; boundary=047d7bd9131a6e5bf3051973c0b9

--047d7bd9131a6e5bf3051973c0b9
Content-Type: text/plain; charset=UTF-8

Hi,
In the current thread, I mentioned 3 issues. Issue 1 and 3 are resolved but
issue number 2 is still giving me headaches.

My problem:
My cluster now consists of 3 machines. Each one of them properly configured
(Apparently). From my master machine when I start Hadoop and Hama, I can
see the processes started on other 2 machines. If I check the maximum tasks
that my cluster can support then I get 9 (3 tasks on each machine).

When I run the PI example, it uses 9 tasks and runs fine. When I run my
program with 3 tasks, everything runs fine. But when I increase the tasks
(to 4) by using "setNumBspTask". Hama freezes. I do not understand what can
go wrong.

I checked the logs files and things look fine. I just sometimes get an
exception that hama was not able to delete the sytem directory
(bsp.system.dir) defined in the hama-site.xml.

Any help or clue would be great.

Regards,
Behroz Sikander

On Thu, Jun 25, 2015 at 1:13 PM, Behroz Sikander <behroz89@gmail.com> wrote:

> Thank you :)
>
> On Thu, Jun 25, 2015 at 12:14 AM, Edward J. Yoon <edwardyoon@apache.org>
> wrote:
>
>> Hi,
>>
>> You can get the maximum number of available tasks like following code:
>>
>>     BSPJobClient jobClient = new BSPJobClient(conf);
>>     ClusterStatus cluster = jobClient.getClusterStatus(true);
>>
>>     // Set to maximum
>>     bsp.setNumBspTask(cluster.getMaxTasks());
>>
>>
>> On Wed, Jun 24, 2015 at 11:20 PM, Behroz Sikander <behroz89@gmail.com>
>> wrote:
>> > Hi,
>> > 1) Thank you for this.
>> > 2) Here are the images. I will look into the log files of PI example
>> >
>> > *Result of JPS command on slave*
>> >
>> http://s17.postimg.org/gpwe2bbfj/Screen_Shot_2015_06_22_at_7_23_31_PM.png
>> >
>> > *Result of JPS command on Master*
>> >
>> http://s14.postimg.org/s9922em5p/Screen_Shot_2015_06_22_at_7_23_42_PM.png
>> >
>> > 3) In my current case, I do not have any input submitted to the job.
>> During
>> > run time, I directly fetch data from HDFS. So, I am looking for
>> something
>> > like BSPJob.set*Max*NumBspTask().
>> >
>> > Regards,
>> > Behroz
>> >
>> >
>> >
>> > On Tue, Jun 23, 2015 at 12:57 AM, Edward J. Yoon <edwardyoon@apache.org
>> >
>> > wrote:
>> >
>> >> Hello,
>> >>
>> >> 1) You can get the filesystem URI from a configuration using
>> >> "FileSystem fs = FileSystem.get(conf);". Of course, the fs.defaultFS
>> >> property should be in hama-site.xml
>> >>
>> >>   <property>
>> >>     <name>fs.defaultFS</name>
>> >>     <value>hdfs://host1.mydomain.com:9000/</value>
>> >>     <description>
>> >>       The name of the default file system. Either the literal string
>> >>       "local" or a host:port for HDFS.
>> >>     </description>
>> >>   </property>
>> >>
>> >> 2) The 'bsp.tasks.maximum' is the number of tasks per node. It looks
>> >> cluster configuration issue. Please run Pi example and look at the
>> >> logs for more details. NOTE: you can not attach the images to mailing
>> >> list so I can't see it.
>> >>
>> >> 3) You can use the BSPJob.setNumBspTask(int) method. If input is
>> >> provided, the number of BSP tasks is basically driven by the number of
>> >> DFS blocks. I'll fix it to be more flexible on HAMA-956.
>> >>
>> >> Thanks!
>> >>
>> >>
>> >> On Tue, Jun 23, 2015 at 2:33 AM, Behroz Sikander <behroz89@gmail.com>
>> >> wrote:
>> >> > Hi,
>> >> > Recently, I moved from a single machine setup to a 2 machine setup.
>> I was
>> >> > successfully able to run my job that uses the HDFS to get data. I
>> have 3
>> >> > trivial questions
>> >> >
>> >> > 1- To access HDFS, I have to manually give the IP address of server
>> >> running
>> >> > HDFS. I thought that Hama will automatically pick from the
>> configurations
>> >> > but it does not. I am probably doing something wrong. Right now my
>> code
>> >> work
>> >> > by using the following.
>> >> >
>> >> > FileSystem fs = FileSystem.get(new URI("hdfs://server_ip:port/"),
>> conf);
>> >> >
>> >> > 2- On my master server, when I start hama it automatically starts
>> hama in
>> >> > the slave machine (all good). Both master and slave are set as
>> >> groomservers.
>> >> > This means that I have 2 servers to run my job which means that I can
>> >> open
>> >> > more BSPPeerChild processes. And if I submit my jar with 3 bsp tasks
>> then
>> >> > everything works fine. But when I move to 4 tasks, Hama freezes.
>> Here is
>> >> the
>> >> > result of JPS command on slave.
>> >> >
>> >> >
>> >> > Result of JPS command on Master
>> >> >
>> >> >
>> >> >
>> >> > You can see that it is only opening tasks on slaves but not on
>> master.
>> >> >
>> >> > Note: I tried to change the bsp.tasks.maximum property in
>> >> hama-default.xml
>> >> > to 4 but still same result.
>> >> >
>> >> > 3- I want my cluster to open as many BSPPeerChild processes as
>> possible.
>> >> Is
>> >> > there any setting that can I do to achieve that ? Or hama picks up
>> the
>> >> > values from hama-default.xml to open tasks ?
>> >> >
>> >> >
>> >> > Regards,
>> >> >
>> >> > Behroz Sikander
>> >>
>> >>
>> >>
>> >> --
>> >> Best Regards, Edward J. Yoon
>> >>
>>
>>
>>
>> --
>> Best Regards, Edward J. Yoon
>>
>
>

--047d7bd9131a6e5bf3051973c0b9--