hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Behroz Sikander <behro...@gmail.com>
Subject Re: Groomserer BSPPeerChild limit
Date Thu, 25 Jun 2015 11:13:25 GMT
Thank you :)

On Thu, Jun 25, 2015 at 12:14 AM, Edward J. Yoon <edwardyoon@apache.org>
wrote:

> Hi,
>
> You can get the maximum number of available tasks like following code:
>
>     BSPJobClient jobClient = new BSPJobClient(conf);
>     ClusterStatus cluster = jobClient.getClusterStatus(true);
>
>     // Set to maximum
>     bsp.setNumBspTask(cluster.getMaxTasks());
>
>
> On Wed, Jun 24, 2015 at 11:20 PM, Behroz Sikander <behroz89@gmail.com>
> wrote:
> > Hi,
> > 1) Thank you for this.
> > 2) Here are the images. I will look into the log files of PI example
> >
> > *Result of JPS command on slave*
> >
> http://s17.postimg.org/gpwe2bbfj/Screen_Shot_2015_06_22_at_7_23_31_PM.png
> >
> > *Result of JPS command on Master*
> >
> http://s14.postimg.org/s9922em5p/Screen_Shot_2015_06_22_at_7_23_42_PM.png
> >
> > 3) In my current case, I do not have any input submitted to the job.
> During
> > run time, I directly fetch data from HDFS. So, I am looking for something
> > like BSPJob.set*Max*NumBspTask().
> >
> > Regards,
> > Behroz
> >
> >
> >
> > On Tue, Jun 23, 2015 at 12:57 AM, Edward J. Yoon <edwardyoon@apache.org>
> > wrote:
> >
> >> Hello,
> >>
> >> 1) You can get the filesystem URI from a configuration using
> >> "FileSystem fs = FileSystem.get(conf);". Of course, the fs.defaultFS
> >> property should be in hama-site.xml
> >>
> >>   <property>
> >>     <name>fs.defaultFS</name>
> >>     <value>hdfs://host1.mydomain.com:9000/</value>
> >>     <description>
> >>       The name of the default file system. Either the literal string
> >>       "local" or a host:port for HDFS.
> >>     </description>
> >>   </property>
> >>
> >> 2) The 'bsp.tasks.maximum' is the number of tasks per node. It looks
> >> cluster configuration issue. Please run Pi example and look at the
> >> logs for more details. NOTE: you can not attach the images to mailing
> >> list so I can't see it.
> >>
> >> 3) You can use the BSPJob.setNumBspTask(int) method. If input is
> >> provided, the number of BSP tasks is basically driven by the number of
> >> DFS blocks. I'll fix it to be more flexible on HAMA-956.
> >>
> >> Thanks!
> >>
> >>
> >> On Tue, Jun 23, 2015 at 2:33 AM, Behroz Sikander <behroz89@gmail.com>
> >> wrote:
> >> > Hi,
> >> > Recently, I moved from a single machine setup to a 2 machine setup. I
> was
> >> > successfully able to run my job that uses the HDFS to get data. I
> have 3
> >> > trivial questions
> >> >
> >> > 1- To access HDFS, I have to manually give the IP address of server
> >> running
> >> > HDFS. I thought that Hama will automatically pick from the
> configurations
> >> > but it does not. I am probably doing something wrong. Right now my
> code
> >> work
> >> > by using the following.
> >> >
> >> > FileSystem fs = FileSystem.get(new URI("hdfs://server_ip:port/"),
> conf);
> >> >
> >> > 2- On my master server, when I start hama it automatically starts
> hama in
> >> > the slave machine (all good). Both master and slave are set as
> >> groomservers.
> >> > This means that I have 2 servers to run my job which means that I can
> >> open
> >> > more BSPPeerChild processes. And if I submit my jar with 3 bsp tasks
> then
> >> > everything works fine. But when I move to 4 tasks, Hama freezes. Here
> is
> >> the
> >> > result of JPS command on slave.
> >> >
> >> >
> >> > Result of JPS command on Master
> >> >
> >> >
> >> >
> >> > You can see that it is only opening tasks on slaves but not on master.
> >> >
> >> > Note: I tried to change the bsp.tasks.maximum property in
> >> hama-default.xml
> >> > to 4 but still same result.
> >> >
> >> > 3- I want my cluster to open as many BSPPeerChild processes as
> possible.
> >> Is
> >> > there any setting that can I do to achieve that ? Or hama picks up the
> >> > values from hama-default.xml to open tasks ?
> >> >
> >> >
> >> > Regards,
> >> >
> >> > Behroz Sikander
> >>
> >>
> >>
> >> --
> >> Best Regards, Edward J. Yoon
> >>
>
>
>
> --
> Best Regards, Edward J. Yoon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message