Mailing-List: contact hama-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hama-dev@incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <AANLkTik6Ygy+Lh+H+mL0+q68vha1tcncd175sfjs+=X1@mail.gmail.com>
References: <AANLkTik6Ygy+Lh+H+mL0+q68vha1tcncd175sfjs+=X1@mail.gmail.com>
Date: Mon, 27 Sep 2010 13:20:19 +0900
Message-ID: <AANLkTi=tJZq+CLteaL6dz++k9a-HvAnKM31kde6ZiLBW@mail.gmail.com>
Subject: Re: question about job files (.xml and .jar) directories
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: hama-dev@incubator.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hello,

When user submit a job via following command ($ bin/hama jar
build/hama-0.2.0-dev-examples.jar pi) , the flow is as below:

1) jar and job files (named as a id) will be copied from the local to
(jobid named) temp directory on HDFS.

10/09/27 11:57:27 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/job_201009271157_=
0001

2) then, they will be copied to bspMaster's local file system from
HDFS, at JobInProgress. Because, the job can be submitted from some
slave node.

10/09/27 11:57:27 DEBUG bsp.JobInProgress: JobInProgress.localJobFile:
/tmp/hadoop-edward/bsp/local/bspMaster/job_201009271157_0001.xml
10/09/27 11:57:27 DEBUG bsp.JobInProgress: JobInProgress.localJarFile:
/tmp/hadoop-edward/bsp/local/bspMaster/job_201009271157_0001.jar

These files on a bspMaster local fs, will be used for loading the
input fomatter class and creating tasks on the bspMaster server. BTW,
since input/output system is not planned from Hama 0.2 version, we
don't need to care them at this time.

3) "actions" will be sent to slave nodes via heartbeat, and tasks will
be launched by launchTask() method.

For loading a BSP class and invoking the user defined BSP() method on
the slave server, the localizing step is in this process, too.

10/09/27 12:27:29 DEBUG bsp.GroomServer: localJobFile:
/tmp/hadoop-edward/bsp/local/groomServer/task_0/job.xml

On Mon, Sep 27, 2010 at 4:51 AM, Filipe David Manana
<fdmanana@apache.org> wrote:
> Hi, one simple question:
>
>
> In BSPJobClient.java, method submitJobInternal() we have:
>
> =C2=A0 =C2=A0Path submitJobDir =3D new Path(getSystemDir(), jobId.toStrin=
g());
> =C2=A0 =C2=A0Path submitJarFile =3D new Path(submitJobDir, "job.jar");
> =C2=A0 =C2=A0Path submitJobFile =3D new Path(submitJobDir, "job.xml");
>
> getSystemDir() will call BSPMaster.getSystemDir() which reads the
> config property "bsp.system.dir".
>
> However, in the constructor of JobInProgress.java we have:
>
> =C2=A0 =C2=A0this.localJobFile =3D master.getLocalPath(BSPMaster.SUBDIR +=
 "/" + jobId
> =C2=A0 =C2=A0 =C2=A0 =C2=A0+ ".xml");
> =C2=A0 =C2=A0this.localJarFile =3D master.getLocalPath(BSPMaster.SUBDIR +=
 "/" + jobId
> =C2=A0 =C2=A0 =C2=A0 =C2=A0+ ".jar");
>
> and getLocalPath() (BSPMaster class) will read the config property
> "bsp.local.dir"
>
>
> So, aren't we supposed to use the same paths in both places for the
> job files? (either local or system dir, but not both)
>
> cheers
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
> =C2=A0Unreasonable men adapt the world to themselves.
> =C2=A0That's why all progress depends on unreasonable men."
>


--=20
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org