incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: question about job files (.xml and .jar) directories
Date Mon, 27 Sep 2010 04:20:19 GMT
Hello,

When user submit a job via following command ($ bin/hama jar
build/hama-0.2.0-dev-examples.jar pi) , the flow is as below:

1) jar and job files (named as a id) will be copied from the local to
(jobid named) temp directory on HDFS.

10/09/27 11:57:27 DEBUG bsp.BSPJobClient: BSPJobClient.submitJobDir:
hdfs://slave.udanax.org:9000/tmp/hadoop-edward/bsp/system/job_201009271157_0001

2) then, they will be copied to bspMaster's local file system from
HDFS, at JobInProgress. Because, the job can be submitted from some
slave node.

10/09/27 11:57:27 DEBUG bsp.JobInProgress: JobInProgress.localJobFile:
/tmp/hadoop-edward/bsp/local/bspMaster/job_201009271157_0001.xml
10/09/27 11:57:27 DEBUG bsp.JobInProgress: JobInProgress.localJarFile:
/tmp/hadoop-edward/bsp/local/bspMaster/job_201009271157_0001.jar

These files on a bspMaster local fs, will be used for loading the
input fomatter class and creating tasks on the bspMaster server. BTW,
since input/output system is not planned from Hama 0.2 version, we
don't need to care them at this time.

3) "actions" will be sent to slave nodes via heartbeat, and tasks will
be launched by launchTask() method.

For loading a BSP class and invoking the user defined BSP() method on
the slave server, the localizing step is in this process, too.

10/09/27 12:27:29 DEBUG bsp.GroomServer: localJobFile:
/tmp/hadoop-edward/bsp/local/groomServer/task_0/job.xml

On Mon, Sep 27, 2010 at 4:51 AM, Filipe David Manana
<fdmanana@apache.org> wrote:
> Hi, one simple question:
>
>
> In BSPJobClient.java, method submitJobInternal() we have:
>
>    Path submitJobDir = new Path(getSystemDir(), jobId.toString());
>    Path submitJarFile = new Path(submitJobDir, "job.jar");
>    Path submitJobFile = new Path(submitJobDir, "job.xml");
>
> getSystemDir() will call BSPMaster.getSystemDir() which reads the
> config property "bsp.system.dir".
>
> However, in the constructor of JobInProgress.java we have:
>
>    this.localJobFile = master.getLocalPath(BSPMaster.SUBDIR + "/" + jobId
>        + ".xml");
>    this.localJarFile = master.getLocalPath(BSPMaster.SUBDIR + "/" + jobId
>        + ".jar");
>
> and getLocalPath() (BSPMaster class) will read the config property
> "bsp.local.dir"
>
>
> So, aren't we supposed to use the same paths in both places for the
> job files? (either local or system dir, but not both)
>
> cheers
>
>
> --
> Filipe David Manana,
> fdmanana@gmail.com, fdmanana@apache.org
>
> "Reasonable men adapt themselves to the world.
>  Unreasonable men adapt the world to themselves.
>  That's why all progress depends on unreasonable men."
>



-- 
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org

Mime
View raw message