hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Hung Lin <cli...@googlemail.com>
Subject Re: Hama Scheduler
Date Tue, 24 Dec 2013 14:15:01 GMT
BSPMaster makes use of TaskScheduler for scheduling tasks

Class<? extends TaskScheduler> schedulerClass = conf.getClass(
        "bsp.master.taskscheduler", SimpleTaskScheduler.class,
this.taskScheduler = ReflectionUtils.newInstance(schedulerClass, conf);

Then in SimpleTaskScheduler, tasks are scheduled through schedule
function. And tasks are obtained by related JobInProgress.

That's roughly the execution path.

IIRC the split size is more related to a single job so increasing
split size may not allow more jobs to be ran on the same cluster.

At the moment the scheduling mechanism is done by creating a task per
GroomServer, and each GroomServer allows default maxTasks up to 3. So
increasing maxTasks may be a way to increase job running concurrently;
or restricting tasks scheduled to GroomServer, and then scheduling
tasks in the new job to free slots may also help increase the
concurrent job execution.

Right now scheduling is just a simple FCFS. It's welcome the
improvment on adding something like policy so that the scheduling
mechanism is more flexible.

On 24 December 2013 20:06, Yuesheng Hu <yueshenghu@gmail.com> wrote:
> Hi, I want to implement a scheduler for hama as my thesis project.
> Here are my ideas:
> 1. make the split size as big as possible, so more jobs can be run on the
> cluster;
> 2. the scheduler can select a job based on the job's type, like graph job
> (message intensive) or iterative job
> Any advice?
> Best Regards!

View raw message