Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of vliaskov@gmail.com designates
 209.85.212.186 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=DcN+TSHWnuXbYE+SkrvWlzFoLoQw2FpHZ88vYjoEV/zmYub04fZWL9AJcT1AfR318v
         tN4k+MtQDNeOjL2EQzzM90JTzCWHRXknmBr34SI6bu3aDdGX3XZkKaug6U4IQOPPdj2X
         zT5HUCfQwHOb226OIV5kie+p3qBA1Kc2NLvak=
MIME-Version: 1.0
Date: Sat, 26 Sep 2009 20:37:34 -0500
Message-ID: <bec373e0909261837o142aa8bqdf70ca452d24829a@mail.gmail.com>
Subject: default job scheduler behaviour
From: Vasilis Liaskovitis <vliaskov@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Hi,

given a single cluster running with the default job scheduler: Is only
one job executing on the cluster, regardless of how many task
map/reduce slots it can keep busy?
In other words, If a job does not use all task slots, would the
default scheduler consider scheduling map/reduce from other jobs that
have already been submitted to the system?

I am using an 8-node cluster to run some test jobs based on gridmix
(the synthetic benchmark found in the hadoop distribution under
src/banchmarks/gridmix). The gridmix workload submits many different
jobs in parallel - 5 different kinds of jobs of varying sizes for each
kind: small, medium, large. While running, I am noticing that at any
time only one job is making progress - at least according to the
jobtracker web ui. I think this is happening even for small-size jobs,
which don't take up all slots of the cluster's tasktrackers/nodes.

If the default scheduler is not capable of scheduling tasks for
multiple jobs, would I have to use the capacity scheduler? Or
something else?

thanks for any help,

- Vasilis