Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of zjffdu@gmail.com designates
 209.85.222.173 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=etiEcbPY0gPqQSnuqfSWj8XL0oQszmZQZDAWVsemLrlLC8BF3yWgl1oTzYPpxd3BKr
         yEo3qLE5OfMt7alqpK5XjjXbpW5CFNJGEmjpVtj2SA64tEn4uyRb1/70cpuTC7Fzjs+z
         99oOW+6mpCCf2ple4nLRuhD6UtcSQzStCdC3A=
MIME-Version: 1.0
In-Reply-To: <5a8ac5821002040010m73f256a1g7cb8a6c5a3f95d76@mail.gmail.com>
References: <5a8ac5821002040010m73f256a1g7cb8a6c5a3f95d76@mail.gmail.com>
Date: Thu, 4 Feb 2010 17:55:54 +0800
Message-ID: <8211a1321002040155m55bca611ha6f9e2561e61f390@mail.gmail.com>
Subject: Re: The idea to enhance MapReduce to resolve the skew problem
From: Jeff Zhang <zjffdu@gmail.com>
To: mapreduce-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00504502b960404edf047ec357d7

--00504502b960404edf047ec357d7
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

Do you mean do resplitting and recombining in each mapper task ? I am sure
what the purpose, as my understanding, the Partitioner determine which
reducer the output of mapper task go. So I don't think you method can solve
the skew problem.


2010/2/4 =E6=98=93=E5=89=91 <myhadoop@gmail.com>

> Currently, only map tasks are balanced, and reduce tasks possible are ske=
w,
> the timeslice is also different, which lead the scheduler is not smart. I
> have an idea to improve it.
>
> We can break the output of map to N*M splits, N is the number of nodes, a=
nd
> M >=3D1=EF=BC=8Cand regroup to new splits bycombining the smaller splits =
and
> resplitting the bigger splits, until the size of every splits is balanced
> with the specified value.
>
> There are three cases:
> 1. Too many values for a key
> 2. Too many keys hash to a partition
> 3. Every partition is balanced in the size
>
> If too many values for a key, adding a new MapReduce procedure is
> necessary.
> If too many keys hash to a partition, resplitting is necessary.
>
> If every splitting is balanced, we can consider a task (map or reduce) to=
 a
> scheduler timeslice, the scheduler will be smart like OS's scheduler.
>


--=20
Best Regards

Jeff Zhang

--00504502b960404edf047ec357d7--