Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ankur.goel@corp.aol.com
 designates 64.236.137.26 as permitted sender)
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----_=_NextPart_001_01C91187.BEF302E0"
Subject: Multithreaded reduce
Date: Mon, 8 Sep 2008 13:21:47 +0530
Message-ID: 
 <8F11722A0562BB4F80A680ED4CFED0D5053F4337@EVSBNG02.ad.office.aol.com>
Thread-Topic: Multithreaded reduce
Thread-Index: AckRh77QK5fSQEPGQfyPZok/xjXfpQ==
From: "Goel, Ankur" <ankur.goel@corp.aol.com>
To: <core-dev@hadoop.apache.org>

------_=_NextPart_001_01C91187.BEF302E0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi Folks,

             I have a setup where I am using a thread-pool
implementation (provided by java.util.concurrent package) in reduce
phase to do database I/O to speed up the database upload. The DB here is
MySQL. I decided to go for additional parallelism via threads as=20

1. It considerably speeds up the upload while consuming less cluster
resources (i.e. less number of reducers required).=20

2. The upload speed is not limited by the reduce task capacity of the
cluster but by the DB's ability to handle max connections simultaneously
and effectively.

=20

Each reduce task has 2 thread pools. One that does the DB I/O and whose
return a java.util.concurrent.FutureTask. Another pool that fetches
result from this future task to do disc I/O i.e.
outputCollector.collect(...).

=20

When multiple threads from the second pool try to do a disc I/O, I get
an AlreadyBeingCreatedException in the logs. If I set the second pool to
only have 1 thread then things work fine!

=20

It looks like the output collector was never assumed to be used from
multiple threads.

=20

Any thoughts on this?

=20

Thanks

-Ankur

=20


------_=_NextPart_001_01C91187.BEF302E0--