Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of dhruv21@gmail.com designates
 209.85.216.176 as permitted sender)
MIME-Version: 1.0
Date: Thu, 1 Nov 2012 15:45:21 -0700
Message-ID: 
 <CAFk0Lrc_FqqgyjVJgNvQxQM9-95_j=cSH2iO5N3qKqzT8LGqXw@mail.gmail.com>
Subject: OutputFormat and Reduce Task
From: Dhruv <dhruv21@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e68ee9852c7a0504cd76c672

--0016e68ee9852c7a0504cd76c672
Content-Type: text/plain; charset=ISO-8859-1

I'm trying to optimize the performance of my OutputFormat's
implementation. I'm doing things similar to HBase's
TableOutputFormat--sending the reducer's output to a distributed k-v store.
So, the context.write() call basically winds up doing a Put() on the store.

Although I haven't profiled, a sequence of thread dumps on the reduce tasks
reveal that the threads are RUNNABLE and hanging out in the put() and its
subsequent method calls. So, I proceeded to decouple these two by
implementing the producer (context.write()) consumer (RecordWriter.write())
pattern using ExecutorService.

My understanding is that Context.write() calls RecordWriter.write() and
that these two are synchronous calls. The first will block until the second
method completes.Each reduce phase blocks until the context.write()
finishes, so the next reduce on the next key also blocks, making things run
slow in my case. Is this correct? Does this mean that OutputFormat is
instantiated once by the TaskTracker for the Job's reduce logic and all
keys operated on by the reducers get the same instance of the OutputFormat.
Or, is it that for each key operated by the reducer, a new OutputFormat is
instantiated?

Thanks,
Dhruv

--0016e68ee9852c7a0504cd76c672
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I&#39;m trying to optimize the performance of my OutputFormat&#39;s impleme=
ntation.=A0I&#39;m doing things similar to HBase&#39;s TableOutputFormat--s=
ending the reducer&#39;s output to a distributed k-v store. So, the context=
.write() call basically winds up doing a Put() on the store.=A0<div>
<br></div><div>Although I haven&#39;t profiled, a sequence of thread dumps =
on the reduce tasks reveal that the threads are RUNNABLE and hanging out in=
 the put() and its subsequent method calls. So,=A0I proceeded to decouple t=
hese two by implementing the producer (context.write()) consumer (RecordWri=
ter.write()) pattern using ExecutorService.</div>
<div><br></div><div>My understanding is that Context.write() calls RecordWr=
iter.write() and that these two are synchronous calls. The first will block=
 until the second method completes.Each reduce phase blocks until the conte=
xt.write() finishes, so the next reduce on the next key also blocks, making=
 things run slow in my case. Is this correct? Does this mean that OutputFor=
mat is instantiated once by the TaskTracker for the Job&#39;s reduce logic =
and all keys operated on by the reducers get the same instance of the Outpu=
tFormat. Or, is it that for each key operated by the reducer, a new OutputF=
ormat is instantiated?</div>
<div><br></div><div>Thanks,</div><div>Dhruv</div>

--0016e68ee9852c7a0504cd76c672--