hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Hadoop RPC call response post processing
Date Wed, 29 Dec 2010 01:56:31 GMT
On Tue, Dec 28, 2010 at 1:00 PM, Stefan Groschupf <sg@101tec.com> wrote:

> Hi Todd,
> Right, that is the code I'm looking into. Though Responder is inner private
> class and is created  "responder = new Responder();"
> It would be great if the Responder implementation could be configured.
> Do you have any idea how to overwrite the Responder?
>

Nope, it's not currently pluggable, nor do I think there's any compelling
reason to make it pluggable. It's coupled quite tightly to the
implementation right now.

Perhaps you can hack something in a git branch, and if it has good results
on something like NNBench it could be a general contribution?

-Todd


> On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:
>
> > Hi Stefan,
> >
> > Sounds interesting.
> >
> > Maybe you're looking for o.a.h.ipc.Server$Responder?
> >
> > -Todd
> >
> > On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg@101tec.com> wrote:
> >
> >> Hi All,
> >> I'm browsing the RPC code since quite a while now trying to find any
> entry
> >> point / interceptor slot that allows me to handle a RPC call response
> >> writable after it was send over the wire.
> >> Does anybody has an idea how break into the RPC code from outside. All
> the
> >> interesting methods are private. :(
> >>
> >> Background:
> >> Heavy use of the RPC allocates hugh amount of Writable objects. We saw
> in
> >> multiple systems  that the garbage collect can get so busy that the jvm
> >> almost freezes for seconds. Things like zookeeper sessions time out in
> that
> >> cases.
> >> My idea is to create an object pool for writables. Borrowing an object
> from
> >> the pool is simple since this happen in our custom code, though we do
> know
> >> when the writable return was send over the wire and can be returned into
> the
> >> pool.
> >> A dirty hack would be to overwrite the write(out) method in the
> writable,
> >> assuming that is the last thing done with the writable, though turns out
> >> that this method is called in other cases too, e.g. to measure
> throughput.
> >>
> >> Any ideas?
> >>
> >> Thanks,
> >> Stefan
> >
> >
> >
> >
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message