hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Groschupf ...@101tec.com>
Subject Re: Hadoop RPC call response post processing
Date Tue, 28 Dec 2010 21:00:12 GMT
Hi Todd, 
Right, that is the code I'm looking into. Though Responder is inner private class and is created
 "responder = new Responder();"
It would be great if the Responder implementation could be configured. 
Do you have any idea how to overwrite the Responder?

On Dec 27, 2010, at 8:21 PM, Todd Lipcon wrote:

> Hi Stefan,
> Sounds interesting.
> Maybe you're looking for o.a.h.ipc.Server$Responder?
> -Todd
> On Mon, Dec 27, 2010 at 8:07 PM, Stefan Groschupf <sg@101tec.com> wrote:
>> Hi All,
>> I'm browsing the RPC code since quite a while now trying to find any entry
>> point / interceptor slot that allows me to handle a RPC call response
>> writable after it was send over the wire.
>> Does anybody has an idea how break into the RPC code from outside. All the
>> interesting methods are private. :(
>> Background:
>> Heavy use of the RPC allocates hugh amount of Writable objects. We saw in
>> multiple systems  that the garbage collect can get so busy that the jvm
>> almost freezes for seconds. Things like zookeeper sessions time out in that
>> cases.
>> My idea is to create an object pool for writables. Borrowing an object from
>> the pool is simple since this happen in our custom code, though we do know
>> when the writable return was send over the wire and can be returned into the
>> pool.
>> A dirty hack would be to overwrite the write(out) method in the writable,
>> assuming that is the last thing done with the writable, though turns out
>> that this method is called in other cases too, e.g. to measure throughput.
>> Any ideas?
>> Thanks,
>> Stefan
> -- 
> Todd Lipcon
> Software Engineer, Cloudera

View raw message