hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Groschupf ...@101tec.com>
Subject Hadoop RPC call response post processing
Date Tue, 28 Dec 2010 04:07:34 GMT
Hi All, 
I'm browsing the RPC code since quite a while now trying to find any entry point / interceptor
slot that allows me to handle a RPC call response writable after it was send over the wire.
Does anybody has an idea how break into the RPC code from outside. All the interesting methods
are private. :(

Heavy use of the RPC allocates hugh amount of Writable objects. We saw in multiple systems
 that the garbage collect can get so busy that the jvm almost freezes for seconds. Things
like zookeeper sessions time out in that cases.
My idea is to create an object pool for writables. Borrowing an object from the pool is simple
since this happen in our custom code, though we do know when the writable return was send
over the wire and can be returned into the pool.
A dirty hack would be to overwrite the write(out) method in the writable, assuming that is
the last thing done with the writable, though turns out that this method is called in other
cases too, e.g. to measure throughput.

Any ideas?

View raw message