hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anfernee Xu <anfernee...@gmail.com>
Subject Socket Reader Thread was crashed with Hadoop Writable engine, and client reached to the dead end
Date Mon, 03 Nov 2014 07:00:05 GMT
Hi experts,

I'm running with Hadoop 2.2.0,  I have a distributed application written
with Hadoop Writable and I'm pretty happy with it, pretty stable. But a few
days back, I ran into one strange issue, one client was stucking in talking
to one of the servers, the RPC was made in a background thread and I have a
timer to guard against timeout, the call was timeout as expected, then
another user made the same call to the same server, still timeout, after
some time, I found I ran out of threads for the threadpool which is used
for making the RPC call. Also I found the Hadoop RPC Socket Reader thread
was crashed for some reason(I cannot find it from JConsole) on the server
side. I also looked the code, I think below code is problematic.

org.apache.hadoop.ipc.Client.call method, around line#1336, as below

while (!call.done) {
try
{
     call.wait(); // wait for the result }
catch (InterruptedException ie)
{ // save the fact that we were interrupted

 interrupted = true; }

}

Looks like even I tried to interrupt the call, the client was still trying
to wait for the response, is this a bug?

Thanks

-- 
--Anfernee

Mime
View raw message