hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adi <adi.pan...@gmail.com>
Subject Re: mapper java process not exiting
Date Fri, 13 May 2011 00:33:28 GMT
Which version of hadoop are you running?
>
> Hadoop 0.21.0 with some patches.



> Are you running on linux?
>
> Yes
Linux 2.6.18-238.9.1.el5 #1 SMP  x86_64 x86_64 x86_64 GNU/Linux
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)

I set up 0.21.0 on another linux box and am not seeing this issue as hadoop
is reusing JVMs(as configured).
In the production cluster it is not re-using JVMs and runs out of memory
because of mapper JVMs staying alive even after they have ended according to
hadoop.

The production node is a 64 bit OS/JVM. Is there any known issue workaround
for enabling JVM reuse in 64 bit environments.

Test node is 32 bit:
Linux 2.6.18-194.32.1.el5.centos.plus #1 SMP i686 i686 i386 GNU/Linux
java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-i386)
OpenJDK Server VM (build 14.0-b16, mixed mode)

Even if I can get it to reuse JVM it will be grrreat.

-Adi





> -Joey
>
> On Thu, May 12, 2011 at 1:39 PM, Adi <adi.pandit@gmail.com> wrote:
> > For one long running job we are noticing that the mapper jvms do not exit
> > even after the mapper is done. Any suggestions on why this could be
> > happening.
> > The java processes get cleaned up if I do a hadoop job -kill <job_id>.
> The
> > java processes get cleaned up of I run in it in a smaller batch and the
> job
> > gets done fairly quickly(say half an hour). For larger inputs the nodes
> > eventually run out of memory because of these java processes that the
> > cluster thinks are gone but they haven't been cleaned up yet. I am
> > suspecting the TaskTrackers are failing to kill JVMs for some reason by
> > themselves.
> > The following exceptions can be seen in the hadoop logs.
> >
> > 2011-05-12 13:52:04,147 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -12545: No such
> process
> > 2011-05-12 13:52:08,071 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -11061: No such
> process
> > 2011-05-12 13:52:09,009 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -11151: No such
> process
> > 2011-05-12 13:52:12,009 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -25057: No such
> process
> > 2011-05-12 13:52:13,306 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -19805: No such
> process
> > 2011-05-12 13:52:14,996 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -11103: No such
> process
> >
> > 2011-05-12 15:51:41,105 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -17202: No such
> process
> > 2011-05-12 15:51:43,481 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -15981: No such
> process
> > 2011-05-12 15:51:45,916 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -17931: No such
> process
> > 2011-05-12 15:52:06,328 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -14867: No such
> process
> > 2011-05-12 15:52:34,503 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -29376: No such
> process
> > 2011-05-12 15:52:38,607 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -32491: No such
> process
> > 2011-05-12 15:52:39,292 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -31529: No such
> process
> > 2011-05-12 15:52:46,547 WARN
> org.apache.hadoop.mapreduce.util.ProcessTree:
> > Error executing shell command
> > org.apache.hadoop.util.Shell$ExitCodeException: kill -15140: No such
> process
> >
> > Some other exceptions also seen in the logs may or may not be related to
> the
> > above problem.
> > 2011-05-12 16:01:20,534 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 33465 caught: java.nio.channels.ClosedChannelException
> > 2011-05-12 16:01:48,869 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 80 on 33465 caught: java.nio.channels.ClosedChannelException
> > 2011-05-12 16:01:53,922 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 59 on 33465 caught: java.nio.channels.ClosedChannelException
> > 2011-05-12 16:01:58,977 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 28 on 33465 caught: java.nio.channels.ClosedChannelException
> > 2011-05-12 16:02:04,040 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 37 on 33465 caught: java.nio.channels.ClosedChannelException
> > 2011-05-12 16:02:09,095 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 100 on 33465 caught: java.nio.channels.ClosedChannelException
> >
> > Thanks.
> >
> > -Adi
> >
>
>
>
> --
> Joseph Echeverria
> Cloudera, Inc.
> 443.305.9434
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message