hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anthony garnier <sokar6...@hotmail.com>
Subject RE: Task process exit with nonzero status of 134
Date Tue, 27 Dec 2011 14:18:27 GMT

Alex
I don't know why but now I got other error code : 

On Jobtracker : 
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)

On taskTracker : 

2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.DefaultTaskController: Output from DefaultTaskController's
launchTask follows:
2011-12-27 15:14:50,663 INFO org.apache.hadoop.mapred.TaskController:
2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201112271514_0001_m_-1820714542
but just removed
2011-12-27 15:14:50,664 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201112271514_0001_m_-1820714542
exited with exit code 1. Number of tasks it ran: 0
2011-12-27 15:14:50,666 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201112271514_0001_m_000002_0
: Child Error
java.io.IOException: Task process exit with nonzero status of 1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
2011-12-27 15:14:53,676 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free
slots : 2


From: sokar6012@hotmail.com
To: wget.null@googlemail.com; common-user@hadoop.apache.org
Subject: RE: Task process exit with nonzero status of 134
Date: Tue, 27 Dec 2011 13:54:22 +0000







Alex

-XX:+UseCompressedOops option is the default in 1.6.0_24 and above on 64 bit JVMs (http://wiki.apache.org/hadoop/HadoopJavaVersions)
Anyway, I tested it but same result.
Is it wise to test hadoop with the new jdk7_2 ?

Anthony


> Date: Tue, 27 Dec 2011 13:47:03 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: wget.null@googlemail.com

> To: sokar6012@hotmail.com
> 
> Anthony,
> 
> 134 depends mostly on JRE (Bug) or defect RAM. _30 is the newest
> update, could be a bug inside. Can you test SE 7u2?
> Todd mentioned in a older post to use -XX:+UseCompressedOops
> (hadoop-env.sh). Another option could be to take a closer look at
> garbage collection with compressed option.
> 
> - Alex
> 
> On Tue, Dec 27, 2011 at 1:20 PM, anthony garnier <sokar6012@hotmail.com> wrote:
> > Alex,
> >
> > Memory available on namenode / Jobtracker :
> > Tasks: 435 total,   1 running, 434 sleeping,   0 stopped,   0 zombie
> > Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> > 0.0%st
> > Mem:     15360M total,    11609M used,     3750M free,      311M buffers
> > Swap:     2047M total,        1M used,     2046M free,     8833M cached
> >
> > On datanode / Tasktracker :
> > top - 13:15:27 up 6 days, 21:11,  1 user,  load average: 0.03, 0.28, 0.26
> > Tasks: 377 total,   1 running, 376 sleeping,   0 stopped,   0 zombie
> > Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,
> > 0.0%st
> > Mem:     72373M total,     4321M used,    68051M free,      348M buffers
> > Swap:     2047M total,        0M used,     2047M free,     2771M cached
> >
> > src/mapred/mapred-default.xml :
> > <property>
> >   <name>mapred.child.java.opts</name>
> >   <value>-Xmx200m</value>
> > </property>
> >
> >
> > So there should be enough memory
> >
> > Anthony
> >
> >
> >> Date: Tue, 27 Dec 2011 11:58:46 +0100
> >
> >> Subject: Re: Task process exit with nonzero status of 134
> >> From: wget.null@googlemail.com
> >> To: sokar6012@hotmail.com
> >
> >>
> >> Anthony,
> >>
> >> How much memory you have available? Did the system going into swap?
> >>
> >> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
> >> - what says "top -Hc"?
> >>
> >> - Alex
> >>
> >> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <sokar6012@hotmail.com>
> >> wrote:
> >> > Hi,
> >> >
> >> > I got Nothing in the dmesg
> >> > I've checked the Tasktracker and this is what I got :
> >> >
> >> > /************************************************************
> >> > STARTUP_MSG: Starting TaskTracker
> >> > STARTUP_MSG:   host = ylal2960.inetpsa.com/10.68.217.86
> >> > STARTUP_MSG:   args = []
> >> > STARTUP_MSG:   version = 0.20.203.0
> >> > STARTUP_MSG:   build =
> >> >
> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> >> > -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
> >> > ************************************************************/
> >> > 2011-12-23 15:11:02,275 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsConfig:
> >> > loaded properties from hadoop-metrics2.properties
> >> > 2011-12-23 15:11:02,330 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > MetricsSystem,sub=Stats registered.
> >> > 2011-12-23 15:11:02,331 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> >> > period
> >> > at 10 second(s).
> >> > 2011-12-23 15:11:02,331 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> >> > system started
> >> > 2011-12-23 15:11:02,597 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > ugi
> >> > registered.
> >> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to
> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> > org.mortbay.log.Slf4jLog
> >> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added
> >> > global
> >> > filtersafety
> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> >> > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> >> > Initializing logs' truncater with mapRetainSize=-1 and
> >> > reduceRetainSize=-1
> >> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > Starting
> >> > tasktracker with owner as root
> >> > 2011-12-23 15:11:02,870 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > jvm
> >> > registered.
> >> > 2011-12-23 15:11:02,871 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > TaskTrackerMetrics registered.
> >> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting
> >> > SocketReader
> >> > 2011-12-23 15:11:02,900 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > RpcDetailedActivityForPort58709 registered.
> >> > 2011-12-23 15:11:02,900 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > RpcActivityForPort58709 registered.
> >> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > Responder: starting
> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > listener on 58709: starting
> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 0 on 58709: starting
> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 1 on 58709: starting
> >> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 2 on 58709: starting
> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > TaskTracker up at: localhost/127.0.0.1:58709
> >> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > Starting
> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> >> > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 3 on 58709: starting
> >> > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > Using
> >> > MemoryCalculatorPlugin :
> >> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1
> >> > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > Starting
> >> > thread: Map-events fetcher for all reduce tasks on
> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> >> > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid
> >> > exited with exit code 0
> >> > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker:
> >> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> >> > disabled.
> >> > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache:
> >> > IndexCache
> >> > created with max memory = 10485760
> >> > 2011-12-23 15:11:02,976 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > ShuffleServerMetrics registered.
> >> > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port
> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> >> > -1.
> >> > Opening the listener on 50060
> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer:
> >> > listener.getLocalPort() returned 50060
> >> > webServer.getConnectors()[0].getLocalPort() returned 50060
> >> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty
> >> > bound
> >> > to port 50060
> >> > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26
> >> > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started
> >> > SelectChannelConnector@0.0.0.0:50060
> >> > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231432_0002 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231121_0001 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231312_0002 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231142_0002 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112230933_0002 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231357_0001 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231052_0002 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner:
> >> > Adding
> >> > job_201112231426_0001 for user-log deletion with
> >> > retainTimeStamp:1324735862967
> >> > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0
> >> > task's
> >> > state:UNASSIGNED
> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > Trying to
> >> > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots
> >> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In
> >> > TaskLauncher, current free slots : 2 and trying to launch
> >> > attempt_201112231511_0001_m_000002_0 which needs 1 slots
> >> > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer:
> >> > Initializing user root on this TT.
> >> > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> >> > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In
> >> > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851
> >> > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM
> >> > Runner
> >> > jvm_201112231511_0001_m_-1653678851 spawned.
> >> > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController:
> >> > Writing commands to
> >> >
> >> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh
> >> > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> >> > with
> >> > ID: jvm_201112231511_0001_m_-1653678851 given task:
> >> > attempt_201112231511_0001_m_000002_0
> >> > 2011-12-23 15:11:52,675 WARN
> >> > org.apache.hadoop.mapred.DefaultTaskController:
> >> > Exit code from task is : 134
> >> > 2011-12-23 15:11:52,675 INFO
> >> > org.apache.hadoop.mapred.DefaultTaskController:
> >> > Output from DefaultTaskController's launchTask follows:
> >> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController:
> >> >
> >> > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> >> > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of
> >> > tasks it ran: 0
> >> > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner:
> >> > attempt_201112231511_0001_m_000002_0 : Child Error
> >> > java.io.IOException: Task process exit with nonzero status of 134.
> >> >         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> >> >
> >> >
> >> >
> >> > I also checked the logs of the task :
> >> >
> >> >
> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout
> >> >
> >> >  A fatal error has been detected by the Java Runtime Environment:
> >> > #
> >> > #  SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472
> >> > #
> >> > # JRE version: 6.0_30-b12
> >> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode
> >> > linux-amd64 compressed oops)
> >> > # Problematic frame:
> >> > # C  [ld-linux-x86-64.so.2+0x882f]  double+0xcf
> >> > #
> >> > # An error report file with more information is saved as:
> >> > #
> >> >
> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log
> >> > #
> >> > # If you would like to submit a bug report, please visit:
> >> > #   http://java.sun.com/webapps/bugreport/crash.jsp
> >> > # The crash happened outside the Java Virtual Machine in native code.
> >> > # See problematic frame for where to report the bug.
> >> > ---------------
> >> >
> >> >
> >> >
> >> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog
> >> >
> >> > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration:
> >> >
> >> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> >> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> >> >
> >> >
> >> > On the Job tracker :
> >> > STARTUP_MSG: Starting JobTracker
> >> > STARTUP_MSG:   host = yval4950.inetpsa.com/10.68.76.61
> >> > STARTUP_MSG:   args = []
> >> > STARTUP_MSG:   version = 0.20.203.0
> >> > STARTUP_MSG:   build =
> >> >
> >> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> >> > -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
> >> > ************************************************************/
> >> > 2011-12-23 15:11:01,401 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsConfig:
> >> > loaded properties from hadoop-metrics2.properties
> >> > 2011-12-23 15:11:01,416 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > MetricsSystem,sub=Stats registered.
> >> > 2011-12-23 15:11:01,417 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> >> > period
> >> > at 10 second(s).
> >> > 2011-12-23 15:11:01,417 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics
> >> > system
> >> > started
> >> > 2011-12-23 15:11:01,727 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > ugi
> >> > registered.
> >> > 2011-12-23 15:11:01,733 INFO
> >> >
> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> >> > Updating the current master key for generating delegation tokens
> >> > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Scheduler
> >> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> >> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> >> > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader:
> >> > Refreshing hosts (include/exclude) list
> >> > 2011-12-23 15:11:01,745 INFO
> >> >
> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> >> > Starting expired delegation token remover thread,
> >> > tokenRemoverScanInterval=60 min(s)
> >> > 2011-12-23 15:11:01,745 INFO
> >> >
> >> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> >> > Updating the current master key for generating delegation tokens
> >> > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Starting
> >> > jobtracker with owner as root
> >> > 2011-12-23 15:11:01,836 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > RpcDetailedActivityForPort9001 registered.
> >> > 2011-12-23 15:11:01,837 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > RpcActivityForPort9001 registered.
> >> > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting
> >> > SocketReader
> >> > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to
> >> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> >> > org.mortbay.log.Slf4jLog
> >> > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added
> >> > global
> >> > filtersafety
> >> > (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> >> > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port
> >> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> >> > -1.
> >> > Opening the listener on 50030
> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer:
> >> > listener.getLocalPort() returned 50030
> >> > webServer.getConnectors()[0].getLocalPort() returned 50030
> >> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty
> >> > bound
> >> > to port 50030
> >> > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26
> >> > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started
> >> > SelectChannelConnector@0.0.0.0:50030
> >> > 2011-12-23 15:11:02,372 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > jvm
> >> > registered.
> >> > 2011-12-23 15:11:02,373 INFO
> >> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> >> > JobTrackerMetrics registered.
> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker:
> >> > JobTracker
> >> > up at: 9001
> >> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker:
> >> > JobTracker
> >> > webserver: 50030
> >> > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Cleaning
> >> > up the system directory
> >> > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker:
> >> > History
> >> > server being initialized in embedded mode
> >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobHistoryServer:
> >> > Started job history server at: localhost:50030
> >> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job
> >> > History Server web address: localhost:50030
> >> > 2011-12-23 15:11:02,585 INFO
> >> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> >> > inactive
> >> > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Refreshing
> >> > hosts information
> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader:
> >> > Setting
> >> > the includes file to
> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader:
> >> > Setting
> >> > the excludes file to
> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader:
> >> > Refreshing hosts (include/exclude) list
> >> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Decommissioning 0 nodes
> >> > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > Responder: starting
> >> > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > listener on 9001: starting
> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 0 on 9001: starting
> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 1 on 9001: starting
> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 2 on 9001: starting
> >> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 3 on 9001: starting
> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 4 on 9001: starting
> >> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 6 on 9001: starting
> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 7 on 9001: starting
> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 9 on 9001: starting
> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 5 on 9001: starting
> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Starting
> >> > RUNNING
> >> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 8 on 9001: starting
> >> > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology:
> >> > Adding a
> >> > new node: /default-rack/ylal2960.inetpsa.com
> >> > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: Adding
> >> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host
> >> > ylal2960.inetpsa.com
> >> > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology:
> >> > Adding a
> >> > new node: /default-rack/ylal2950.inetpsa.com
> >> > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: Adding
> >> > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host
> >> > ylal2950.inetpsa.com
> >> > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration:
> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a
> >> > attempt to
> >> > override final parameter: hadoop.tmp.dir;  Ignoring.
> >> > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration:
> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a
> >> > attempt to
> >> > override final parameter: dfs.data.dir;  Ignoring.
> >> > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration:
> >> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a
> >> > attempt to
> >> > override final parameter: dfs.name.dir;  Ignoring.
> >> > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > job_201112231511_0001: nMaps=1 nReduces=1 max=-1
> >> > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job
> >> > job_201112231511_0001 added successfully for user 'root' to queue
> >> > 'default'
> >> > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger:
> >> > USER=root    IP=10.68.76.61  OPERATION=SUBMIT_JOB
> >> > TARGET=job_201112231511_0001    RESULT=SUCCESS
> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Initializing job_201112231511_0001
> >> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > Initializing job_201112231511_0001
> >> > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > jobToken generated and stored with users keys in
> >> > /users/hadoop/mapred/system/job_201112231511_0001/jobToken
> >> > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > Input
> >> > size for job job_201112231511_0001 = 661630. Number of splits = 1
> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > tip:task_201112231511_0001_m_000000 has split on
> >> > node:/default-rack/ylal2960.inetpsa.com
> >> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress:
> >> > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5
> >> > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: Job
> >> > job_201112231511_0001 initialized successfully with 1 map tasks and 1
> >> > reduce
> >> > tasks.
> >> > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: Adding
> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip
> >> > task_201112231511_0001_m_000002, for tracker
> >> > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709'
> >> > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_0 is 151247 ms debug.
> >> > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_0 is 351649 ms debug.
> >> > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_0 is 551655 ms debug.
> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: Lost
> >> > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709'
> >> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress:
> >> > Error
> >> > from attempt_201112231511_0001_m_000002_0: Lost task tracker:
> >> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Removing
> >> > task 'attempt_201112231511_0001_m_000002_0'
> >> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_0 is 752357 ms debug.
> >> > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Launching
> >> > task attempt_201112231511_0001_m_000002_0 timed out.
> >> > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: Adding
> >> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip
> >> > task_201112231511_0001_m_000002, for tracker
> >> > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842'
> >> > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_1 is 198155 ms debug.
> >> > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_1 is 398161 ms debug.
> >> > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_1 is 598166 ms debug.
> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker:
> >> > attempt_201112231511_0001_m_000002_1 is 798172 ms debug.
> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Launching
> >> > task attempt_201112231511_0001_m_000002_1 timed out.
> >> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress:
> >> > Error
> >> > from attempt_201112231511_0001_m_000002_1: Error launching task
> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Lost
> >> > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842'
> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress:
> >> > Error
> >> > from attempt_201112231511_0001_m_000002_1: Lost task tracker:
> >> > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842
> >> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker:
> >> > Removing
> >> > task 'attempt_201112231511_0001_m_000002_1'
> >> >
> >> >
> >> >
> >> > Date: Fri, 23 Dec 2011 15:40:20 +0100
> >> > Subject: Re: Task process exit with nonzero status of 134
> >> > From: wget.null@googlemail.com
> >> > To: common-user@hadoop.apache.org
> >> >
> >> >
> >> > Hi,
> >> >
> >> > take a look into the logs for the failed attempt at your Tasktracker.
> >> > Also check the system logs with dmesg or /var/log/kern*. Could be a
> >> > syskill (segfault).
> >> >
> >> > - Alex
> >> >
> >> > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <sokar6012@hotmail.com>
> >> > wrote:
> >> >>
> >> >> Hi folks,
> >> >>
> >> >> I've just done a fresh install of Hadoop, Namenode and datanode are
up,
> >> >> Task/job Tracker also up, but when I run the Map reduce worcount
> >> >> exemple I
> >> >> got this error on Task tracker:
> >> >>
> >> >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM
:
> >> >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number
> >> >> of
> >> >> tasks it ran: 0
> >> >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner:
> >> >> attempt_201112231511_0001_m_000002_0 : Child Error
> >> >> java.io.IOException: Task process exit with nonzero status of 134.
> >> >>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> >> >>
> >> >> And job tracker stuck :
> >> >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/
> >> >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to
> >> >> process
> >> >> : 1
> >> >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job:
> >> >> job_201112231511_0001
> >> >> 11/12/23 15:11:50 INFO mapred.JobClient:  map 0% reduce 0%
> >> >>
> >> >>
> >> >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25
> >> >>
> >> >> I've done some googling, apparently the JVM crash hard (maybe Out of
> >> >> memory), does someone have any hint ?
> >> >>
> >> >> Regards,
> >> >>
> >> >> Anthony Garnier
> >> >> /DSIN/ASTI/ETSO
> >> >> IT Center
> >> >> PSA Peugeot Citroen
> >> >> Bessoncourt 90160
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Alexander Lorenz
> >> > http://mapredit.blogspot.com
> >> >
> >> > P Think of the environment: please don't print this email unless you
> >> > really need to.
> >>
> >>
> >>
> >> --
> >> Alexander Lorenz
> >> http://mapredit.blogspot.com
> >>
> >> P Think of the environment: please don't print this email unless you
> >> really need to.
> 
> 
> 
> -- 
> Alexander Lorenz
> http://mapredit.blogspot.com
> 
> P Think of the environment: please don't print this email unless you
> really need to.
 		 	   		   		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message