hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anthony garnier <sokar6...@hotmail.com>
Subject RE: Task process exit with nonzero status of 134
Date Tue, 27 Dec 2011 12:20:02 GMT

Alex,

Memory available on namenode / Jobtracker : 
Tasks: 435 total,   1 running, 434 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     15360M total,    11609M used,     3750M free,      311M buffers
Swap:     2047M total,        1M used,     2046M free,     8833M cached

On datanode / Tasktracker : 
top - 13:15:27 up 6 days, 21:11,  1 user,  load average: 0.03, 0.28, 0.26
Tasks: 377 total,   1 running, 376 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:     72373M total,     4321M used,    68051M free,      348M buffers
Swap:     2047M total,        0M used,     2047M free,     2771M cached

src/mapred/mapred-default.xml :
<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx200m</value>
</property>


So there should be enough memory 

Anthony


> Date: Tue, 27 Dec 2011 11:58:46 +0100
> Subject: Re: Task process exit with nonzero status of 134
> From: wget.null@googlemail.com
> To: sokar6012@hotmail.com
> 
> Anthony,
> 
> How much memory you have available? Did the system going into swap?
> 
> - Check mapred.map.child.java.opts (mapred.xml) for given MaxSize (xmx).
> - what says "top -Hc"?
> 
> - Alex
> 
> On Tue, Dec 27, 2011 at 11:49 AM, anthony garnier <sokar6012@hotmail.com> wrote:
> > Hi,
> >
> > I got Nothing in the dmesg
> > I've checked the Tasktracker and this is what I got :
> >
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG:   host = ylal2960.inetpsa.com/10.68.217.86
> > STARTUP_MSG:   args = []
> > STARTUP_MSG:   version = 0.20.203.0
> > STARTUP_MSG:   build =
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> > -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
> > ************************************************************/
> > 2011-12-23 15:11:02,275 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> > loaded properties from hadoop-metrics2.properties
> > 2011-12-23 15:11:02,330 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > MetricsSystem,sub=Stats registered.
> > 2011-12-23 15:11:02,331 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> > at 10 second(s).
> > 2011-12-23 15:11:02,331 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: TaskTracker metrics
> > system started
> > 2011-12-23 15:11:02,597 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> > registered.
> > 2011-12-23 15:11:02,738 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-12-23 15:11:02,803 INFO org.apache.hadoop.http.HttpServer: Added global
> > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2011-12-23 15:11:02,827 INFO org.apache.hadoop.mapred.TaskLogsTruncater:
> > Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
> > 2011-12-23 15:11:02,832 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> > tasktracker with owner as root
> > 2011-12-23 15:11:02,870 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> > registered.
> > 2011-12-23 15:11:02,871 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > TaskTrackerMetrics registered.
> > 2011-12-23 15:11:02,897 INFO org.apache.hadoop.ipc.Server: Starting
> > SocketReader
> > 2011-12-23 15:11:02,900 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcDetailedActivityForPort58709 registered.
> > 2011-12-23 15:11:02,900 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcActivityForPort58709 registered.
> > 2011-12-23 15:11:02,903 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 58709: starting
> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 58709: starting
> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 58709: starting
> > 2011-12-23 15:11:02,904 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 58709: starting
> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker up at: localhost/127.0.0.1:58709
> > 2011-12-23 15:11:02,905 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> > 2011-12-23 15:11:02,908 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 58709: starting
> > 2011-12-23 15:11:02,958 INFO org.apache.hadoop.mapred.TaskTracker:  Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@7054f9f1
> > 2011-12-23 15:11:02,960 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> > 2011-12-23 15:11:02,964 INFO org.apache.hadoop.util.ProcessTree: setsid
> > exited with exit code 0
> > 2011-12-23 15:11:02,965 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-12-23 15:11:02,970 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
> > created with max memory = 10485760
> > 2011-12-23 15:11:02,976 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > ShuffleServerMetrics registered.
> > 2011-12-23 15:11:02,978 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
> > Opening the listener on 50060
> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50060
> > webServer.getConnectors()[0].getLocalPort() returned 50060
> > 2011-12-23 15:11:02,979 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> > to port 50060
> > 2011-12-23 15:11:02,979 INFO org.mortbay.log: jetty-6.1.26
> > 2011-12-23 15:11:03,257 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50060
> > 2011-12-23 15:11:03,257 INFO org.apache.hadoop.mapred.TaskTracker:
> > FILE_CACHE_SIZE for mapOutputServlet set to : 2000
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231432_0002 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231121_0001 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231312_0002 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231142_0002 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112230933_0002 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231357_0001 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231052_0002 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:03,264 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding
> > job_201112231426_0001 for user-log deletion with
> > retainTimeStamp:1324735862967
> > 2011-12-23 15:11:51,501 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201112231511_0001_m_000002_0 task's
> > state:UNASSIGNED
> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
> > launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots
> > 2011-12-23 15:11:51,505 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201112231511_0001_m_000002_0 which needs 1 slots
> > 2011-12-23 15:11:51,633 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> > 2011-12-23 15:11:51,635 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> > 2011-12-23 15:11:51,636 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> > 2011-12-23 15:11:51,685 INFO org.apache.hadoop.mapred.JobLocalizer:
> > Initializing user root on this TT.
> > 2011-12-23 15:11:51,720 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> > 2011-12-23 15:11:51,722 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> > 2011-12-23 15:11:51,723 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> > 2011-12-23 15:11:51,845 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.data.dir;  Ignoring.
> > 2011-12-23 15:11:51,847 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: dfs.name.dir;  Ignoring.
> > 2011-12-23 15:11:51,858 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201112231511_0001_m_-1653678851
> > 2011-12-23 15:11:51,859 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
> > jvm_201112231511_0001_m_-1653678851 spawned.
> > 2011-12-23 15:11:51,861 INFO org.apache.hadoop.mapred.TaskController:
> > Writing commands to
> > /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh
> > 2011-12-23 15:11:52,427 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
> > ID: jvm_201112231511_0001_m_-1653678851 given task:
> > attempt_201112231511_0001_m_000002_0
> > 2011-12-23 15:11:52,675 WARN org.apache.hadoop.mapred.DefaultTaskController:
> > Exit code from task is : 134
> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.DefaultTaskController:
> > Output from DefaultTaskController's launchTask follows:
> > 2011-12-23 15:11:52,675 INFO org.apache.hadoop.mapred.TaskController:
> >
> > 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of
> > tasks it ran: 0
> > 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner:
> > attempt_201112231511_0001_m_000002_0 : Child Error
> > java.io.IOException: Task process exit with nonzero status of 134.
> >         at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> >
> >
> >
> > I also checked the logs of the task :
> >
> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/stdout
> >
> >  A fatal error has been detected by the Java Runtime Environment:
> > #
> > #  SIGFPE (0x8) at pc=0x00002acfa3c5f82f, pid=19378, tid=1076017472
> > #
> > # JRE version: 6.0_30-b12
> > # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode
> > linux-amd64 compressed oops)
> > # Problematic frame:
> > # C  [ld-linux-x86-64.so.2+0x882f]  double+0xcf
> > #
> > # An error report file with more information is saved as:
> > #
> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log
> > #
> > # If you would like to submit a bug report, please visit:
> > #   http://java.sun.com/webapps/bugreport/crash.jsp
> > # The crash happened outside the Java Virtual Machine in native code.
> > # See problematic frame for where to report the bug.
> > ---------------
> >
> >
> > logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/syslog
> >
> > 2011-12-23 15:11:52,571 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xml:a
> > attempt to override final parameter: hadoop.tmp.dir;  Ignoring.
> >
> >
> > On the Job tracker :
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG:   host = yval4950.inetpsa.com/10.68.76.61
> > STARTUP_MSG:   args = []
> > STARTUP_MSG:   version = 0.20.203.0
> > STARTUP_MSG:   build =
> > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203
> > -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
> > ************************************************************/
> > 2011-12-23 15:11:01,401 INFO org.apache.hadoop.metrics2.impl.MetricsConfig:
> > loaded properties from hadoop-metrics2.properties
> > 2011-12-23 15:11:01,416 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > MetricsSystem,sub=Stats registered.
> > 2011-12-23 15:11:01,417 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> > at 10 second(s).
> > 2011-12-23 15:11:01,417 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system
> > started
> > 2011-12-23 15:11:01,727 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi
> > registered.
> > 2011-12-23 15:11:01,733 INFO
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Updating the current master key for generating delegation tokens
> > 2011-12-23 15:11:01,736 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2011-12-23 15:11:01,737 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2011-12-23 15:11:01,745 INFO
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Starting expired delegation token remover thread,
> > tokenRemoverScanInterval=60 min(s)
> > 2011-12-23 15:11:01,745 INFO
> > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
> > Updating the current master key for generating delegation tokens
> > 2011-12-23 15:11:01,801 INFO org.apache.hadoop.mapred.JobTracker: Starting
> > jobtracker with owner as root
> > 2011-12-23 15:11:01,836 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcDetailedActivityForPort9001 registered.
> > 2011-12-23 15:11:01,837 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > RpcActivityForPort9001 registered.
> > 2011-12-23 15:11:01,837 INFO org.apache.hadoop.ipc.Server: Starting
> > SocketReader
> > 2011-12-23 15:11:01,904 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-12-23 15:11:01,983 INFO org.apache.hadoop.http.HttpServer: Added global
> > filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
> > 2011-12-23 15:11:02,029 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
> > Opening the listener on 50030
> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50030
> > webServer.getConnectors()[0].getLocalPort() returned 50030
> > 2011-12-23 15:11:02,031 INFO org.apache.hadoop.http.HttpServer: Jetty bound
> > to port 50030
> > 2011-12-23 15:11:02,031 INFO org.mortbay.log: jetty-6.1.26
> > 2011-12-23 15:11:02,365 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50030
> > 2011-12-23 15:11:02,372 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm
> > registered.
> > 2011-12-23 15:11:02,373 INFO
> > org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source
> > JobTrackerMetrics registered.
> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
> > up at: 9001
> > 2011-12-23 15:11:02,374 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
> > webserver: 50030
> > 2011-12-23 15:11:02,517 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
> > up the system directory
> > 2011-12-23 15:11:02,576 INFO org.apache.hadoop.mapred.JobTracker: History
> > server being initialized in embedded mode
> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobHistoryServer:
> > Started job history server at: localhost:50030
> > 2011-12-23 15:11:02,581 INFO org.apache.hadoop.mapred.JobTracker: Job
> > History Server web address: localhost:50030
> > 2011-12-23 15:11:02,585 INFO
> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> > inactive
> > 2011-12-23 15:11:02,683 INFO org.apache.hadoop.mapred.JobTracker: Refreshing
> > hosts information
> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: Setting
> > the includes file to
> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader: Setting
> > the excludes file to
> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.util.HostsFileReader:
> > Refreshing hosts (include/exclude) list
> > 2011-12-23 15:11:02,711 INFO org.apache.hadoop.mapred.JobTracker:
> > Decommissioning 0 nodes
> > 2011-12-23 15:11:02,712 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-12-23 15:11:02,713 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 9001: starting
> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 9001: starting
> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 9001: starting
> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 9001: starting
> > 2011-12-23 15:11:02,714 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 9001: starting
> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 9001: starting
> > 2011-12-23 15:11:02,735 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 9001: starting
> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 9001: starting
> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 9001: starting
> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 9001: starting
> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.mapred.JobTracker: Starting
> > RUNNING
> > 2011-12-23 15:11:02,736 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 9001: starting
> > 2011-12-23 15:11:06,371 INFO org.apache.hadoop.net.NetworkTopology: Adding a
> > new node: /default-rack/ylal2960.inetpsa.com
> > 2011-12-23 15:11:06,373 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host
> > ylal2960.inetpsa.com
> > 2011-12-23 15:11:06,555 INFO org.apache.hadoop.net.NetworkTopology: Adding a
> > new node: /default-rack/ylal2950.inetpsa.com
> > 2011-12-23 15:11:06,556 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host
> > ylal2950.inetpsa.com
> > 2011-12-23 15:11:49,111 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to
> > override final parameter: hadoop.tmp.dir;  Ignoring.
> > 2011-12-23 15:11:49,113 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to
> > override final parameter: dfs.data.dir;  Ignoring.
> > 2011-12-23 15:11:49,114 WARN org.apache.hadoop.conf.Configuration:
> > /users/hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to
> > override final parameter: dfs.name.dir;  Ignoring.
> > 2011-12-23 15:11:49,120 INFO org.apache.hadoop.mapred.JobInProgress:
> > job_201112231511_0001: nMaps=1 nReduces=1 max=-1
> > 2011-12-23 15:11:49,124 INFO org.apache.hadoop.mapred.JobTracker: Job
> > job_201112231511_0001 added successfully for user 'root' to queue 'default'
> > 2011-12-23 15:11:49,126 INFO org.apache.hadoop.mapred.AuditLogger:
> > USER=root    IP=10.68.76.61  OPERATION=SUBMIT_JOB
> > TARGET=job_201112231511_0001    RESULT=SUCCESS
> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobTracker:
> > Initializing job_201112231511_0001
> > 2011-12-23 15:11:49,130 INFO org.apache.hadoop.mapred.JobInProgress:
> > Initializing job_201112231511_0001
> > 2011-12-23 15:11:49,299 INFO org.apache.hadoop.mapred.JobInProgress:
> > jobToken generated and stored with users keys in
> > /users/hadoop/mapred/system/job_201112231511_0001/jobToken
> > 2011-12-23 15:11:49,315 INFO org.apache.hadoop.mapred.JobInProgress: Input
> > size for job job_201112231511_0001 = 661630. Number of splits = 1
> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201112231511_0001_m_000000 has split on
> > node:/default-rack/ylal2960.inetpsa.com
> > 2011-12-23 15:11:49,316 INFO org.apache.hadoop.mapred.JobInProgress:
> > job_201112231511_0001 LOCALITY_WAIT_FACTOR=0.5
> > 2011-12-23 15:11:49,317 INFO org.apache.hadoop.mapred.JobInProgress: Job
> > job_201112231511_0001 initialized successfully with 1 map tasks and 1 reduce
> > tasks.
> > 2011-12-23 15:11:51,465 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip
> > task_201112231511_0001_m_000002, for tracker
> > 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709'
> > 2011-12-23 15:14:22,715 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_0 is 151247 ms debug.
> > 2011-12-23 15:17:43,117 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_0 is 351649 ms debug.
> > 2011-12-23 15:21:03,123 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_0 is 551655 ms debug.
> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.JobTracker: Lost
> > tracker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709'
> > 2011-12-23 15:24:23,821 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> > from attempt_201112231511_0001_m_000002_0: Lost task tracker:
> > tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709
> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker: Removing
> > task 'attempt_201112231511_0001_m_000002_0'
> > 2011-12-23 15:24:23,827 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_0 is 752357 ms debug.
> > 2011-12-23 15:24:23,828 INFO org.apache.hadoop.mapred.JobTracker: Launching
> > task attempt_201112231511_0001_m_000002_0 timed out.
> > 2011-12-23 15:24:25,676 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip
> > task_201112231511_0001_m_000002, for tracker
> > 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842'
> > 2011-12-23 15:27:43,831 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_1 is 198155 ms debug.
> > 2011-12-23 15:31:03,837 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_1 is 398161 ms debug.
> > 2011-12-23 15:34:23,843 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_1 is 598166 ms debug.
> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker:
> > attempt_201112231511_0001_m_000002_1 is 798172 ms debug.
> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.JobTracker: Launching
> > task attempt_201112231511_0001_m_000002_1 timed out.
> > 2011-12-23 15:37:43,849 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> > from attempt_201112231511_0001_m_000002_1: Error launching task
> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Lost
> > tracker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842'
> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.TaskInProgress: Error
> > from attempt_201112231511_0001_m_000002_1: Lost task tracker:
> > tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842
> > 2011-12-23 15:37:43,853 INFO org.apache.hadoop.mapred.JobTracker: Removing
> > task 'attempt_201112231511_0001_m_000002_1'
> >
> >
> >
> > Date: Fri, 23 Dec 2011 15:40:20 +0100
> > Subject: Re: Task process exit with nonzero status of 134
> > From: wget.null@googlemail.com
> > To: common-user@hadoop.apache.org
> >
> >
> > Hi,
> >
> > take a look into the logs for the failed attempt at your Tasktracker.
> > Also check the system logs with dmesg or /var/log/kern*. Could be a
> > syskill (segfault).
> >
> > - Alex
> >
> > On Fri, Dec 23, 2011 at 3:32 PM, anthony garnier <sokar6012@hotmail.com>
> > wrote:
> >>
> >> Hi folks,
> >>
> >> I've just done a fresh install of Hadoop, Namenode and datanode are up,
> >> Task/job Tracker also up, but when I run the Map reduce worcount exemple I
> >> got this error on Task tracker:
> >>
> >> 2011-12-23 15:11:52,679 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> >> jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of
> >> tasks it ran: 0
> >> 2011-12-23 15:11:52,681 WARN org.apache.hadoop.mapred.TaskRunner:
> >> attempt_201112231511_0001_m_000002_0 : Child Error
> >> java.io.IOException: Task process exit with nonzero status of 134.
> >>        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
> >>
> >> And job tracker stuck :
> >> # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/
> >> 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to process
> >> : 1
> >> 11/12/23 15:11:49 INFO mapred.JobClient: Running job:
> >> job_201112231511_0001
> >> 11/12/23 15:11:50 INFO mapred.JobClient:  map 0% reduce 0%
> >>
> >>
> >> I'running hadoop 0.20.203.0, java 1.6.0 rev 25
> >>
> >> I've done some googling, apparently the JVM crash hard (maybe Out of
> >> memory), does someone have any hint ?
> >>
> >> Regards,
> >>
> >> Anthony Garnier
> >> /DSIN/ASTI/ETSO
> >> IT Center
> >> PSA Peugeot Citroen
> >> Bessoncourt 90160
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Alexander Lorenz
> > http://mapredit.blogspot.com
> >
> > P Think of the environment: please don't print this email unless you
> > really need to.
> 
> 
> 
> -- 
> Alexander Lorenz
> http://mapredit.blogspot.com
> 
> P Think of the environment: please don't print this email unless you
> really need to.
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message