Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 826337D1D for ; Tue, 27 Dec 2011 10:49:47 +0000 (UTC) Received: (qmail 25133 invoked by uid 500); 27 Dec 2011 10:49:43 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 25095 invoked by uid 500); 27 Dec 2011 10:49:43 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 25087 invoked by uid 99); 27 Dec 2011 10:49:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Dec 2011 10:49:43 +0000 X-ASF-Spam-Status: No, hits=2.3 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of sokar6012@hotmail.com designates 65.54.190.157 as permitted sender) Received: from [65.54.190.157] (HELO bay0-omc3-s19.bay0.hotmail.com) (65.54.190.157) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Dec 2011 10:49:35 +0000 Received: from BAY159-W21 ([65.54.190.189]) by bay0-omc3-s19.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Tue, 27 Dec 2011 02:49:13 -0800 Message-ID: Content-Type: multipart/alternative; boundary="_0f0ed19b-c5cb-43b0-ad60-5593d5881034_" X-Originating-IP: [81.255.178.243] From: anthony garnier To: , Subject: Re: Task process exit with nonzero status of 134 Date: Tue, 27 Dec 2011 10:49:12 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 27 Dec 2011 10:49:13.0584 (UTC) FILETIME=[2C0FF700:01CCC485] X-Virus-Checked: Checked by ClamAV on apache.org --_0f0ed19b-c5cb-43b0-ad60-5593d5881034_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi=2C I got Nothing in the dmesg I've checked the Tasktracker and this is what I got :=20 /************************************************************ STARTUP_MSG: Starting TaskTracker STARTUP_MSG: host =3D ylal2960.inetpsa.com/10.68.217.86 STARTUP_MSG: args =3D [] STARTUP_MSG: version =3D 0.20.203.0 STARTUP_MSG: build =3D http://svn.apache.org/repos/asf/hadoop/common/bran= ches/branch-0.20-security-203 -r 1099333=3B compiled by 'oom' on Wed May 4= 07:57:50 PDT 2011 ************************************************************/ 2011-12-23 15:11:02=2C275 INFO org.apache.hadoop.metrics2.impl.MetricsConfi= g: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:02=2C330 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source MetricsSystem=2Csub=3DStats registered. 2011-12-23 15:11:02=2C331 INFO org.apache.hadoop.metrics2.impl.MetricsSyste= mImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:02=2C331 INFO org.apache.hadoop.metrics2.impl.MetricsSyste= mImpl: TaskTracker metrics system started 2011-12-23 15:11:02=2C597 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source ugi registered. 2011-12-23 15:11:02=2C738 INFO org.mortbay.log: Logging to org.slf4j.impl.L= og4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:02=2C803 INFO org.apache.hadoop.http.HttpServer: Added glo= bal filtersafety (class=3Dorg.apache.hadoop.http.HttpServer$QuotingInputFil= ter) 2011-12-23 15:11:02=2C827 INFO org.apache.hadoop.mapred.TaskLogsTruncater: = Initializing logs' truncater with mapRetainSize=3D-1 and reduceRetainSize= =3D-1 2011-12-23 15:11:02=2C832 INFO org.apache.hadoop.mapred.TaskTracker: Starti= ng tasktracker with owner as root 2011-12-23 15:11:02=2C870 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source jvm registered. 2011-12-23 15:11:02=2C871 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source TaskTrackerMetrics registered. 2011-12-23 15:11:02=2C897 INFO org.apache.hadoop.ipc.Server: Starting Socke= tReader 2011-12-23 15:11:02=2C900 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source RpcDetailedActivityForPort58709 registered. 2011-12-23 15:11:02=2C900 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source RpcActivityForPort58709 registered. 2011-12-23 15:11:02=2C903 INFO org.apache.hadoop.ipc.Server: IPC Server Res= ponder: starting 2011-12-23 15:11:02=2C904 INFO org.apache.hadoop.ipc.Server: IPC Server lis= tener on 58709: starting 2011-12-23 15:11:02=2C904 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 0 on 58709: starting 2011-12-23 15:11:02=2C904 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 1 on 58709: starting 2011-12-23 15:11:02=2C904 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 2 on 58709: starting 2011-12-23 15:11:02=2C905 INFO org.apache.hadoop.mapred.TaskTracker: TaskTr= acker up at: localhost/127.0.0.1:58709 2011-12-23 15:11:02=2C905 INFO org.apache.hadoop.mapred.TaskTracker: Starti= ng tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 2011-12-23 15:11:02=2C908 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 3 on 58709: starting 2011-12-23 15:11:02=2C958 INFO org.apache.hadoop.mapred.TaskTracker: Using= MemoryCalculatorPlugin : org.apache.hadoop.util.LinuxMemoryCalculatorPlugi= n@7054f9f1 2011-12-23 15:11:02=2C960 INFO org.apache.hadoop.mapred.TaskTracker: Starti= ng thread: Map-events fetcher for all reduce tasks on tracker_ylal2960.inet= psa.com:localhost/127.0.0.1:58709 2011-12-23 15:11:02=2C964 INFO org.apache.hadoop.util.ProcessTree: setsid e= xited with exit code 0 2011-12-23 15:11:02=2C965 WARN org.apache.hadoop.mapred.TaskTracker: TaskTr= acker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is disabled. 2011-12-23 15:11:02=2C970 INFO org.apache.hadoop.mapred.IndexCache: IndexCa= che created with max memory =3D 10485760 2011-12-23 15:11:02=2C976 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source ShuffleServerMetrics registered. 2011-12-23 15:11:02=2C978 INFO org.apache.hadoop.http.HttpServer: Port retu= rned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Op= ening the listener on 50060 2011-12-23 15:11:02=2C979 INFO org.apache.hadoop.http.HttpServer: listener.= getLocalPort() returned 50060 webServer.getConnectors()[0].getLocalPort() r= eturned 50060 2011-12-23 15:11:02=2C979 INFO org.apache.hadoop.http.HttpServer: Jetty bou= nd to port 50060 2011-12-23 15:11:02=2C979 INFO org.mortbay.log: jetty-6.1.26 2011-12-23 15:11:03=2C257 INFO org.mortbay.log: Started SelectChannelConnec= tor@0.0.0.0:50060 2011-12-23 15:11:03=2C257 INFO org.apache.hadoop.mapred.TaskTracker: FILE_C= ACHE_SIZE for mapOutputServlet set to : 2000 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231432_0002 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231121_0001 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231312_0002 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231142_0002 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112230933_0002 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231357_0001 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231052_0002 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:03=2C264 INFO org.apache.hadoop.mapred.UserLogCleaner: Add= ing job_201112231426_0001 for user-log deletion with retainTimeStamp:132473= 5862967 2011-12-23 15:11:51=2C501 INFO org.apache.hadoop.mapred.TaskTracker: Launch= TaskAction (registerTask): attempt_201112231511_0001_m_000002_0 task's stat= e:UNASSIGNED 2011-12-23 15:11:51=2C505 INFO org.apache.hadoop.mapred.TaskTracker: Trying= to launch : attempt_201112231511_0001_m_000002_0 which needs 1 slots 2011-12-23 15:11:51=2C505 INFO org.apache.hadoop.mapred.TaskTracker: In Tas= kLauncher=2C current free slots : 2 and trying to launch attempt_2011122315= 11_0001_m_000002_0 which needs 1 slots 2011-12-23 15:11:51=2C633 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: hadoop.tmp.dir=3B Ignor= ing. 2011-12-23 15:11:51=2C635 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: dfs.data.dir=3B Ignorin= g. 2011-12-23 15:11:51=2C636 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: dfs.name.dir=3B Ignorin= g. 2011-12-23 15:11:51=2C685 INFO org.apache.hadoop.mapred.JobLocalizer: Initi= alizing user root on this TT. 2011-12-23 15:11:51=2C720 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: hadoop.tmp.dir=3B Ignor= ing. 2011-12-23 15:11:51=2C722 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: dfs.data.dir=3B Ignorin= g. 2011-12-23 15:11:51=2C723 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/ttprivate/taskTracker/root/jobcache/job_201112231511_0= 001/job.xml:a attempt to override final parameter: dfs.name.dir=3B Ignorin= g. 2011-12-23 15:11:51=2C845 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xm= l:a attempt to override final parameter: hadoop.tmp.dir=3B Ignoring. 2011-12-23 15:11:51=2C847 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xm= l:a attempt to override final parameter: dfs.data.dir=3B Ignoring. 2011-12-23 15:11:51=2C847 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xm= l:a attempt to override final parameter: dfs.name.dir=3B Ignoring. 2011-12-23 15:11:51=2C858 INFO org.apache.hadoop.mapred.JvmManager: In JvmR= unner constructed JVM ID: jvm_201112231511_0001_m_-1653678851 2011-12-23 15:11:51=2C859 INFO org.apache.hadoop.mapred.JvmManager: JVM Run= ner jvm_201112231511_0001_m_-1653678851 spawned. 2011-12-23 15:11:51=2C861 INFO org.apache.hadoop.mapred.TaskController: Wri= ting commands to /users/hadoop/mapred/local/ttprivate/taskTracker/root/jobc= ache/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/taskjvm.sh 2011-12-23 15:11:52=2C427 INFO org.apache.hadoop.mapred.TaskTracker: JVM wi= th ID: jvm_201112231511_0001_m_-1653678851 given task: attempt_201112231511= _0001_m_000002_0 2011-12-23 15:11:52=2C675 WARN org.apache.hadoop.mapred.DefaultTaskControll= er: Exit code from task is : 134 2011-12-23 15:11:52=2C675 INFO org.apache.hadoop.mapred.DefaultTaskControll= er: Output from DefaultTaskController's launchTask follows: 2011-12-23 15:11:52=2C675 INFO org.apache.hadoop.mapred.TaskController: 2011-12-23 15:11:52=2C679 INFO org.apache.hadoop.mapred.JvmManager: JVM : j= vm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of tas= ks it ran: 0 2011-12-23 15:11:52=2C681 WARN org.apache.hadoop.mapred.TaskRunner: attempt= _201112231511_0001_m_000002_0 : Child Error java.io.IOException: Task process exit with nonzero status of 134. at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) I also checked the logs of the task : logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/st= dout A fatal error has been detected by the Java Runtime Environment: # # SIGFPE (0x8) at pc=3D0x00002acfa3c5f82f=2C pid=3D19378=2C tid=3D10760174= 72 # # JRE version: 6.0_30-b12 # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.5-b03 mixed mode linux-amd= 64 compressed oops) # Problematic frame: # C [ld-linux-x86-64.so.2+0x882f] double+0xcf # # An error report file with more information is saved as: # /users/hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_000= 1/attempt_201112231511_0001_m_000002_0/work/hs_err_pid19378.log # # If you would like to submit a bug report=2C please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. --------------- logs/userlogs/job_201112231511_0001/attempt_201112231511_0001_m_000002_0/sy= slog 2011-12-23 15:11:52=2C571 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/taskTracker/root/jobcache/job_201112231511_0001/job.xm= l:a attempt to override final parameter: hadoop.tmp.dir=3B Ignoring. On the Job tracker :=20 STARTUP_MSG: Starting JobTracker STARTUP_MSG: host =3D yval4950.inetpsa.com/10.68.76.61 STARTUP_MSG: args =3D [] STARTUP_MSG: version =3D 0.20.203.0 STARTUP_MSG: build =3D http://svn.apache.org/repos/asf/hadoop/common/bran= ches/branch-0.20-security-203 -r 1099333=3B compiled by 'oom' on Wed May 4= 07:57:50 PDT 2011 ************************************************************/ 2011-12-23 15:11:01=2C401 INFO org.apache.hadoop.metrics2.impl.MetricsConfi= g: loaded properties from hadoop-metrics2.properties 2011-12-23 15:11:01=2C416 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source MetricsSystem=2Csub=3DStats registered. 2011-12-23 15:11:01=2C417 INFO org.apache.hadoop.metrics2.impl.MetricsSyste= mImpl: Scheduled snapshot period at 10 second(s). 2011-12-23 15:11:01=2C417 INFO org.apache.hadoop.metrics2.impl.MetricsSyste= mImpl: JobTracker metrics system started 2011-12-23 15:11:01=2C727 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source ugi registered. 2011-12-23 15:11:01=2C733 INFO org.apache.hadoop.security.token.delegation.= AbstractDelegationTokenSecretManager: Updating the current master key for g= enerating delegation tokens 2011-12-23 15:11:01=2C736 INFO org.apache.hadoop.mapred.JobTracker: Schedul= er configured with (memSizeForMapSlotOnJT=2C memSizeForReduceSlotOnJT=2C li= mitMaxMemForMapTasks=2C limitMaxMemForReduceTasks) (-1=2C -1=2C -1=2C -1) 2011-12-23 15:11:01=2C737 INFO org.apache.hadoop.util.HostsFileReader: Refr= eshing hosts (include/exclude) list 2011-12-23 15:11:01=2C745 INFO org.apache.hadoop.security.token.delegation.= AbstractDelegationTokenSecretManager: Starting expired delegation token rem= over thread=2C tokenRemoverScanInterval=3D60 min(s) 2011-12-23 15:11:01=2C745 INFO org.apache.hadoop.security.token.delegation.= AbstractDelegationTokenSecretManager: Updating the current master key for g= enerating delegation tokens 2011-12-23 15:11:01=2C801 INFO org.apache.hadoop.mapred.JobTracker: Startin= g jobtracker with owner as root 2011-12-23 15:11:01=2C836 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source RpcDetailedActivityForPort9001 registered. 2011-12-23 15:11:01=2C837 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source RpcActivityForPort9001 registered. 2011-12-23 15:11:01=2C837 INFO org.apache.hadoop.ipc.Server: Starting Socke= tReader 2011-12-23 15:11:01=2C904 INFO org.mortbay.log: Logging to org.slf4j.impl.L= og4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2011-12-23 15:11:01=2C983 INFO org.apache.hadoop.http.HttpServer: Added glo= bal filtersafety (class=3Dorg.apache.hadoop.http.HttpServer$QuotingInputFil= ter) 2011-12-23 15:11:02=2C029 INFO org.apache.hadoop.http.HttpServer: Port retu= rned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Op= ening the listener on 50030 2011-12-23 15:11:02=2C031 INFO org.apache.hadoop.http.HttpServer: listener.= getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() r= eturned 50030 2011-12-23 15:11:02=2C031 INFO org.apache.hadoop.http.HttpServer: Jetty bou= nd to port 50030 2011-12-23 15:11:02=2C031 INFO org.mortbay.log: jetty-6.1.26 2011-12-23 15:11:02=2C365 INFO org.mortbay.log: Started SelectChannelConnec= tor@0.0.0.0:50030 2011-12-23 15:11:02=2C372 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source jvm registered. 2011-12-23 15:11:02=2C373 INFO org.apache.hadoop.metrics2.impl.MetricsSourc= eAdapter: MBean for source JobTrackerMetrics registered. 2011-12-23 15:11:02=2C374 INFO org.apache.hadoop.mapred.JobTracker: JobTrac= ker up at: 9001 2011-12-23 15:11:02=2C374 INFO org.apache.hadoop.mapred.JobTracker: JobTrac= ker webserver: 50030 2011-12-23 15:11:02=2C517 INFO org.apache.hadoop.mapred.JobTracker: Cleanin= g up the system directory 2011-12-23 15:11:02=2C576 INFO org.apache.hadoop.mapred.JobTracker: History= server being initialized in embedded mode 2011-12-23 15:11:02=2C581 INFO org.apache.hadoop.mapred.JobHistoryServer: S= tarted job history server at: localhost:50030 2011-12-23 15:11:02=2C581 INFO org.apache.hadoop.mapred.JobTracker: Job His= tory Server web address: localhost:50030 2011-12-23 15:11:02=2C585 INFO org.apache.hadoop.mapred.CompletedJobStatusS= tore: Completed job store is inactive 2011-12-23 15:11:02=2C683 INFO org.apache.hadoop.mapred.JobTracker: Refresh= ing hosts information 2011-12-23 15:11:02=2C711 INFO org.apache.hadoop.util.HostsFileReader: Sett= ing the includes file to 2011-12-23 15:11:02=2C711 INFO org.apache.hadoop.util.HostsFileReader: Sett= ing the excludes file to 2011-12-23 15:11:02=2C711 INFO org.apache.hadoop.util.HostsFileReader: Refr= eshing hosts (include/exclude) list 2011-12-23 15:11:02=2C711 INFO org.apache.hadoop.mapred.JobTracker: Decommi= ssioning 0 nodes 2011-12-23 15:11:02=2C712 INFO org.apache.hadoop.ipc.Server: IPC Server Res= ponder: starting 2011-12-23 15:11:02=2C713 INFO org.apache.hadoop.ipc.Server: IPC Server lis= tener on 9001: starting 2011-12-23 15:11:02=2C714 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 0 on 9001: starting 2011-12-23 15:11:02=2C714 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 1 on 9001: starting 2011-12-23 15:11:02=2C714 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 2 on 9001: starting 2011-12-23 15:11:02=2C714 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 3 on 9001: starting 2011-12-23 15:11:02=2C735 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 4 on 9001: starting 2011-12-23 15:11:02=2C735 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 6 on 9001: starting 2011-12-23 15:11:02=2C736 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 7 on 9001: starting 2011-12-23 15:11:02=2C736 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 9 on 9001: starting 2011-12-23 15:11:02=2C736 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 5 on 9001: starting 2011-12-23 15:11:02=2C736 INFO org.apache.hadoop.mapred.JobTracker: Startin= g RUNNING 2011-12-23 15:11:02=2C736 INFO org.apache.hadoop.ipc.Server: IPC Server han= dler 8 on 9001: starting 2011-12-23 15:11:06=2C371 INFO org.apache.hadoop.net.NetworkTopology: Addin= g a new node: /default-rack/ylal2960.inetpsa.com 2011-12-23 15:11:06=2C373 INFO org.apache.hadoop.mapred.JobTracker: Adding = tracker tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709 to host ylal= 2960.inetpsa.com 2011-12-23 15:11:06=2C555 INFO org.apache.hadoop.net.NetworkTopology: Addin= g a new node: /default-rack/ylal2950.inetpsa.com 2011-12-23 15:11:06=2C556 INFO org.apache.hadoop.mapred.JobTracker: Adding = tracker tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842 to host ylal= 2950.inetpsa.com 2011-12-23 15:11:49=2C111 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to over= ride final parameter: hadoop.tmp.dir=3B Ignoring. 2011-12-23 15:11:49=2C113 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to over= ride final parameter: dfs.data.dir=3B Ignoring. 2011-12-23 15:11:49=2C114 WARN org.apache.hadoop.conf.Configuration: /users= /hadoop/mapred/local/jobTracker/job_201112231511_0001.xml:a attempt to over= ride final parameter: dfs.name.dir=3B Ignoring. 2011-12-23 15:11:49=2C120 INFO org.apache.hadoop.mapred.JobInProgress: job_= 201112231511_0001: nMaps=3D1 nReduces=3D1 max=3D-1 2011-12-23 15:11:49=2C124 INFO org.apache.hadoop.mapred.JobTracker: Job job= _201112231511_0001 added successfully for user 'root' to queue 'default' 2011-12-23 15:11:49=2C126 INFO org.apache.hadoop.mapred.AuditLogger: USER= =3Droot IP=3D10.68.76.61 OPERATION=3DSUBMIT_JOB TARGET=3Djob_2011122= 31511_0001 RESULT=3DSUCCESS 2011-12-23 15:11:49=2C130 INFO org.apache.hadoop.mapred.JobTracker: Initial= izing job_201112231511_0001 2011-12-23 15:11:49=2C130 INFO org.apache.hadoop.mapred.JobInProgress: Init= ializing job_201112231511_0001 2011-12-23 15:11:49=2C299 INFO org.apache.hadoop.mapred.JobInProgress: jobT= oken generated and stored with users keys in /users/hadoop/mapred/system/jo= b_201112231511_0001/jobToken 2011-12-23 15:11:49=2C315 INFO org.apache.hadoop.mapred.JobInProgress: Inpu= t size for job job_201112231511_0001 =3D 661630. Number of splits =3D 1 2011-12-23 15:11:49=2C316 INFO org.apache.hadoop.mapred.JobInProgress: tip:= task_201112231511_0001_m_000000 has split on node:/default-rack/ylal2960.in= etpsa.com 2011-12-23 15:11:49=2C316 INFO org.apache.hadoop.mapred.JobInProgress: job_= 201112231511_0001 LOCALITY_WAIT_FACTOR=3D0.5 2011-12-23 15:11:49=2C317 INFO org.apache.hadoop.mapred.JobInProgress: Job = job_201112231511_0001 initialized successfully with 1 map tasks and 1 reduc= e tasks. 2011-12-23 15:11:51=2C465 INFO org.apache.hadoop.mapred.JobTracker: Adding = task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_0' to tip task_2011122= 31511_0001_m_000002=2C for tracker 'tracker_ylal2960.inetpsa.com:localhost/= 127.0.0.1:58709' 2011-12-23 15:14:22=2C715 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_0 is 151247 ms debug. 2011-12-23 15:17:43=2C117 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_0 is 351649 ms debug. 2011-12-23 15:21:03=2C123 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_0 is 551655 ms debug. 2011-12-23 15:24:23=2C821 INFO org.apache.hadoop.mapred.JobTracker: Lost tr= acker 'tracker_ylal2960.inetpsa.com:localhost/127.0.0.1:58709' 2011-12-23 15:24:23=2C821 INFO org.apache.hadoop.mapred.TaskInProgress: Err= or from attempt_201112231511_0001_m_000002_0: Lost task tracker: tracker_yl= al2960.inetpsa.com:localhost/127.0.0.1:58709 2011-12-23 15:24:23=2C827 INFO org.apache.hadoop.mapred.JobTracker: Removin= g task 'attempt_201112231511_0001_m_000002_0' 2011-12-23 15:24:23=2C827 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_0 is 752357 ms debug. 2011-12-23 15:24:23=2C828 INFO org.apache.hadoop.mapred.JobTracker: Launchi= ng task attempt_201112231511_0001_m_000002_0 timed out. 2011-12-23 15:24:25=2C676 INFO org.apache.hadoop.mapred.JobTracker: Adding = task (JOB_SETUP) 'attempt_201112231511_0001_m_000002_1' to tip task_2011122= 31511_0001_m_000002=2C for tracker 'tracker_ylal2950.inetpsa.com:localhost/= 127.0.0.1:39842' 2011-12-23 15:27:43=2C831 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_1 is 198155 ms debug. 2011-12-23 15:31:03=2C837 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_1 is 398161 ms debug. 2011-12-23 15:34:23=2C843 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_1 is 598166 ms debug. 2011-12-23 15:37:43=2C849 INFO org.apache.hadoop.mapred.JobTracker: attempt= _201112231511_0001_m_000002_1 is 798172 ms debug. 2011-12-23 15:37:43=2C849 INFO org.apache.hadoop.mapred.JobTracker: Launchi= ng task attempt_201112231511_0001_m_000002_1 timed out. 2011-12-23 15:37:43=2C849 INFO org.apache.hadoop.mapred.TaskInProgress: Err= or from attempt_201112231511_0001_m_000002_1: Error launching task 2011-12-23 15:37:43=2C853 INFO org.apache.hadoop.mapred.JobTracker: Lost tr= acker 'tracker_ylal2950.inetpsa.com:localhost/127.0.0.1:39842' 2011-12-23 15:37:43=2C853 INFO org.apache.hadoop.mapred.TaskInProgress: Err= or from attempt_201112231511_0001_m_000002_1: Lost task tracker: tracker_yl= al2950.inetpsa.com:localhost/127.0.0.1:39842 2011-12-23 15:37:43=2C853 INFO org.apache.hadoop.mapred.JobTracker: Removin= g task 'attempt_201112231511_0001_m_000002_1' Date: Fri=2C 23 Dec 2011 15:40:20 +0100 Subject: Re: Task process exit with nonzero status of 134 From: wget.null@googlemail.com To: common-user@hadoop.apache.org Hi=2C =20 take a look into the logs for the failed attempt at your Tasktracker. Also check the system logs with dmesg or /var/log/kern*. Could be a syskill (segfault). =20 - Alex =20 On Fri=2C Dec 23=2C 2011 at 3:32 PM=2C anthony garnier wrote: > > Hi folks=2C > > I've just done a fresh install of Hadoop=2C Namenode and datanode are up= =2C Task/job Tracker also up=2C but when I run the Map reduce worcount exem= ple I got this error on Task tracker: > > 2011-12-23 15:11:52=2C679 INFO org.apache.hadoop.mapred.JvmManager: JVM := jvm_201112231511_0001_m_-1653678851 exited with exit code 134. Number of t= asks it ran: 0 > 2011-12-23 15:11:52=2C681 WARN org.apache.hadoop.mapred.TaskRunner: attem= pt_201112231511_0001_m_000002_0 : Child Error > java.io.IOException: Task process exit with nonzero status of 134. > at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258) > > And job tracker stuck : > # hadoop jar hadoop-examples*.jar wordcount input/test.txt output/ > 11/12/23 15:11:48 INFO input.FileInputFormat: Total input paths to proces= s : 1 > 11/12/23 15:11:49 INFO mapred.JobClient: Running job: job_201112231511_00= 01 > 11/12/23 15:11:50 INFO mapred.JobClient: map 0% reduce 0% > > > I'running hadoop 0.20.203.0=2C java 1.6.0 rev 25 > > I've done some googling=2C apparently the JVM crash hard (maybe Out of me= mory)=2C does someone have any hint ? > > Regards=2C > > Anthony Garnier > /DSIN/ASTI/ETSO > IT Center > PSA Peugeot Citroen > Bessoncourt 90160 > > > > =20 =20 =20 --=20 Alexander Lorenz http://mapredit.blogspot.com =20 P Think of the environment: please don't print this email unless you really need to. = --_0f0ed19b-c5cb-43b0-ad60-5593d5881034_--