hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ratner, Alan S (IS)" <Alan.Rat...@ngc.com>
Subject Irreproducible Streaming Jobs
Date Fri, 11 Jun 2010 13:35:24 GMT
Our Hadoop streaming jobs are unpredictable.  The following illustrates
this problem.  I reset the system first to start with a clean slate
(deleting tmp files on master and all slaves, reformatting name node,
etc.).  I then ran a streaming job which gave the expected results.  I
then ran the exact same job again (using a different output file name)
and it bombed.  Curiously although the second run generated "no such
file or directory" errors in the log file but so did the first run.  I
suspect that our streaming job is doing something wrong (despite the
correct results) and corrupting the tmp files.  (These jobs were run on
40 servers configured as 1 master and 39 slaves.)  Any suggestions?

FIRST RUN 
ngc@hadoop1:~/hadoop-0.20.2$ bin/hadoop jar
contrib/streaming/hadoop-0.20.2-streaming.jar -file
streamingapps/Steg/SDMapper -mapper streamingapps/Steg/SDMapper -file
streamingapps/Steg/Test2 -reducer streamingapps/Steg/Test2 -input
ImageList -output StegOut1
packageJobJar: [streamingapps/Steg/SDMapper, streamingapps/Steg/Test2,
/tmp/hadoop-ngc/hadoop-unjar1378560104000865091/] []
/tmp/streamjob1574130777925793571.jar tmpDir=null
10/06/11 08:15:09 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/06/11 08:15:09 INFO streaming.StreamJob: getLocalDirs():
[/tmp/hadoop-ngc/mapred/local, /media/hd2/mapred/local,
/media/hd3/mapred/local, /media/hd4/mapred/local]
10/06/11 08:15:09 INFO streaming.StreamJob: Running job:
job_201006110802_0013
10/06/11 08:15:09 INFO streaming.StreamJob: To kill this job, run:
10/06/11 08:15:09 INFO streaming.StreamJob:
/home/ngc/hadoop-0.20.2/bin/../bin/hadoop job
-Dmapred.job.tracker=10.64.147.2:9002 -kill job_201006110802_0013
10/06/11 08:15:09 INFO streaming.StreamJob: Tracking URL:
http://hadoop1.aj.c2fse.northgrum.com:50030/jobdetails.jsp?jobid=job_201
006110802_0013
10/06/11 08:15:10 INFO streaming.StreamJob:  map 0%  reduce 0%
10/06/11 08:15:22 INFO streaming.StreamJob:  map 50%  reduce 0%
10/06/11 08:15:28 INFO streaming.StreamJob:  map 100%  reduce 0%
10/06/11 08:15:31 INFO streaming.StreamJob:  map 100%  reduce 17%
10/06/11 08:15:40 INFO streaming.StreamJob:  map 100%  reduce 100%
10/06/11 08:15:43 INFO streaming.StreamJob: Job complete:
job_201006110802_0013
10/06/11 08:15:43 INFO streaming.StreamJob: Output: StegOut1

SECOND RUN
ngc@hadoop1:~/hadoop-0.20.2$ bin/hadoop jar
contrib/streaming/hadoop-0.20.2-streaming.jar -file
streamingapps/Steg/SDMapper -mapper streamingapps/Steg/SDMapper -file
streamingapps/Steg/Test2 -reducer streamingapps/Steg/Test2 -input
ImageList -output StegOut2
packageJobJar: [streamingapps/Steg/SDMapper, streamingapps/Steg/Test2,
/tmp/hadoop-ngc/hadoop-unjar7292727069710530665/] []
/tmp/streamjob8749080748645718359.jar tmpDir=null
10/06/11 08:16:00 INFO mapred.FileInputFormat: Total input paths to
process : 1
10/06/11 08:16:00 INFO streaming.StreamJob: getLocalDirs():
[/tmp/hadoop-ngc/mapred/local, /media/hd2/mapred/local,
/media/hd3/mapred/local, /media/hd4/mapred/local]
10/06/11 08:16:00 INFO streaming.StreamJob: Running job:
job_201006110802_0016
10/06/11 08:16:00 INFO streaming.StreamJob: To kill this job, run:
10/06/11 08:16:00 INFO streaming.StreamJob:
/home/ngc/hadoop-0.20.2/bin/../bin/hadoop job
-Dmapred.job.tracker=10.64.147.2:9002 -kill job_201006110802_0016
10/06/11 08:16:00 INFO streaming.StreamJob: Tracking URL:
http://hadoop1.aj.c2fse.northgrum.com:50030/jobdetails.jsp?jobid=job_201
006110802_0016
10/06/11 08:16:01 INFO streaming.StreamJob:  map 0%  reduce 0%
10/06/11 08:16:19 INFO streaming.StreamJob:  map 50%  reduce 0%
10/06/11 08:16:28 INFO streaming.StreamJob:  map 50%  reduce 17%
10/06/11 08:16:32 INFO streaming.StreamJob:  map 100%  reduce 100%
10/06/11 08:16:32 INFO streaming.StreamJob: To kill this job, run:
10/06/11 08:16:32 INFO streaming.StreamJob:
/home/ngc/hadoop-0.20.2/bin/../bin/hadoop job
-Dmapred.job.tracker=10.64.147.2:9002 -kill job_201006110802_0016
10/06/11 08:16:32 INFO streaming.StreamJob: Tracking URL:
http://hadoop1.aj.c2fse.northgrum.com:50030/jobdetails.jsp?jobid=job_201
006110802_0016
10/06/11 08:16:32 ERROR streaming.StreamJob: Job not Successful!
10/06/11 08:16:32 INFO streaming.StreamJob: killJob...
Streaming Job Failed!

MAP LOG FILE OF SECOND RUN
Meta VERSION="1" .
Job JOBID="job_201006110802_0016"
JOBNAME="streamjob8749080748645718359\.jar" USER="ngc"
SUBMIT_TIME="1276258560297"
JOBCONF="hdfs://hadoop1\.aj\.c2fse\.northgrum\.com:9000/tmp/hadoop-ngc/m
apred/system/job_201006110802_0016/job\.xml" .
Job JOBID="job_201006110802_0016" JOB_PRIORITY="NORMAL" .
Job JOBID="job_201006110802_0016" LAUNCH_TIME="1276258560564"
TOTAL_MAPS="2" TOTAL_REDUCES="1" JOB_STATUS="PREP" .
Task TASKID="task_201006110802_0016_m_000003" TASK_TYPE="SETUP"
START_TIME="1276258560932" SPLITS="" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201006110802_0016_m_000003"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000003_0"
START_TIME="1276229924181"
TRACKER_NAME="tracker_hadoop6\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:49372" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201006110802_0016_m_000003"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000003_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229924886"
HOSTNAME="/default-rack/hadoop6\.aj\.c2fse\.northgrum\.com"
STATE_STRING="setup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201006110802_0016_m_000003" TASK_TYPE="SETUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258563937"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201006110802_0016" JOB_STATUS="RUNNING" .
Task TASKID="task_201006110802_0016_m_000000" TASK_TYPE="MAP"
START_TIME="1276258563937"
SPLITS="/default-rack/hadoop20\.aj\.c2fse\.northgrum\.com,/default-rack/
hadoop17\.aj\.c2fse\.northgrum\.com" .
Task TASKID="task_201006110802_0016_m_000001" TASK_TYPE="MAP"
START_TIME="1276258564747"
SPLITS="/default-rack/hadoop20\.aj\.c2fse\.northgrum\.com,/default-rack/
hadoop17\.aj\.c2fse\.northgrum\.com" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_0"
START_TIME="1276229927063"
TRACKER_NAME="tracker_hadoop6\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:49372" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_0"
TASK_STATUS="FAILED" FINISH_TIME="1276229931379"
HOSTNAME="hadoop6\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/tmp/hadoop-ngc/mapred/local/taskTracker/jobcache/job_201006110802_001
6/attempt_201006110802_0016_m_000000_0/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_0"
START_TIME="1276229819233"
TRACKER_NAME="tracker_hadoop8\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:35631" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_0"
TASK_STATUS="FAILED" FINISH_TIME="1276229823283"
HOSTNAME="hadoop8\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd3/mapred/local/taskTracker/jobcache/job_201006110802_0016/att
empt_201006110802_0016_m_000001_0/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_1"
START_TIME="1276230008685"
TRACKER_NAME="tracker_hadoop3\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:34294" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_1"
TASK_STATUS="FAILED" FINISH_TIME="1276230012735"
HOSTNAME="hadoop3\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd2/mapred/local/taskTracker/jobcache/job_201006110802_0016/att
empt_201006110802_0016_m_000000_1/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_1"
START_TIME="1276229520733"
TRACKER_NAME="tracker_hadoop23\.aj\.c2fse\.northgrum\.com:localhost/127\
.0\.0\.1:58025" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_1"
TASK_STATUS="FAILED" FINISH_TIME="1276229524784"
HOSTNAME="hadoop23\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/tmp/hadoop-ngc/mapred/local/taskTracker/jobcache/job_201006110802_001
6/attempt_201006110802_0016_m_000001_1/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_2"
START_TIME="1276229773743"
TRACKER_NAME="tracker_hadoop2\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:51829" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000000_2"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229774851"
HOSTNAME="/default-rack/hadoop2\.aj\.c2fse\.northgrum\.com"
STATE_STRING="Records R/W\=1/1"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(18)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(94)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(1)][(SPILLED_RECORDS)(Spilled
Records)(1)][(MAP_OUTPUT_BYTES)(Map output
bytes)(54)][(MAP_INPUT_BYTES)(Map input
bytes)(18)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(1)]}" .
Task TASKID="task_201006110802_0016_m_000000" TASK_TYPE="MAP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258579350"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(18)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(94)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(1)][(SPILLED_RECORDS)(Spilled
Records)(1)][(MAP_OUTPUT_BYTES)(Map output
bytes)(54)][(MAP_INPUT_BYTES)(Map input
bytes)(18)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(1)]}" .
Task TASKID="task_201006110802_0016_r_000000" TASK_TYPE="REDUCE"
START_TIME="1276258579351" SPLITS="" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_2"
START_TIME="1276316358506"
TRACKER_NAME="tracker_hadoop24\.aj\.c2fse\.northgrum\.com:localhost/127\
.0\.0\.1:38111" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_2"
TASK_STATUS="FAILED" FINISH_TIME="1276316362553"
HOSTNAME="hadoop24\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/tmp/hadoop-ngc/mapred/local/taskTracker/jobcache/job_201006110802_001
6/attempt_201006110802_0016_m_000001_2/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_3"
START_TIME="1276229596341"
TRACKER_NAME="tracker_hadoop19\.aj\.c2fse\.northgrum\.com:localhost/127\
.0\.0\.1:56345" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0016_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000001_3"
TASK_STATUS="FAILED" FINISH_TIME="1276229600392"
HOSTNAME="hadoop19\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd2/mapred/local/taskTracker/jobcache/job_201006110802_0016/att
empt_201006110802_0016_m_000001_3/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
Task TASKID="task_201006110802_0016_m_000001" TASK_TYPE="MAP"
TASK_STATUS="FAILED" FINISH_TIME="1276229600392"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd2/mapred/local/taskTracker/jobcache/job_201006110802_0016/att
empt_201006110802_0016_m_000001_3/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" TASK_ATTEMPT_ID="" .
Task TASKID="task_201006110802_0016_m_000002" TASK_TYPE="CLEANUP"
START_TIME="1276258589119" SPLITS="" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201006110802_0016_m_000002"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000002_0"
START_TIME="1276229602209"
TRACKER_NAME="tracker_hadoop19\.aj\.c2fse\.northgrum\.com:localhost/127\
.0\.0\.1:56345" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201006110802_0016_m_000002"
TASK_ATTEMPT_ID="attempt_201006110802_0016_m_000002_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229602877"
HOSTNAME="/default-rack/hadoop19\.aj\.c2fse\.northgrum\.com"
STATE_STRING="cleanup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201006110802_0016_m_000002" TASK_TYPE="CLEANUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258592124"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201006110802_0016" FINISH_TIME="1276258592124"
JOB_STATUS="FAILED" FINISHED_MAPS="1" FINISHED_REDUCES="0" .

MAP LOG FILE OF FIRST RUN
eta VERSION="1" .
Job JOBID="job_201006110802_0013"
JOBNAME="streamjob1574130777925793571\.jar" USER="ngc"
SUBMIT_TIME="1276258509559"
JOBCONF="hdfs://hadoop1\.aj\.c2fse\.northgrum\.com:9000/tmp/hadoop-ngc/m
apred/system/job_201006110802_0013/job\.xml" .
Job JOBID="job_201006110802_0013" JOB_PRIORITY="NORMAL" .
Job JOBID="job_201006110802_0013" LAUNCH_TIME="1276258509729"
TOTAL_MAPS="2" TOTAL_REDUCES="1" JOB_STATUS="PREP" .
Task TASKID="task_201006110802_0013_m_000003" TASK_TYPE="SETUP"
START_TIME="1276258509836" SPLITS="" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201006110802_0013_m_000003"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000003_0"
START_TIME="1276229873134"
TRACKER_NAME="tracker_hadoop6\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:49372" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="SETUP" TASKID="task_201006110802_0013_m_000003"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000003_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229873917"
HOSTNAME="/default-rack/hadoop6\.aj\.c2fse\.northgrum\.com"
STATE_STRING="setup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201006110802_0013_m_000003" TASK_TYPE="SETUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258512842"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201006110802_0013" JOB_STATUS="RUNNING" .
Task TASKID="task_201006110802_0013_m_000000" TASK_TYPE="MAP"
START_TIME="1276258512842"
SPLITS="/default-rack/hadoop20\.aj\.c2fse\.northgrum\.com,/default-rack/
hadoop17\.aj\.c2fse\.northgrum\.com" .
Task TASKID="task_201006110802_0013_m_000001" TASK_TYPE="MAP"
START_TIME="1276258512946"
SPLITS="/default-rack/hadoop20\.aj\.c2fse\.northgrum\.com,/default-rack/
hadoop17\.aj\.c2fse\.northgrum\.com" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_0"
START_TIME="1276229875959"
TRACKER_NAME="tracker_hadoop6\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:49372" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_0"
TASK_STATUS="FAILED" FINISH_TIME="1276229880125"
HOSTNAME="hadoop6\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/tmp/hadoop-ngc/mapred/local/taskTracker/jobcache/job_201006110802_001
3/attempt_201006110802_0013_m_000000_0/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000001_0"
START_TIME="1276229951632"
TRACKER_NAME="tracker_hadoop3\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:34294" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000001_0"
TASK_STATUS="FAILED" FINISH_TIME="1276229955681"
HOSTNAME="hadoop3\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd4/mapred/local/taskTracker/jobcache/job_201006110802_0013/att
empt_201006110802_0013_m_000001_0/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000001_1"
START_TIME="1276229716711"
TRACKER_NAME="tracker_hadoop2\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:51829" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000001"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000001_1"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229717868"
HOSTNAME="/default-rack/hadoop2\.aj\.c2fse\.northgrum\.com"
STATE_STRING="hdfs://hadoop1\.aj\.c2fse\.northgrum\.com:9000/user/ngc/Im
ageList/imagelist:9+9"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(10)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(38)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(0)][(SPILLED_RECORDS)(Spilled
Records)(0)][(MAP_OUTPUT_BYTES)(Map output
bytes)(0)][(MAP_INPUT_BYTES)(Map input
bytes)(0)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(0)]}" .
Task TASKID="task_201006110802_0013_m_000001" TASK_TYPE="MAP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258522269"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(10)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(38)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(0)][(SPILLED_RECORDS)(Spilled
Records)(0)][(MAP_OUTPUT_BYTES)(Map output
bytes)(0)][(MAP_INPUT_BYTES)(Map input
bytes)(0)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(0)]}" .
Task TASKID="task_201006110802_0013_r_000000" TASK_TYPE="REDUCE"
START_TIME="1276258522270" SPLITS="" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_1"
START_TIME="1276229957552"
TRACKER_NAME="tracker_hadoop3\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:34294" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_1"
TASK_STATUS="FAILED" FINISH_TIME="1276229961598"
HOSTNAME="hadoop3\.aj\.c2fse\.northgrum\.com"
ERROR="java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:354)
	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307)
	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 5 more
Caused by: java\.lang\.RuntimeException: Error in configuring object
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:93)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setConf(ReflectionUtils\.jav
a:64)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.newInstance(ReflectionUtils\
.java:117)
	at
org\.apache\.hadoop\.mapred\.MapRunner\.configure(MapRunner\.java:34)
	\.\.\. 10 more
Caused by: java\.lang\.reflect\.InvocationTargetException
	at sun\.reflect\.NativeMethodAccessorImpl\.invoke0(Native
Method)
	at
sun\.reflect\.NativeMethodAccessorImpl\.invoke(NativeMethodAccessorImpl\
.java:39)
	at
sun\.reflect\.DelegatingMethodAccessorImpl\.invoke(DelegatingMethodAcces
sorImpl\.java:25)
	at java\.lang\.reflect\.Method\.invoke(Method\.java:597)
	at
org\.apache\.hadoop\.util\.ReflectionUtils\.setJobConf(ReflectionUtils\.
java:88)
	\.\.\. 13 more
Caused by: java\.lang\.RuntimeException: configuration exception
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
82)
	at
org\.apache\.hadoop\.streaming\.PipeMapper\.configure(PipeMapper\.java:6
3)
	\.\.\. 18 more
Caused by: java\.io\.IOException: Cannot run program
\"/media/hd3/mapred/local/taskTracker/jobcache/job_201006110802_0013/att
empt_201006110802_0013_m_000000_1/work/\./SDMapper\":
java\.io\.IOException: error\=2, No such file or directory
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:460)
	at
org\.apache\.hadoop\.streaming\.PipeMapRed\.configure(PipeMapRed\.java:1
66)
	\.\.\. 19 more
Caused by: java\.io\.IOException: java\.io\.IOException: error\=2, No
such file or directory
	at java\.lang\.UNIXProcess\.<init>(UNIXProcess\.java:148)
	at java\.lang\.ProcessImpl\.start(ProcessImpl\.java:65)
	at java\.lang\.ProcessBuilder\.start(ProcessBuilder\.java:453)
	\.\.\. 20 more
" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_2"
START_TIME="1276229722583"
TRACKER_NAME="tracker_hadoop2\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:51829" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="MAP" TASKID="task_201006110802_0013_m_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000000_2"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229723735"
HOSTNAME="/default-rack/hadoop2\.aj\.c2fse\.northgrum\.com"
STATE_STRING="Records R/W\=1/1"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(18)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(94)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(1)][(SPILLED_RECORDS)(Spilled
Records)(1)][(MAP_OUTPUT_BYTES)(Map output
bytes)(54)][(MAP_INPUT_BYTES)(Map input
bytes)(18)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(1)]}" .
Task TASKID="task_201006110802_0013_m_000000" TASK_TYPE="MAP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258528279"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(HDFS_BYTES_READ)(HD
FS_BYTES_READ)(18)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(94)]}{(org\
.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(1)][(SPILLED_RECORDS)(Spilled
Records)(1)][(MAP_OUTPUT_BYTES)(Map output
bytes)(54)][(MAP_INPUT_BYTES)(Map input
bytes)(18)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(MAP_OUTPUT_RECORDS)(Map output records)(1)]}" .
ReduceAttempt TASK_TYPE="REDUCE"
TASKID="task_201006110802_0013_r_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_r_000000_0"
START_TIME="1276229719549"
TRACKER_NAME="tracker_hadoop2\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:51829" HTTP_PORT="50060" .
ReduceAttempt TASK_TYPE="REDUCE"
TASKID="task_201006110802_0013_r_000000"
TASK_ATTEMPT_ID="attempt_201006110802_0013_r_000000_0"
TASK_STATUS="SUCCESS" SHUFFLE_FINISHED="1276229731481"
SORT_FINISHED="1276229731538" FINISH_TIME="1276229734818"
HOSTNAME="/default-rack/hadoop2\.aj\.c2fse\.northgrum\.com"
STATE_STRING="Records R/W\=1/1 > reduce"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(FILE_BYTES_READ)(FI
LE_BYTES_READ)(62)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(62)][(HDFS_
BYTES_WRITTEN)(HDFS_BYTES_WRITTEN)(63)]}{(org\.apache\.hadoop\.mapred\.T
ask$Counter)(Map-Reduce Framework)[(REDUCE_INPUT_GROUPS)(Reduce input
groups)(1)][(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(REDUCE_SHUFFLE_BYTES)(Reduce shuffle
bytes)(6)][(REDUCE_OUTPUT_RECORDS)(Reduce output
records)(2)][(SPILLED_RECORDS)(Spilled
Records)(1)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(REDUCE_INPUT_RECORDS)(Reduce input records)(1)]}" .
Task TASKID="task_201006110802_0013_r_000000" TASK_TYPE="REDUCE"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258540288"
COUNTERS="{(FileSystemCounters)(FileSystemCounters)[(FILE_BYTES_READ)(FI
LE_BYTES_READ)(62)][(FILE_BYTES_WRITTEN)(FILE_BYTES_WRITTEN)(62)][(HDFS_
BYTES_WRITTEN)(HDFS_BYTES_WRITTEN)(63)]}{(org\.apache\.hadoop\.mapred\.T
ask$Counter)(Map-Reduce Framework)[(REDUCE_INPUT_GROUPS)(Reduce input
groups)(1)][(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(REDUCE_SHUFFLE_BYTES)(Reduce shuffle
bytes)(6)][(REDUCE_OUTPUT_RECORDS)(Reduce output
records)(2)][(SPILLED_RECORDS)(Spilled
Records)(1)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(REDUCE_INPUT_RECORDS)(Reduce input records)(1)]}" .
Task TASKID="task_201006110802_0013_m_000002" TASK_TYPE="CLEANUP"
START_TIME="1276258540289" SPLITS="" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201006110802_0013_m_000002"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000002_0"
START_TIME="1276229737575"
TRACKER_NAME="tracker_hadoop2\.aj\.c2fse\.northgrum\.com:localhost/127\.
0\.0\.1:51829" HTTP_PORT="50060" .
MapAttempt TASK_TYPE="CLEANUP" TASKID="task_201006110802_0013_m_000002"
TASK_ATTEMPT_ID="attempt_201006110802_0013_m_000002_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1276229738283"
HOSTNAME="/default-rack/hadoop2\.aj\.c2fse\.northgrum\.com"
STATE_STRING="cleanup"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Task TASKID="task_201006110802_0013_m_000002" TASK_TYPE="CLEANUP"
TASK_STATUS="SUCCESS" FINISH_TIME="1276258543294"
COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
Job JOBID="job_201006110802_0013" FINISH_TIME="1276258543294"
JOB_STATUS="SUCCESS" FINISHED_MAPS="2" FINISHED_REDUCES="1"
FAILED_MAPS="3" FAILED_REDUCES="0"
COUNTERS="{(org\.apache\.hadoop\.mapred\.JobInProgress$Counter)(Job
Counters )[(TOTAL_LAUNCHED_REDUCES)(Launched reduce
tasks)(1)][(RACK_LOCAL_MAPS)(Rack-local map
tasks)(5)][(TOTAL_LAUNCHED_MAPS)(Launched map
tasks)(5)]}{(FileSystemCounters)(FileSystemCounters)[(FILE_BYTES_READ)(F
ILE_BYTES_READ)(62)][(HDFS_BYTES_READ)(HDFS_BYTES_READ)(28)][(FILE_BYTES
_WRITTEN)(FILE_BYTES_WRITTEN)(194)][(HDFS_BYTES_WRITTEN)(HDFS_BYTES_WRIT
TEN)(63)]}{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(REDUCE_INPUT_GROUPS)(Reduce input
groups)(1)][(COMBINE_OUTPUT_RECORDS)(Combine output
records)(0)][(MAP_INPUT_RECORDS)(Map input
records)(1)][(REDUCE_SHUFFLE_BYTES)(Reduce shuffle
bytes)(6)][(REDUCE_OUTPUT_RECORDS)(Reduce output
records)(2)][(SPILLED_RECORDS)(Spilled
Records)(2)][(MAP_OUTPUT_BYTES)(Map output
bytes)(54)][(MAP_INPUT_BYTES)(Map input
bytes)(18)][(MAP_OUTPUT_RECORDS)(Map output
records)(1)][(COMBINE_INPUT_RECORDS)(Combine input
records)(0)][(REDUCE_INPUT_RECORDS)(Reduce input records)(1)]}" .

Mime
View raw message