chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CHUKWA-132) Handle multiline output in Job History file
Date Tue, 14 Apr 2009 21:19:16 GMT

    [ https://issues.apache.org/jira/browse/CHUKWA-132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698939#action_12698939
] 

Jerome Boulon commented on CHUKWA-132:
--------------------------------------

Neither saving the log, nor process full log (Logs can be more than 100MB) are valid option.
Hadoop contains a parser for JobHistory, what are they doing? Similar code could be applied
on Chukwa side at the adaptor level.


> Handle multiline output in Job History file
> -------------------------------------------
>
>                 Key: CHUKWA-132
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-132
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: Data Processors
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Cheng
>            Priority: Blocker
>         Attachments: chukwa-132.patch
>
>
> When there are multi line output in the Job History file, the parser fails with exception
like this:
> MapAttempt TASK_TYPE="MAP" TASKID="task_200904060626_2141_m_000108" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000108_1"
START_TIME="1239190934835" TRACKER_NAME="tracker_kry50024\.inktomisearch\.com:localhost/127\.0\.0\.1:39507"
HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="MAP" TASKID="task_200904060626_2141_m_000108" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000108_1"
TASK_STATUS="FAILED" FINISH_TIME="1239190949062" HOSTNAME="kry50024\.inktomisearch\.com" ERROR="java\.io\.IOException:
MROutput/MRErrThread failed:java\.lang\.ArrayIndexOutOfBoundsException: -1
> 	at org\.apache\.hadoop\.mapred\.lib\.KeyFieldBasedPartitioner\.hashCode(KeyFieldBasedPartitioner\.java:95)
> 	at org\.apache\.hadoop\.mapred\.lib\.KeyFieldBasedPartitioner\.getPartition(KeyFieldBasedPartitioner\.java:87)
> 	at org\.apache\.hadoop\.mapred\.MapTask$MapOutputBuffer\.collect(MapTask\.java:801)
> 	at org\.apache\.hadoop\.streaming\.PipeMapRed$MROutputThread\.run(PipeMapRed\.java:378)
> 	at org\.apache\.hadoop\.streaming\.PipeMapper\.map(PipeMapper\.java:87)
> 	at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:50)
> 	at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36)
> 	at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:356)
> 	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
> 	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:156)
> " .
> MapAttempt TASK_TYPE="CLEANUP" TASKID="task_200904060626_2141_m_000197" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000197_0"
START_TIME="1239190961843" TRACKER_NAME="tracker_kry3083\.inktomisearch\.com:localhost/127\.0\.0\.1:60970"
HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="CLEANUP" TASKID="task_200904060626_2141_m_000197" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000197_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1239190963602" HOSTNAME="/74\.6\.135\.128/kry3083\.inktomisearch\.com"
STATE_STRING="cleanup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled
Records)(0)]}" .
> Task TASKID="task_200904060626_2141_m_000197" TASK_TYPE="CLEANUP" TASK_STATUS="SUCCESS"
FINISH_TIME="1239190963509" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
> Job JOBID="job_200904060626_2141" FINISH_TIME="1239190963510" JOB_STATUS="FAILED" FINISHED_MAPS="0"
FINISHED_REDUCES="0" .
> [cchunkException] :java.lang.StringIndexOutOfBoundsException: String index out of range:
-1
> 	at java.lang.String.substring(String.java:1938)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.JobLog$JobLogLine.<init>(JobLog.java:114)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.JobLog.parse(JobLog.java:39)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.AbstractProcessor.process(AbstractProcessor.java:90)
> 	at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:94)
> 	at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:60)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> [csource] :kry-jt1.red.ygrid.yahoo.com
> [ctags] :cluster="kryptonitered"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message