chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheng (JIRA)" <j...@apache.org>
Subject [jira] Updated: (CHUKWA-132) Handle multiline output in Job History file
Date Tue, 14 Apr 2009 21:09:14 GMT

     [ https://issues.apache.org/jira/browse/CHUKWA-132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cheng updated CHUKWA-132:
-------------------------

    Status: Patch Available  (was: Open)

 Job logs could be split into multiple lines.  New code monitors input lines. If input recordEntry
doesn't end with '"' or '" .', save the log and wait for the next log. Otherwise process the
full log. 


> Handle multiline output in Job History file
> -------------------------------------------
>
>                 Key: CHUKWA-132
>                 URL: https://issues.apache.org/jira/browse/CHUKWA-132
>             Project: Hadoop Chukwa
>          Issue Type: Bug
>          Components: Data Processors
>         Environment: Redhat EL 5.1, Java 6
>            Reporter: Eric Yang
>            Assignee: Cheng
>            Priority: Blocker
>         Attachments: chukwa-132.patch
>
>
> When there are multi line output in the Job History file, the parser fails with exception
like this:
> MapAttempt TASK_TYPE="MAP" TASKID="task_200904060626_2141_m_000108" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000108_1"
START_TIME="1239190934835" TRACKER_NAME="tracker_kry50024\.inktomisearch\.com:localhost/127\.0\.0\.1:39507"
HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="MAP" TASKID="task_200904060626_2141_m_000108" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000108_1"
TASK_STATUS="FAILED" FINISH_TIME="1239190949062" HOSTNAME="kry50024\.inktomisearch\.com" ERROR="java\.io\.IOException:
MROutput/MRErrThread failed:java\.lang\.ArrayIndexOutOfBoundsException: -1
> 	at org\.apache\.hadoop\.mapred\.lib\.KeyFieldBasedPartitioner\.hashCode(KeyFieldBasedPartitioner\.java:95)
> 	at org\.apache\.hadoop\.mapred\.lib\.KeyFieldBasedPartitioner\.getPartition(KeyFieldBasedPartitioner\.java:87)
> 	at org\.apache\.hadoop\.mapred\.MapTask$MapOutputBuffer\.collect(MapTask\.java:801)
> 	at org\.apache\.hadoop\.streaming\.PipeMapRed$MROutputThread\.run(PipeMapRed\.java:378)
> 	at org\.apache\.hadoop\.streaming\.PipeMapper\.map(PipeMapper\.java:87)
> 	at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:50)
> 	at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36)
> 	at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:356)
> 	at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:305)
> 	at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:156)
> " .
> MapAttempt TASK_TYPE="CLEANUP" TASKID="task_200904060626_2141_m_000197" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000197_0"
START_TIME="1239190961843" TRACKER_NAME="tracker_kry3083\.inktomisearch\.com:localhost/127\.0\.0\.1:60970"
HTTP_PORT="50060" .
> MapAttempt TASK_TYPE="CLEANUP" TASKID="task_200904060626_2141_m_000197" TASK_ATTEMPT_ID="attempt_200904060626_2141_m_000197_0"
TASK_STATUS="SUCCESS" FINISH_TIME="1239190963602" HOSTNAME="/74\.6\.135\.128/kry3083\.inktomisearch\.com"
STATE_STRING="cleanup" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce Framework)[(SPILLED_RECORDS)(Spilled
Records)(0)]}" .
> Task TASKID="task_200904060626_2141_m_000197" TASK_TYPE="CLEANUP" TASK_STATUS="SUCCESS"
FINISH_TIME="1239190963509" COUNTERS="{(org\.apache\.hadoop\.mapred\.Task$Counter)(Map-Reduce
Framework)[(SPILLED_RECORDS)(Spilled Records)(0)]}" .
> Job JOBID="job_200904060626_2141" FINISH_TIME="1239190963510" JOB_STATUS="FAILED" FINISHED_MAPS="0"
FINISHED_REDUCES="0" .
> [cchunkException] :java.lang.StringIndexOutOfBoundsException: String index out of range:
-1
> 	at java.lang.String.substring(String.java:1938)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.JobLog$JobLogLine.<init>(JobLog.java:114)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.JobLog.parse(JobLog.java:39)
> 	at org.apache.hadoop.chukwa.extraction.demux.processor.mapper.AbstractProcessor.process(AbstractProcessor.java:90)
> 	at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:94)
> 	at org.apache.hadoop.chukwa.extraction.demux.Demux$MapClass.map(Demux.java:60)
> 	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:47)
> 	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:227)
> 	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2210)
> [csource] :kry-jt1.red.ygrid.yahoo.com
> [ctags] :cluster="kryptonitered"

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message