hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Kendall <mkend...@justin.tv>
Subject Re: what does it mean when a job fails at 100%?
Date Fri, 13 Nov 2009 23:19:20 GMT
oh and just fyi this is the only failed task.  everything else works just
fine.  maybe the data copied over incorrectly or was malformed...  /me
checks

On Fri, Nov 13, 2009 at 3:03 PM, Mike Kendall <mkendall@justin.tv> wrote:

> Hmm..  let's collect some error messages.  looks like the same task failed
> 4 times...  is there a way that i can get better logs about this task?
>
> MapAttempt TASK_TYPE="MAP" TASKID="task_200911131440_0001_m_000307"
> TASK_ATTEMPT_ID="attempt_200911131440_0001_m_000307_0" TASK_STATUS="FAILED"
> FINISH_TIME="1258123661159" HOSTNAME="hadoop3\.justin\.tv"
> ERROR="java\.lang\.RuntimeException: PipeMapRed\.waitOutputThreads():
> subprocess failed with code 1
>         at
> org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311)
>
> MapAttempt TASK_TYPE="MAP" TASKID="task_200911131440_0001_m_000307"
> TASK_ATTEMPT_ID="attempt_200911131440_0001_m_000307_1" TASK_STATUS="FAILED"
> FINISH_TIME="1258123852424" HOSTNAME="hadoop1\.justin\.tv"
> ERROR="java\.lang\.RuntimeException: PipeMapRed\.waitOutputThreads():
> subprocess failed with code 1
>         at
> org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311)
>
> MapAttempt TASK_TYPE="MAP" TASKID="task_200911131440_0001_m_000307"
> TASK_ATTEMPT_ID="attempt_200911131440_0001_m_000307_2" TASK_STATUS="FAILED"
> FINISH_TIME="1258123725938" HOSTNAME="hadoop4\.justin\.tv"
> ERROR="java\.lang\.RuntimeException: PipeMapRed\.waitOutputThreads():
> subprocess failed with code 1        at
> org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311)
>
>
> MapAttempt TASK_TYPE="MAP" TASKID="task_200911131440_0001_m_000307"
> TASK_ATTEMPT_ID="attempt_200911131440_0001_m_000307_3" TASK_STATUS="FAILED"
> FINISH_T
> IME="1258123756980" HOSTNAME="hadoop2\.justin\.tv"
> ERROR="java\.lang\.RuntimeException: PipeMapRed\.waitOutputThreads():
> subprocess failed with code 1
>         at
> org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311)
>
> Task TASKID="task_200911131440_0001_m_000307" TASK_TYPE="MAP"
> TASK_STATUS="FAILED" FINISH_TIME="1258123756980"
> ERROR="java\.lang\.RuntimeException: Pipe
> MapRed\.waitOutputThreads(): subprocess failed with code 1        at
> org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311)
>
>
> On Fri, Nov 13, 2009 at 2:16 PM, Ashutosh Chauhan <
> ashutosh.chauhan@gmail.com> wrote:
>
>> Hi Mike,
>>
>> This % reported represents % of records read by framework not % of records
>> processed. So, for sake of example lets say you only have one record in
>> the
>> data, framework will report 100% as soon as it is read even though you
>> might
>> be doing lot of processing on that record and that processing is still
>> going
>> on. Second, there can be floating point errors here so e.g., after reading
>> 9991 records out of total 10000 for the split, counter will say 100% while
>> some records are still untouched. Lastly, if you are using close() method,
>> your task might be failing there and framework will report 100% before
>> that.
>> I am not expert on counters, so you may to hear from others before
>> believing
>> what I am saying :)
>>
>> Thanks,
>> Ashutosh
>>
>> On Fri, Nov 13, 2009 at 17:15, brien colwell <xcolwell@gmail.com> wrote:
>>
>> > It could be that the result can't be written to HDFS. Is there any hint
>> in
>> > the log? I recently encountered this behavior when writing many files
>> back.
>> >
>> >
>> >
>> > Mike Kendall wrote:
>> >
>> >> title says it all..  this isn't the first job i've written either.
>>  very
>> >> confused.
>> >>
>> >>
>> >>
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message