hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HIVE-277) Files created by redundant tasks that have been killed are taken into consideration
Date Sat, 07 Feb 2009 00:22:59 GMT

     [ https://issues.apache.org/jira/browse/HIVE-277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joydeep Sen Sarma resolved HIVE-277.
------------------------------------

    Resolution: Duplicate

> Files created by redundant tasks that have been killed are taken into consideration
> -----------------------------------------------------------------------------------
>
>                 Key: HIVE-277
>                 URL: https://issues.apache.org/jira/browse/HIVE-277
>             Project: Hadoop Hive
>          Issue Type: Bug
>         Environment: Hive with Hadoop 0.19.0 on a Linux cluster
>            Reporter: Rodrigo Schmidt
>            Priority: Minor
>
> Hadoop starts redundant tasks (mappers, by default) if it some particular mappers are
taking too long to be executed. When one of the redundant tasks finishes, the others are killed.
Killed tasks may generate output files (usually empty) and Hive is considering them as part
of the job output.
> In my case, I'm profiling one of the mappers in an INSERT OVERWRITE TABLE ... SELECT
(map-only) query, and the extra time added by the profiler makes hadoop start a second mapper
for the same part of the input. When one of these redundant mappers finishes, the other is
killed, and /tmp/hive-xxxx/xxxxxxxxx.10000.insclause-0/ will have the following files:
> _tmp.attempt_XX....XX_XXXX_m_000000_0
> attempt_XX....XX_XXXX_m_000000_0
> attempt_XX....XX_XXXX_m_000000_1
> attempt_XX....XX_XXXX_m_000000_2
> ...
> The first file is empty, but Hive considers it as part of the generated output and tries
to load it in the destination table, giving the following error message:
> Loading data to table output_table partition {p=p1}
> Failed with exception Cannot load text files into a table stored as SequenceFile.
> FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
> I'm not sure if the files generated by killed tasks will always be empty. If not, this
bug might render the data inconsistent.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message