pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4086) Fix Orc e2e tests for tez
Date Sat, 02 Aug 2014 00:27:38 GMT

     [ https://issues.apache.org/jira/browse/PIG-4086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-4086:
----------------------------

    Attachment: PIG-4086-2.patch

TEZ-1351 will give us a flush hook. Update the patch to reflect that. That should fix the
test once TEZ-1351 checked in.

> Fix Orc e2e tests for tez
> -------------------------
>
>                 Key: PIG-4086
>                 URL: https://issues.apache.org/jira/browse/PIG-4086
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.14.0
>
>         Attachments: PIG-4086-1.patch, PIG-4086-2.patch
>
>
> All Orc e2e tests fail on tez.
> There are two issue:
> 1. hivelibdir etc is not set in tez.conf
> 2. OrcStorage produce empty output file
> Digging into #2, the problem is in this code in PigProcessor:
> {code}
>                 if (fileOutput.isCommitRequired()) {
>                     fileOutput.commit();
>                 }
> {code}
> fileOutput.commit() invokes both RecordWriter.close() and committer.commitTask(). However,
OrcNewOutputFormate will generate output file only after RecordWriter.close (if the output
file is small), fileOutput.isCommitRequired will not detect this file, thus skip fileOutput.commit().
> Changing the code to invoke fileOutput.close explicitly fix the issue. fileOutput.commit
will invoke close again, but there is no side effect since close will check if it has been
already called.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message