kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang Cheng (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KYLIN-3084) File not found Exception when processing union-all in TEZ mode
Date Tue, 05 Dec 2017 05:57:00 GMT

     [ https://issues.apache.org/jira/browse/KYLIN-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wang Cheng updated KYLIN-3084:
------------------------------
    Description: 
If hive.execution.engine=TEZ and hql contains union all, it causes exception like: file not
found when materializing the view or redistributing flat hive table.

Here is the reason:
http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union

i.e. "The Tez execution of UNION is entirely parallel &
the task-ids overlaps - so the files created have to have unique names.

But the total counts for "Map 1" and "Map 2" are only available as the job
runs, so they write to different dirs."


 

  was:
If hive.execution.engine=TEZ and hql contains union all, it causes exception like: file not
found when materializing the view or redistributing flat hive table during.

Here is the reason:
http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union

i.e. "The Tez execution of UNION is entirely parallel &
the task-ids overlaps - so the files created have to have unique names.

But the total counts for "Map 1" and "Map 2" are only available as the job
runs, so they write to different dirs."


 


> File not found Exception when processing union-all in TEZ mode
> --------------------------------------------------------------
>
>                 Key: KYLIN-3084
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3084
>             Project: Kylin
>          Issue Type: Bug
>            Reporter: Wang Cheng
>            Assignee: Wang Cheng
>            Priority: Minor
>
> If hive.execution.engine=TEZ and hql contains union all, it causes exception like: file
not found when materializing the view or redistributing flat hive table.
> Here is the reason:
> http://grokbase.com/t/hive/user/162r80a2g9/anyway-to-avoid-creating-subdirectories-by-insert-with-union
> i.e. "The Tez execution of UNION is entirely parallel &
> the task-ids overlaps - so the files created have to have unique names.
> But the total counts for "Map 1" and "Map 2" are only available as the job
> runs, so they write to different dirs."
>  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message