spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API
Date Thu, 12 Jan 2017 00:12:16 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15819623#comment-15819623
] 

Apache Spark commented on SPARK-19183:
--------------------------------------

User 'ericl' has created a pull request for this issue:
https://github.com/apache/spark/pull/16554

> Add deleteWithJob hook to internal commit protocol API
> ------------------------------------------------------
>
>                 Key: SPARK-19183
>                 URL: https://issues.apache.org/jira/browse/SPARK-19183
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>            Reporter: Eric Liang
>
> Currently in SQL we implement overwrites by calling fs.delete() directly on the original
data. This is not ideal since we the original files end up deleted even if the job aborts.
We should extend the commit protocol to allow file overwrites to be managed as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message