spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24194) HadoopFsRelation cannot overwrite a path that is also being read from
Date Mon, 07 May 2018 10:14:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16465732#comment-16465732
] 

Apache Spark commented on SPARK-24194:
--------------------------------------

User 'zheh12' has created a pull request for this issue:
https://github.com/apache/spark/pull/21257

> HadoopFsRelation cannot overwrite a path that is also being read from
> ---------------------------------------------------------------------
>
>                 Key: SPARK-24194
>                 URL: https://issues.apache.org/jira/browse/SPARK-24194
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.4.0
>         Environment: spark master
>            Reporter: yangz
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 2.4.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> When
> {code:java}
> INSERT OVERWRITE TABLE territory_count_compare select * from territory_count_compare
where shop_count!=real_shop_count
> {code}
> And territory_count_compare is a table with parquet, there will be a error 
> Cannot overwrite a path that is also being read from
>  
> And in file MetastoreDataSourceSuite.scala, there have a test case
>  
>  
> {code:java}
> table(tableName).write.mode(SaveMode.Overwrite).insertInto(tableName)
> {code}
>  
> But when the table territory_count_compare is a common hive table, there is no error. 
> So I think the reason is when insert overwrite into hadoopfs relation with static partition,
it first delete the partition in the output. But it should be the time when the job commited.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message