spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fangshi Li (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-23815) Spark writer dynamic partition overwrite mode fails to write output on multi level partition
Date Thu, 29 Mar 2018 05:17:00 GMT
Fangshi Li created SPARK-23815:
----------------------------------

             Summary: Spark writer dynamic partition overwrite mode fails to write output
on multi level partition
                 Key: SPARK-23815
                 URL: https://issues.apache.org/jira/browse/SPARK-23815
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.3.0
            Reporter: Fangshi Li


Spark introduced new writer mode to overwrite only related partitions in SPARK-20236. While
we are using this feature in our production cluster, we found a bug when writing multi-level
partitions on HDFS.


A simple test case to reproduce this issue:
val df = Seq(("1","2","3")).toDF("col1", "col2","col3")
df.write.partitionBy("col1","col2").mode("overwrite").save("/my/hdfs/location")


If HDFS location "/my/hdfs/location" does not exist, there will be no output.


This seems to be caused by the job commit change in SPARK-20236 in HadoopMapReduceCommitProtocol.


In the commit job process, the output has been written into staging dir /my/hdfs/location/.spark-staging.xxx/col1=1/col2=2,
and then the code calls fs.rename to rename /my/hdfs/location/.spark-staging.xxx/col1=1/col2=2
to /my/hdfs/location/col1=1/col2=2. However, in our case the operation will fail on HDFS because
/my/hdfs/location/col1=1 does not exists. HDFS rename can not create directory for more than
one level. 

This does not happen in unit test covered with SPARK-20236 with local file system.


We are proposing a fix. When cleaning current partition dir /my/hdfs/location/col1=1/col2=2
before the rename op, if the delete op fails (because /my/hdfs/location/col1=1/col2=2 may
not exist), we call mkdirs op to create the parent dir /my/hdfs/location/col1=1 (if the parent
dir does not exist) so the following rename op can succeed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message