spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wenchen Fan (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-26188) Spark 2.4.0 Partitioning behavior breaks backwards compatibility
Date Fri, 30 Nov 2018 04:03:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-26188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Wenchen Fan resolved SPARK-26188.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 2.4.1
                   3.0.0

Issue resolved by pull request 23165
[https://github.com/apache/spark/pull/23165]

> Spark 2.4.0 Partitioning behavior breaks backwards compatibility
> ----------------------------------------------------------------
>
>                 Key: SPARK-26188
>                 URL: https://issues.apache.org/jira/browse/SPARK-26188
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Damien Doucet-Girard
>            Assignee: Gengliang Wang
>            Priority: Critical
>             Fix For: 3.0.0, 2.4.1
>
>
> My team uses spark to partition and output parquet files to amazon S3. We typically use
256 partitions, from 00 to ff.
> We've observed that in spark 2.3.2 and prior, it reads the partitions as strings by default.
However, in spark 2.4.0 and later, the type of each partition is inferred by default, and
partitions such as 00 become 0 and 4d become 4.0.
>  Here is a log sample of this behavior from one of our jobs:
>  2.4.0:
> {code:java}
> 18/11/27 14:02:27 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=00/part-00061-hashredacted.parquet,
range: 0-662, partition values: [0]
> 18/11/27 14:02:28 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet,
range: 0-662, partition values: [ef]
> 18/11/27 14:02:29 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet,
range: 0-662, partition values: [4a]
> 18/11/27 14:02:30 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=74/part-00180-hashredacted.parquet,
range: 0-662, partition values: [74]
> 18/11/27 14:02:32 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet,
range: 0-662, partition values: [f5]
> 18/11/27 14:02:33 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=50/part-00195-hashredacted.parquet,
range: 0-662, partition values: [50]
> 18/11/27 14:02:34 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=70/part-00054-hashredacted.parquet,
range: 0-662, partition values: [70]
> 18/11/27 14:02:35 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet,
range: 0-662, partition values: [b9]
> 18/11/27 14:02:37 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet,
range: 0-662, partition values: [d2]
> 18/11/27 14:02:38 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=51/part-00003-hashredacted.parquet,
range: 0-662, partition values: [51]
> 18/11/27 14:02:39 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=84/part-00135-hashredacted.parquet,
range: 0-662, partition values: [84]
> 18/11/27 14:02:40 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet,
range: 0-662, partition values: [b5]
> 18/11/27 14:02:41 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=88/part-00143-hashredacted.parquet,
range: 0-662, partition values: [88]
> 18/11/27 14:02:42 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet,
range: 0-662, partition values: [4.0]
> 18/11/27 14:02:43 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet,
range: 0-662, partition values: [ac]
> 18/11/27 14:02:44 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=24/part-00139-hashredacted.parquet,
range: 0-662, partition values: [24]
> 18/11/27 14:02:45 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet,
range: 0-662, partition values: [fd]
> 18/11/27 14:02:46 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=52/part-00033-hashredacted.parquet,
range: 0-662, partition values: [52]
> 18/11/27 14:02:47 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet,
range: 0-662, partition values: [ab]
> 18/11/27 14:02:48 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet,
range: 0-662, partition values: [f8]
> 18/11/27 14:02:49 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet,
range: 0-662, partition values: [7a]
> 18/11/27 14:02:50 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet,
range: 0-662, partition values: [ba]
> 18/11/27 14:02:51 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet,
range: 0-662, partition values: [2.0]
> 18/11/27 14:02:52 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=03/part-00099-hashredacted.parquet,
range: 0-662, partition values: [3]
> 18/11/27 14:02:53 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=57/part-00196-hashredacted.parquet,
range: 0-662, partition values: [57]
> 18/11/27 14:02:54 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=81/part-00122-hashredacted.parquet,
range: 0-662, partition values: [81]
> 18/11/27 14:02:55 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet,
range: 0-662, partition values: [6.0]
> 18/11/27 14:02:56 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=36/part-00154-hashredacted.parquet,
range: 0-662, partition values: [36]
> 18/11/27 14:02:57 INFO FileScanRDD: Reading File path: s3a://bucketnamereadacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet,
range: 0-662, partition values: [4b]{code}
> 2.3.2:
> {code:java}
> 18/11/27 14:09:00 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=60/part-00082-hashredacted.parquet,
range: 0-662, partition values: [60]
> 18/11/27 14:09:01 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=00/part-00061-hashredacted.parquet,
range: 0-662, partition values: [00]
> 18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ef/part-00034-hashredacted.parquet,
range: 0-662, partition values: [ef]
> 18/11/27 14:09:02 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4a/part-00151-hashredacted.parquet,
range: 0-662, partition values: [4a]
> 18/11/27 14:09:03 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=74/part-00180-hashredacted.parquet,
range: 0-662, partition values: [74]
> 18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f5/part-00156-hashredacted.parquet,
range: 0-662, partition values: [f5]
> 18/11/27 14:09:04 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=50/part-00195-hashredacted.parquet,
range: 0-662, partition values: [50]
> 18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=70/part-00054-hashredacted.parquet,
range: 0-662, partition values: [70]
> 18/11/27 14:09:05 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b9/part-00012-hashredacted.parquet,
range: 0-662, partition values: [b9]
> 18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=d2/part-00016-hashredacted.parquet,
range: 0-662, partition values: [d2]
> 18/11/27 14:09:06 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=51/part-00003-hashredacted.parquet,
range: 0-662, partition values: [51]
> 18/11/27 14:09:07 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=84/part-00135-hashredacted.parquet,
range: 0-662, partition values: [84]
> 18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=b5/part-00190-hashredacted.parquet,
range: 0-662, partition values: [b5]
> 18/11/27 14:09:08 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=88/part-00143-hashredacted.parquet,
range: 0-662, partition values: [88]
> 18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4d/part-00120-hashredacted.parquet,
range: 0-662, partition values: [4d]
> 18/11/27 14:09:09 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ac/part-00119-hashredacted.parquet,
range: 0-662, partition values: [ac]
> 18/11/27 14:09:10 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=24/part-00139-hashredacted.parquet,
range: 0-662, partition values: [24]
> 18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=fd/part-00167-hashredacted.parquet,
range: 0-662, partition values: [fd]
> 18/11/27 14:09:11 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=52/part-00033-hashredacted.parquet,
range: 0-662, partition values: [52]
> 18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ab/part-00083-hashredacted.parquet,
range: 0-662, partition values: [ab]
> 18/11/27 14:09:12 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=f8/part-00018-hashredacted.parquet,
range: 0-662, partition values: [f8]
> 18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=7a/part-00093-hashredacted.parquet,
range: 0-662, partition values: [7a]
> 18/11/27 14:09:13 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=ba/part-00020-hashredacted.parquet,
range: 0-662, partition values: [ba]
> 18/11/27 14:09:14 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=2d/part-00085-hashredacted.parquet,
range: 0-662, partition values: [2d]
> 18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=03/part-00099-hashredacted.parquet,
range: 0-662, partition values: [03]
> 18/11/27 14:09:15 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=57/part-00196-hashredacted.parquet,
range: 0-662, partition values: [57]
> 18/11/27 14:09:16 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=81/part-00122-hashredacted.parquet,
range: 0-662, partition values: [81]
> 18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=6d/part-00167-hashredacted.parquet,
range: 0-662, partition values: [6d]
> 18/11/27 14:09:17 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=36/part-00154-hashredacted.parquet,
range: 0-662, partition values: [36]
> 18/11/27 14:09:18 INFO FileScanRDD: Reading File path: s3a://bucketnameredacted/ddgirard/suffix=4b/part-00093-hashredacted.parquet,
range: 0-662, partition values: [4b]
> {code}
> After some investigation, we've isolated the issue to
>  [https://github.com/apache/spark/blob/02b510728c31b70e6035ad541bfcdc2b59dcd79a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L132-L136]
>   
> In the inferPartitioning method, 2.3.2 sets the type inference to false by default:
> {code:java}
> val spec = PartitioningUtils.parsePartitions(
>   leafDirs,
>   typeInference = false,
>   basePaths = basePaths,
>   timeZoneId = timeZoneId){code}
> However, in version 2.4.0, the typeInference flag has been replace with a config flag
> [https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PartitioningAwareFileIndex.scala#L129-L133]
>  
> {code:java}
> val inferredPartitionSpec = PartitioningUtils.parsePartitions(
>   leafDirs,
>   typeInference = sparkSession.sessionState.conf.partitionColumnTypeInferenceEnabled,
>   basePaths = basePaths,
>   timeZoneId = timeZoneId){code}
> And this conf's default value is true
> {code:java}
> val PARTITION_COLUMN_TYPE_INFERENCE =
> buildConf("spark.sql.sources.partitionColumnTypeInference.enabled")
> .doc("When true, automatically infer the data types for partitioned columns.")
> .booleanConf
> .createWithDefault(true){code}
> [https://github.com/apache/spark/blob/075447b3965489ffba4e6afb2b120880bc307505/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L636-L640]
>   
> I was wondering if a bug report would be appropriate to preserve backwards compatibility
and change the default conf value to false.
>  
>   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message