spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiao Li (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-21786) The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)
Date Sat, 06 Jan 2018 10:23:03 GMT

     [ https://issues.apache.org/jira/browse/SPARK-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xiao Li resolved SPARK-21786.
-----------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0

> The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables
with partition field(s)
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21786
>                 URL: https://issues.apache.org/jira/browse/SPARK-21786
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Jinhua Fu
>            Assignee: Jinhua Fu
>             Fix For: 2.3.0
>
>
> For tables created like below,  'spark.sql.parquet.compression.codec' doesn't take any
effect when insert data. And because the default compression codec is 'uncompressed', if I
want to change the compression codec, I have to change it by 'set parquet.compression=xxxx'.
> Contrast,tables without any partition field will work normal with 'spark.sql.parquet.compression.codec',and
the default compression codec is 'snappy', but it seems 'parquet.compression' no longer in
effect.
> Should we use the ‘spark.sql.parquet.compression.codec’ configuration uniformly?
> ----------------------------------------------------------------------------------------------------
> CREATE TABLE Test_Parquet(provincecode int, citycode int, districtcode int)
> PARTITIONED BY (p_provincecode int)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE Test_Parquet select * from TableB;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message