spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-21786) The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables with partition field(s)
Date Mon, 25 Dec 2017 07:45:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16303121#comment-16303121
] 

Apache Spark commented on SPARK-21786:
--------------------------------------

User 'fjh100456' has created a pull request for this issue:
https://github.com/apache/spark/pull/20076

> The 'spark.sql.parquet.compression.codec' configuration doesn't take effect on tables
with partition field(s)
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-21786
>                 URL: https://issues.apache.org/jira/browse/SPARK-21786
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Jinhua Fu
>
> For tables created like below,  'spark.sql.parquet.compression.codec' doesn't take any
effect when insert data. And because the default compression codec is 'uncompressed', if I
want to change the compression codec, I have to change it by 'set parquet.compression=xxxx'.
> Contrast,tables without any partition field will work normal with 'spark.sql.parquet.compression.codec',and
the default compression codec is 'snappy', but it seems 'parquet.compression' no longer in
effect.
> Should we use the ‘spark.sql.parquet.compression.codec’ configuration uniformly?
> ----------------------------------------------------------------------------------------------------
> CREATE TABLE Test_Parquet(provincecode int, citycode int, districtcode int)
> PARTITIONED BY (p_provincecode int)
> STORED AS PARQUET;
> INSERT OVERWRITE TABLE Test_Parquet select * from TableB;



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message