spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Allman <mich...@videoamp.com>
Subject reading/writing parquet decimal type
Date Sun, 12 Oct 2014 20:51:21 GMT
Hello,

I'm interested in reading/writing parquet SchemaRDDs that support the Parquet Decimal converted
type. The first thing I did was update the Spark parquet dependency to version 1.5.0, as this
version introduced support for decimals in parquet. However, conversion between the catalyst
decimal type and the parquet decimal type is complicated by the fact that the catalyst type
does not specify a decimal precision and scale but the parquet type requires them.

I'm wondering if perhaps we could add an optional precision and scale to the catalyst decimal
type? The catalyst decimal type would have unspecified precision and scale by default for
backwards compatibility, but users who want to serialize a SchemaRDD with decimal(s) to parquet
would have to narrow their decimal type(s) by specifying a precision and scale.

Thoughts?

Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message