spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ritika Maheshwari (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-22241) Apache spark giving InvalidSchemaException: Cannot write a schema with an empty group: optional group element {
Date Wed, 11 Oct 2017 02:33:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-22241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ritika Maheshwari updated SPARK-22241:
--------------------------------------
    Description: 
I have a bean which has field of type Arraylist of Doubles. Then I do the following

Encoder<T> beanEncoder = Encoders.bean(jClass);
Dataset<T> df = spark.createDataset( Collections.singletonList((T)extractedObj),beanEncoder);
The schema generated says:

|-- pixelSpacing: array (nullable = true)
|-- element: struct (containsNull = true)

Now I try to save this Dataset as parquet 

df.write().mode(SaveMode.Append).parquet(jClass.getName()+"_parquet");

and I get the error Caused by: org.apache.parquet.schema.

InvalidSchemaException: Cannot write a schema with an empty group: optional group element
{}

Kindly direct how to specify an ArrayList of Strings or Doubles in the Bean passed to the
encoders so that the schema generated can be saved in paraquet



> Apache spark giving InvalidSchemaException: Cannot write a schema with an empty group:
optional group element {
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22241
>                 URL: https://issues.apache.org/jira/browse/SPARK-22241
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Ritika Maheshwari
>            Priority: Minor
>
> I have a bean which has field of type Arraylist of Doubles. Then I do the following
> Encoder<T> beanEncoder = Encoders.bean(jClass);
> Dataset<T> df = spark.createDataset( Collections.singletonList((T)extractedObj),beanEncoder);
> The schema generated says:
> |-- pixelSpacing: array (nullable = true)
> |-- element: struct (containsNull = true)
> Now I try to save this Dataset as parquet 
> df.write().mode(SaveMode.Append).parquet(jClass.getName()+"_parquet");
> and I get the error Caused by: org.apache.parquet.schema.
> InvalidSchemaException: Cannot write a schema with an empty group: optional group element
{}
> Kindly direct how to specify an ArrayList of Strings or Doubles in the Bean passed to
the encoders so that the schema generated can be saved in paraquet



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message