spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyukjin Kwon (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-25709) Can't append to a data source V2 unless target already has schema information or fixed schema matching dataframe
Date Fri, 12 Oct 2018 01:48:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyukjin Kwon resolved SPARK-25709.
----------------------------------
    Resolution: Duplicate

> Can't append to a data source V2 unless target already has schema information or fixed
schema matching dataframe
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-25709
>                 URL: https://issues.apache.org/jira/browse/SPARK-25709
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Assaf Mendelson
>            Priority: Major
>
> Let's assume we have a writer data source which writes to a dynamic schema (e.g. writing
to a database table). If we have mode append but the table is missing then writing will fail.
>  
> The reason for this is that in the dataframeWriter save method we have val relation =
DataSourceV2Relation.create(source, options) which in turn contains: val reader = source.createReader(options,
userSpecifiedSchema)
> The problem is that this sets the schema from the reader portion of the data source.
If the data source can't infer the schema then we will fail to write (even thought the schema
is actually part of the dataframe).
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message