spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nan Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-19788) DataStreamReader/DataStreamWriter.option shall accept user-defined type
Date Wed, 01 Mar 2017 16:45:46 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890522#comment-15890522
] 

Nan Zhu edited comment on SPARK-19788 at 3/1/17 4:45 PM:
---------------------------------------------------------

another drawback is that it might look like incompatible with DataFrameReader/DataFrameWriter
(we can also change that?)


was (Author: codingcat):
another drawback is that it might look like incompatible with DataFrameReader (we can also
change that?)

> DataStreamReader/DataStreamWriter.option shall accept user-defined type
> -----------------------------------------------------------------------
>
>                 Key: SPARK-19788
>                 URL: https://issues.apache.org/jira/browse/SPARK-19788
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 2.1.0
>            Reporter: Nan Zhu
>
> There are many other data sources/sinks which has very different configuration ways than
Kafka, FileSystem, etc. 
> The expected type of the configuration entry passed to them might be a nested collection
type, e.g. Map[String, Map[String, String]], or even a user-defined type....(for example,
the one I am working on)
> Right now, option can only accept String -> String/Boolean/Long/Double OR a complete
Map[String, String]...my suggestion is that we can accept Map[String, Any], and the type of
'parameters' in SourceProvider.createSource can also be Map[String, Any], this will create
much more flexibility to the user....
> The drawback is that, it is a breaking change ( we can mitigate this by deprecate the
current one, and progressively evolve to the new one if the proposal is accepted)
> [~zsxwing] what do you think?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message