spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nan Zhu (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-19788) DataStreamReader.option shall accept user-defined type
Date Wed, 01 Mar 2017 16:38:45 GMT
Nan Zhu created SPARK-19788:
-------------------------------

             Summary: DataStreamReader.option shall accept user-defined type
                 Key: SPARK-19788
                 URL: https://issues.apache.org/jira/browse/SPARK-19788
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 2.1.0
            Reporter: Nan Zhu


There are many other data sources which has very different configuration ways than Kafka,
FileSystem, etc. 

The expected type of the configuration entry passed to them might be a nested collection type,
e.g. Map[String, Map[String, String]], or even a user-defined type....(for example, the one
I am working on)

Right now, option can only accept String -> String/Boolean/Long/Double OR a complete Map[String,
String]...my suggestion is that we can accept Map[String, Any], and the type of 'parameters'
in SourceProvider.createSource can also be Map[String, Any], this will create much more flexibility
to the user....

The drawback is that, it is a breaking change ( we can mitigate this by deprecate the current
one, and progressively evolve to the new one if the proposal is accepted)

[~zsxwing] what do you think?




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message