flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8538) Add a Kafka table source factory with JSON format support
Date Thu, 22 Feb 2018 16:35:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373037#comment-16373037
] 

ASF GitHub Bot commented on FLINK-8538:
---------------------------------------

GitHub user twalthr opened a pull request:

    https://github.com/apache/flink/pull/5564

    [FLINK-8538] [table] Add a Kafka table source factory with JSON format

    ## What is the purpose of the change
    
    This PR is a continuation of #5505. Since this is the first connector with format, time
attributes, and table source factory, I encountered many inconsistencies that I try to fix
in this PR.
    
    
    ## Brief change log
    
    - Change property `connector.version` to `connector.property-version` in order to use
`version` for things like Kafka
    - Add more utility functions for better Java<->Scala interoperability
    - Add full rowtime support
    - Derive JSON mapping from schema
    - Derive schema from source if required
    
    
    ## Verifying this change
    
    Various unit tests implemented.
    
    ## Does this pull request potentially affect one of the following parts:
    
      - Dependencies (does it add or upgrade a dependency): yes
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
      - The serializers: no
      - The runtime per-record code paths (performance sensitive): no
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: no
      - The S3 file system connector: no
    
    ## Documentation
    
      - Does this pull request introduce a new feature? yes
      - If yes, how is the feature documented? JavaDocs/ScalaDocs


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/twalthr/flink FLINK-8538

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/5564.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5564
    
----
commit bc6af3a27155651622149cf4a74e52118773471a
Author: Xingcan Cui <xingcanc@...>
Date:   2018-02-12T10:11:36Z

    [FLINK-8538][table]Add a Kafka table source factory with JSON format support

commit 233a8ff0b88359dd702cfd6e447e0585e411c8c2
Author: Timo Walther <twalthr@...>
Date:   2018-02-19T12:35:45Z

    [FLINK-8538] [table] Improve unified table sources

----


> Add a Kafka table source factory with JSON format support
> ---------------------------------------------------------
>
>                 Key: FLINK-8538
>                 URL: https://issues.apache.org/jira/browse/FLINK-8538
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Table API &amp; SQL
>            Reporter: Timo Walther
>            Assignee: Xingcan Cui
>            Priority: Major
>
> Similar to CSVTableSourceFactory a Kafka table source factory for JSON should be added.
This issue includes improving the existing JSON descriptor with validation that can be used
for other connectors as well. It is up for discussion if we want to split the KafkaJsonTableSource
into connector and format such that we can reuse the format for other table sources as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message