flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-9311) PubSub connector
Date Tue, 21 Aug 2018 09:31:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587208#comment-16587208
] 

ASF GitHub Bot commented on FLINK-9311:
---------------------------------------

Xeli opened a new pull request #6594: [FLINK-9311] [pubsub] Added PubSub source connector
with support for checkpointing (ATLEAST_ONCE)
URL: https://github.com/apache/flink/pull/6594
 
 
   **What is the purpose of the change**
   Adding a PubSub connector with support for Checkpointing
   
   **Verifying this change**
   This change added tests and can be verified as follows:
   
   - Added unit tests.
   - Added integration tests to flink-end-to-end which runs against docker.
   - An example has been added in flink-examples which runs against the actual Google PubSub
service. this has been manually verified.
   
   - Is there a need for integration tests?
   We feel like there is and have added them. 
   
   **Does this pull request potentially affect one of the following parts:**
   - Dependencies (does it add or upgrade a dependency): Yes, Google Cloud Sdk for PubSub
 but because it is a connector this does not add any dependencies in flink itself.
   - The public API, i.e., is any changed class annotated with @Public(Evolving): No
   - The serializers: No
   - The runtime per-record code paths (performance sensitive): Nothing has been changed only
a connector has been added.
   - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,

   - Yarn/Mesos, ZooKeeper: No
   - The S3 file system connector: No
   
   **Documentation**
   Does this pull request introduce a new feature? Yes
   If yes, how is the feature documented? JavaDocs, added an example in flink-examples and
we updated the website docs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> PubSub connector
> ----------------
>
>                 Key: FLINK-9311
>                 URL: https://issues.apache.org/jira/browse/FLINK-9311
>             Project: Flink
>          Issue Type: New Feature
>          Components: Streaming Connectors
>            Reporter: Richard Deurwaarder
>            Assignee: Niels Basjes
>            Priority: Minor
>              Labels: pull-request-available
>
> I would like start adding some google cloud connectors starting with a PubSub Source.
I have a basic implementation ready but I want it to be able to:
>  * easily scale up (should I have it extend RichParallelSourceFunction for this?)
>  * Make it easier to provide the google cloud credentials. This would require being able
to send some json string / ServiceAccount to the nodes when starting up this source.
> CouldĀ this be something that would be useful for others and added to the flink connectors
repo?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message