beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nevi_me (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-2639) Unbounded Source for MongoDB
Date Wed, 19 Jul 2017 09:27:00 GMT
nevi_me created BEAM-2639:
-----------------------------

             Summary: Unbounded Source for MongoDB
                 Key: BEAM-2639
                 URL: https://issues.apache.org/jira/browse/BEAM-2639
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: nevi_me
            Assignee: Davor Bonaci
            Priority: Minor


The current MongoDB source is bounded, which means that we can't build streaming pipelines
directly from MongoDB.
MongoDB publishes changes in each collection through the oplog. Would it be possible to create
a connector that reads the oplog to create an unbounded source?

As an oplog is only available through replication, this creates that dependency. We would
need to also consider whether a polling method (using the ObjectId) could be an appropriate
fallback.

Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message