falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Baptiste Onofré (JIRA) <j...@apache.org>
Subject [jira] [Created] (FALCON-267) Add CDC feature
Date Thu, 16 Jan 2014 09:59:22 GMT
Jean-Baptiste Onofré created FALCON-267:

             Summary: Add CDC feature
                 Key: FALCON-267
                 URL: https://issues.apache.org/jira/browse/FALCON-267
             Project: Falcon
          Issue Type: New Feature
            Reporter: Jean-Baptiste Onofré
            Assignee: Jean-Baptiste Onofré

I propose to add a Change Data Capture feature in Falcon.

The idea is to be able to catch the change, firstly on HDFS files, publish the identified
gap to a messaging topic.

It's what I would like to PoC:
- in a feed definition, we had a <capture/> element defining the change check interval.
- we create a coordinator in oozie which execute the following workflow at capture interval
- in the Falcon staging "capture" location on HDFS, we keep the first state of the feed. We
compare (diff) the current content with the staging location, and write the diff in the Falcon
staging. If the file is a binary, we can detect a change (using MD5 for instance) and the
diff is the complete file (like in svn, git, etc).
- if we have a diff, we publish a message in the Falcon "capture" topic (containing a set
of JMS properties and the message body contains the link to the diff (on HDFS, in the Falcon
staging). The "stream" copy is ovewritten by the new one.

The purpose of this CDC is:
1/ thanks to the publication on the topic, to be able to use "external" tools to "react" when
a change occurs. For instance, I plan to make a demo with an Apache Camel route (sending e-mails
for example) when data change.
2/ staying in falcon/oozie/hadoop, to be able to setup a pipeline triggered by data change:
for instance, trigger a job when the data change.

The first PoC is HDFS/fs centric but I think we can do diff on HBase or Hive.

Thoughts ?

This message was sent by Atlassian JIRA

View raw message