flume-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Simon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-2938) JDBC Source
Date Wed, 29 Jun 2016 08:12:45 GMT

    [ https://issues.apache.org/jira/browse/FLUME-2938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354767#comment-15354767

Attila Simon commented on FLUME-2938:

Sqoop's incremental import sounds suspiciosly similar: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

> JDBC Source
> -----------
>                 Key: FLUME-2938
>                 URL: https://issues.apache.org/jira/browse/FLUME-2938
>             Project: Flume
>          Issue Type: New Feature
>          Components: Sinks+Sources
>    Affects Versions: v1.8.0
>            Reporter: Lior Zeno
>             Fix For: v1.8.0
> The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS for archiving
> This source will get a statement to execute and a scheduling policy. It will be able
to fetch timestamped data by performing range queries on a configurable field (this can fetch
data with incremental id as well). For fault-tolerance, the last fetched value can be checkpointed
to a file.
> Dealing with large datasets can be done via the fetch_size parameter. (Ref: https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm)

This message was sent by Atlassian JIRA

View raw message