airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth Anand (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-1575) Add AWS Kinesis Firehose hook for inserting batch records
Date Sun, 29 Apr 2018 06:13:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth Anand resolved AIRFLOW-1575.
--------------------------------------
       Resolution: Fixed
    Fix Version/s: 2.0.0

Issue resolved by pull request #3275
[https://github.com/apache/incubator-airflow/pull/3275]

> Add AWS Kinesis Firehose hook for inserting batch records
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-1575
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1575
>             Project: Apache Airflow
>          Issue Type: New Feature
>            Reporter: Siddharth
>            Assignee: Siddharth
>            Priority: Major
>             Fix For: 2.0.0
>
>
> One of the key components of ETL is data ingestion into multiple sources. Airflow provides
a great platform for multiple data sources to integrate with each other and transfer data
(hive to druid or hive to S3 etc). In AWS ecosystem Kinesis Firehose is an important component
which transfers data to other systems within AWS. Data can directly read in Airflow from any
system (druid, hive, s3, mysql or csv) and can be pushed to Firehose. This PR creates a firehose
hook for inserting batch items. Next - we can build an airflow operator to transfer data from
hive to firehose.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message