airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Siddharth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1575) Add AWS Kinesis Firehose hook for inserting batch records
Date Thu, 07 Sep 2017 15:36:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Siddharth updated AIRFLOW-1575:
-------------------------------
    Description: One of the key components of ETL is data ingestion into multiple sources.
Airflow provides a great platform for multiple data sources to integrate with each other and
transfer data (hive to druid or hive to S3 etc). In AWS ecosystem Kinesis Firehose is an important
component which transfers data to other systems within AWS. Data can directly read in Airflow
from any system (druid, hive, s3, mysql or csv) and can be pushed to Firehose. This PR creates
a firehose hook for inserting batch items. Next - we can build an airflow operator to transfer
data from hive to firehose.

> Add AWS Kinesis Firehose hook for inserting batch records
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-1575
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1575
>             Project: Apache Airflow
>          Issue Type: New Feature
>            Reporter: Siddharth
>            Assignee: Siddharth
>
> One of the key components of ETL is data ingestion into multiple sources. Airflow provides
a great platform for multiple data sources to integrate with each other and transfer data
(hive to druid or hive to S3 etc). In AWS ecosystem Kinesis Firehose is an important component
which transfers data to other systems within AWS. Data can directly read in Airflow from any
system (druid, hive, s3, mysql or csv) and can be pushed to Firehose. This PR creates a firehose
hook for inserting batch items. Next - we can build an airflow operator to transfer data from
hive to firehose.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message