airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1575) Add AWS Kinesis Firehose hook for inserting batch records
Date Sun, 29 Apr 2018 06:12:00 GMT


ASF subversion and git services commented on AIRFLOW-1575:

Commit 2d588e9433cd9a1a1381cf939f579f7d7e53330f in incubator-airflow's branch refs/heads/master
from sid.gupta
[;h=2d588e9 ]

[AIRFLOW-1575] Add AWS Kinesis Firehose Hook for inserting batch records

Closes #3275 from sid88in/feature/kinesis_hookv2

> Add AWS Kinesis Firehose hook for inserting batch records
> ---------------------------------------------------------
>                 Key: AIRFLOW-1575
>                 URL:
>             Project: Apache Airflow
>          Issue Type: New Feature
>            Reporter: Siddharth
>            Assignee: Siddharth
>            Priority: Major
> One of the key components of ETL is data ingestion into multiple sources. Airflow provides
a great platform for multiple data sources to integrate with each other and transfer data
(hive to druid or hive to S3 etc). In AWS ecosystem Kinesis Firehose is an important component
which transfers data to other systems within AWS. Data can directly read in Airflow from any
system (druid, hive, s3, mysql or csv) and can be pushed to Firehose. This PR creates a firehose
hook for inserting batch items. Next - we can build an airflow operator to transfer data from
hive to firehose.

This message was sent by Atlassian JIRA

View raw message