Mailing-List: contact commits-help@airflow.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@airflow.incubator.apache.org
Date: Wed, 6 Sep 2017 04:18:03 +0000 (UTC)
From: "Siddharth (JIRA)" <jira@apache.org>
To: commits@airflow.incubator.apache.org
Message-ID: <JIRA.13099477.1504464979000.35801.1504671483395@Atlassian.JIRA>
In-Reply-To: <JIRA.13099477.1504464979000@Atlassian.JIRA>
References: <JIRA.13099477.1504464979000@Atlassian.JIRA> <JIRA.13099477.1504464979267@jira-lw-us.apache.org>
Subject: [jira] [Updated] (AIRFLOW-1560) Add AWS DynamoDB hook for inserting
 batch items
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 06 Sep 2017 04:18:10 -0000


     [ https://issues.apache.org/jira/browse/AIRFLOW-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Siddharth updated AIRFLOW-1560:
-------------------------------
    Description: 
The PR addresses airflow integration with AWS Dynamodb.  

Currently there is no hook to interact with DynamoDb for reading or writing items (single or batch insertions). To get started, we want to push data in DynamoDB using airflow jobs (scheduled daily). Idea is to read aggregates from Hive and push in DynamoDB (write data job will run everyday to make this happen). First we want to create DynamoDB hooks (this PR addressed the same) and then create operator to move data from Hive to DynamoDB (added hive to dynamo transfer operator)

I noticed that currently airflow has AWS_HOOK (parent hook for connecting to AWS using credentials stored in configs). It has a function to connect to AWS objects using Client API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#client) which is specific to EMR_HOOK. But in case of inserting data we can use DynamoDB Resource API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#service-resource) which provides higher level abstractions for inserting data in DynamoDB). One good question to ask can be difference between client and resource and why use one or the other? "Resources are higher-level abstraction than the raw, low-level calls made by service clients. They can't do anything the clients can't do, but in many cases they are nicer to use. The downside is that they don't always support 100% of the features of a service." (http://boto3.readthedocs.io/en/latest/guide/resources.html) 


  was:
The PR addresses airflow integration with AWS Dynamodb.  

Currently there is no hook to interact with DynamoDb for reading or writing items (single or batch insertions). To get started, we want to push data in DynamoDB using airflow jobs (scheduled daily). Idea is to read aggregates from S3 and push in DynamoDB (write data job will run everyday to make this happen). First we want to create DynamoDB hooks (this PR addressed the same) and then create operator to move data from S3 to DynamoDB (added hive to dynamo transfer operator)

I noticed that currently airflow has AWS_HOOK (parent hook for connecting to AWS using credentials stored in configs). It has a function to connect to AWS objects using Client API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#client) which is specific to EMR_HOOK. But in case of inserting data we can use DynamoDB Resource API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#service-resource) which provides higher level abstractions for inserting data in DynamoDB). One good question to ask can be difference between client and resource and why use one or the other? "Resources are higher-level abstraction than the raw, low-level calls made by service clients. They can't do anything the clients can't do, but in many cases they are nicer to use. The downside is that they don't always support 100% of the features of a service." (http://boto3.readthedocs.io/en/latest/guide/resources.html) 


> Add AWS DynamoDB hook for inserting batch items
> -----------------------------------------------
>
>                 Key: AIRFLOW-1560
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1560
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: aws, boto3, hooks
>            Reporter: Siddharth
>            Assignee: Siddharth
>
> The PR addresses airflow integration with AWS Dynamodb.  
> Currently there is no hook to interact with DynamoDb for reading or writing items (single or batch insertions). To get started, we want to push data in DynamoDB using airflow jobs (scheduled daily). Idea is to read aggregates from Hive and push in DynamoDB (write data job will run everyday to make this happen). First we want to create DynamoDB hooks (this PR addressed the same) and then create operator to move data from Hive to DynamoDB (added hive to dynamo transfer operator)
> I noticed that currently airflow has AWS_HOOK (parent hook for connecting to AWS using credentials stored in configs). It has a function to connect to AWS objects using Client API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#client) which is specific to EMR_HOOK. But in case of inserting data we can use DynamoDB Resource API (http://boto3.readthedocs.io/en/latest/reference/services/dynamodb.html#service-resource) which provides higher level abstractions for inserting data in DynamoDB). One good question to ask can be difference between client and resource and why use one or the other? "Resources are higher-level abstraction than the raw, low-level calls made by service clients. They can't do anything the clients can't do, but in many cases they are nicer to use. The downside is that they don't always support 100% of the features of a service." (http://boto3.readthedocs.io/en/latest/guide/resources.html) 


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)