From commits-return-76301-archive-asf-public=cust-asf.ponee.io@airflow.apache.org Wed Nov 20 10:56:07 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 143A0180661 for ; Wed, 20 Nov 2019 11:56:06 +0100 (CET) Received: (qmail 36509 invoked by uid 500); 20 Nov 2019 10:56:04 -0000 Mailing-List: contact commits-help@airflow.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@airflow.apache.org Delivered-To: mailing list commits@airflow.apache.org Received: (qmail 36408 invoked by uid 99); 20 Nov 2019 10:56:04 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Nov 2019 10:56:04 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 0B5F8C09D2 for ; Wed, 20 Nov 2019 10:56:04 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -113 X-Spam-Level: X-Spam-Status: No, score=-113 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id zRKU82d4WsuX for ; Wed, 20 Nov 2019 10:56:02 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=207.244.88.153; helo=mail.apache.org; envelope-from=jira@apache.org; receiver= Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with SMTP id 206D67DE08 for ; Wed, 20 Nov 2019 10:56:02 +0000 (UTC) Received: (qmail 35449 invoked by uid 99); 20 Nov 2019 10:56:01 -0000 Received: from mailrelay1-us-west.apache.org (HELO mailrelay1-us-west.apache.org) (209.188.14.139) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Nov 2019 10:56:01 +0000 Received: from jira-he-de.apache.org (static.172.67.40.188.clients.your-server.de [188.40.67.172]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 99608E2E14 for ; Wed, 20 Nov 2019 10:56:00 +0000 (UTC) Received: from jira-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira-he-de.apache.org (ASF Mail Server at jira-he-de.apache.org) with ESMTP id 15AE47803DD for ; Wed, 20 Nov 2019 10:56:00 +0000 (UTC) Date: Wed, 20 Nov 2019 10:56:00 +0000 (UTC) From: "Ilya Kisil (Jira)" To: commits@airflow.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (AIRFLOW-5060) Add support of CatalogId to AwsGlueCatalogHook MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/AIRFLOW-5060?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16= 978309#comment-16978309 ]=20 Ilya Kisil commented on AIRFLOW-5060: ------------------------------------- [~jackjack10]=C2=A0another thing when extending glue hook with other availa= ble methods: unit tests. Not sure how straight forward they would be etc. > Add support of CatalogId to AwsGlueCatalogHook > ---------------------------------------------- > > Key: AIRFLOW-5060 > URL: https://issues.apache.org/jira/browse/AIRFLOW-5060 > Project: Apache Airflow > Issue Type: New Feature > Components: hooks > Affects Versions: 1.10.3 > Reporter: Ilya Kisil > Assignee: Ilya Kisil > Priority: Minor > > h2. Use Case > Imagine that you stream data into S3 bucket of an *account A* and update = AWS Glue datacatalog on a daily basis, so that you can query new data with = AWS Athena. Now let's assume that you provided access to=C2=A0this S3 bucke= t for an external *account B*=C2=A0who wants to use its' own AWS Athena to = query your data in an exactly the same way. Unfortunately, an *account B* w= ould need to=C2=A0have exactly the same table definitions in=C2=A0its AWS= =C2=A0Glue Datacatalog, because AWS Athena cannot run against external glue= datacatalog. However,=C2=A0AWS Glue service supports [cross-account dataca= talog access|https://docs.aws.amazon.com/glue/latest/dg/cross-account-acces= s.html], which means that *account B*=C2=A0can simply copy/sync meta inform= ation about database, tables, partitions etc from=C2=A0glue data catalog of= an *account A*, provided additional permissions have been granted. Thus, a= ll methods in *AwsGlueCatalogHook* should an use=C2=A0"CatalogId", i.e.=C2= =A0ID of the Data Catalog from which to retrieve/create/delete. > h2. =C2=A0 > h2. How it fits into Airflow > Assume that you have an=C2=A0AWSAthenaOperator, which queries data once a= day, then result is retrieved, visualised locally and then uploaded to som= e server/website. Then before this happens, you simply need to create an op= erator (even PythonOperator would do) which has two hooks, one to source ca= talog and another to destination catalog. At run time, it would use=C2=A0so= urce hook retrieve information from *account A*, for example [get_partition= s()|https://boto3.amazonaws.com/v1/documentation/api/latest/reference/servi= ces/glue.html#Glue.Client.get_partitions], then parse=C2=A0response=C2=A0an= d=C2=A0remove unnseccary keys and finally use destination hook to update *a= ccount B* datacatalog with [batch_create_partitions()|[https://boto3.amazon= aws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Clien= t.batch_create_partition]] > =C2=A0 > h2. Proposal > * Add a parameter=C2=A0*catalog_id* to=C2=A0AwsGlueCatalogHook, which th= en will be used in all its methods, regardless of this hook associated with= source or destination datacatalog.=C2=A0 > * In order not to break exsisting implementation, we set=C2=A0*catalog_i= d=3DNone.*=C2=A0But we add method=C2=A0*fallback_catalog_id(),*=C2=A0which = uses AWS STS to infer Catalog ID associated with used *aws_conn_id.* Obtain= ed value=C2=A0*=C2=A0*would be used if *catalog_id*=C2=A0hasn't been provid= ed during hook creation. > * Extend available methods of=C2=A0*AwsGlueCatalogHook* in a similar way= to already exsisting once, for convenience of the workflow described above= .=C2=A0Note: all new methods should strictly adhere AWS Glue Client Request= Syntax and do it in transparent manner. This means, that input information= shouldn't be modified within a method. When such actions are required, the= y should be performed outside of the AwsGlueCatalogHook. > h2. Implementation > * I am happy to contribute to airflow if this feature request gets appro= ved. > h2. Other considerations > * At the moment an existing method=C2=A0*get_partitions* doesn't not pro= vide you with all metainformation about partitions=C2=A0available from=C2= =A0glue client, whereas=C2=A0*get_table* does. Don't know the best way arou= nd it, but imho it should be refactored to=C2=A0*get_partitions_values*=C2= =A0or something like that. In this way, we would be able to stay inline wit= h boto3 glue client. > =C2=A0 -- This message was sent by Atlassian Jira (v8.3.4#803005)