From notifications-return-158324-archive-asf-public=cust-asf.ponee.io@asterixdb.apache.org Thu Jun 17 15:42:45 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 20E42180658 for ; Thu, 17 Jun 2021 17:42:45 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 3E84C61A8D for ; Thu, 17 Jun 2021 15:42:41 +0000 (UTC) Received: (qmail 86597 invoked by uid 500); 17 Jun 2021 15:42:40 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 86581 invoked by uid 99); 17 Jun 2021 15:42:40 -0000 Received: from spamproc1-he-fi.apache.org (HELO spamproc1-he-fi.apache.org) (95.217.134.168) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jun 2021 15:42:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-fi.apache.org (ASF Mail Server at spamproc1-he-fi.apache.org) with ESMTP id 2FAF9C049B for ; Thu, 17 Jun 2021 15:42:39 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-fi.apache.org X-Spam-Flag: NO X-Spam-Score: 0.213 X-Spam-Level: X-Spam-Status: No, score=0.213 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=0.2, SPF_NONE=0.001, T_KAM_HTML_FONT_INVALID=0.01, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-fi.apache.org [95.217.134.168]) (amavisd-new, port 10024) with ESMTP id XBs5KOPDbbCh for ; Thu, 17 Jun 2021 15:42:37 +0000 (UTC) Received-SPF: None (mailfrom) identity=mailfrom; client-ip=128.195.1.133; helo=adrian-monk-v3.ics.uci.edu; envelope-from=do-not-reply@asterix-gerrit.ics.uci.edu; receiver= Received: from adrian-monk-v3.ics.uci.edu (adrian-monk-v3.ics.uci.edu [128.195.1.133]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 06B517FFD5 for ; Thu, 17 Jun 2021 15:42:35 +0000 (UTC) Received: from 92d4af910d15 (vitalstatistix.ics.uci.edu [128.195.52.38]) by adrian-monk-v3.ics.uci.edu (Postfix) with ESMTP id 625E8C02938E; Thu, 17 Jun 2021 08:42:25 -0700 (PDT) X-Gerrit-PatchSet: 1 Date: Thu, 17 Jun 2021 15:42:25 +0000 From: AsterixDB Code Review To: Till Westmann Message-ID: Auto-Submitted: auto-generated X-Gerrit-MessageType: newchange Subject: Change in asterixdb[cheshire-cat]: [ASTERIXDB-2919] Azure Blob external datasets: add support to anonymo... X-Gerrit-Change-Id: I19a161a3613371572fa26c786bab690e332fdd08 X-Gerrit-Change-Number: 11983 X-Gerrit-Project: asterixdb X-Gerrit-ChangeURL: X-Gerrit-Commit: 0b613aa6653d5299a265b0ae05fba63f2c0e4116 References: Reply-To: hussainht@gmail.com, dmitry.lychagin@couchbase.com, notifications@asterixdb.apache.org, lwhaywhu@gmail.com, tillw@apache.org MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/3.0.12 Content-Type: multipart/alternative; boundary="MsdOM2gS6FY="; charset=UTF-8 X-ICS-MailScanner-Information: Please send mail to helpdesk@ics.uci.edu or more information X-ICS-MailScanner-ID: 625E8C02938E.A3C69 X-ICS-MailScanner: Found to be clean X-ICS-MailScanner-SpamCheck: not spam, SpamAssassin (not cached, score=-0.985, required 5, ALL_TRUSTED -1.00, FSL_HELO_NON_FQDN_1 0.00, HELO_NO_DOMAIN 0.00, HTML_MESSAGE 0.00, T_KAM_HTML_FONT_INVALID 0.01, URIBL_BLOCKED 0.00, WEIRD_PORT 0.00) X-ICS-MailScanner-From: do-not-reply@asterix-gerrit.ics.uci.edu --MsdOM2gS6FY= Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable From Hussain Towaileb : Hussain Towaileb has upload= ed this change for review=2E ( https://asterix-gerrit=2Eics=2Euci=2Eedu/c/a= sterixdb/+/11983 ) Change subject: [ASTERIXDB-2919] Azure Blob external d= atasets: add support to anonymous authentication =2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E= =2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E=2E [ASTERIXDB-2919] Azure Blob external = datasets: add support to anonymous authentication - user model changes: no= - storage format changes: no - interface changes: no Details: - Added sup= port to creating external datasets with no credentials (anonymous)=2E Ch= ange-Id: I19a161a3613371572fa26c786bab690e332fdd08 --- A asterixdb/asterix-= app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_= storage/auth-methods/anonymous-no-auth-no-endpoint/test=2E000=2Eddl=2Esqlpp= C asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/externa= l-dataset/azure_blob_storage/auth-methods/anonymous-no-auth-no-endpoint/tes= t=2E099=2Eddl=2Esqlpp R asterixdb/asterix-app/src/test/resources/runtimets/= queries_sqlpp/external-dataset/azure_blob_storage/auth-methods/anonymous-no= -auth/test=2E000=2Eddl=2Esqlpp A asterixdb/asterix-app/src/test/resources/r= untimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-methods/ano= nymous-no-auth/test=2E001=2Equery=2Esqlpp R asterixdb/asterix-app/src/test/= resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-= methods/anonymous-no-auth/test=2E099=2Eddl=2Esqlpp A asterixdb/asterix-app/= src/test/resources/runtimets/results/external-dataset/azure_blob_storage/au= th-methods/anonymous-no-auth/result=2E001=2Eadm M asterixdb/asterix-app/src= /test/resources/runtimets/testsuite_external_dataset_azure_blob_storage=2Ex= ml M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/excep= tions/ErrorCode=2Ejava M asterixdb/asterix-common/src/main/resources/asx_er= rormsg/en=2Eproperties M asterixdb/asterix-external-data/src/main/java/org/= apache/asterix/external/util/ExternalDataUtils=2Ejava 10 files changed, 81 = insertions(+), 24 deletions(-) git pull ssh://asterix-gerrit=2Eics=2Eu= ci=2Eedu:29418/asterixdb refs/changes/83/11983/1 diff --git a/asterixdb/as= terix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure= _blob_storage/auth-methods/anonymous-no-auth-no-endpoint/test=2E000=2Eddl= =2Esqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp= /external-dataset/azure_blob_storage/auth-methods/anonymous-no-auth-no-endp= oint/test=2E000=2Eddl=2Esqlpp new file mode 100644 index 0000000=2E=2Ebd3e3= 40 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/q= ueries_sqlpp/external-dataset/azure_blob_storage/auth-methods/anonymous-no-= auth-no-endpoint/test=2E000=2Eddl=2Esqlpp @@ -0,0 +1,34 @@ +/* + * Licensed= to the Apache Software Foundation (ASF) under one + * or more contributor = license agreements=2E See the NOTICE file + * distributed with this work f= or additional information + * regarding copyright ownership=2E The ASF lic= enses this file + * to you under the Apache License, Version 2=2E0 (the + *= "License"); you may not use this file except in compliance + * with the Li= cense=2E You may obtain a copy of the License at + * + * http://www=2Eap= ache=2Eorg/licenses/LICENSE-2=2E0 + * + * Unless required by applicable law= or agreed to in writing, + * software distributed under the License is dis= tributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY += * KIND, either express or implied=2E See the License for the + * specific= language governing permissions and limitations + * under the License=2E + = */ + +drop dataverse test if exists; +create dataverse test; +use test; + += drop type test if exists; +create type test as open { +}; + +// bad case: n= o auth method is provided +drop dataset test if exists; +CREATE EXTERNAL DA= TASET test(test) USING AZUREBLOB ( +("container"=3D"playground"), +("defini= tion"=3D"json-data/reviews/single-line/json"), +("format"=3D"json") +); \ N= o newline at end of file diff --git a/asterixdb/asterix-app/src/test/resour= ces/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-method= s/invalid-no-auth/test=2E099=2Eddl=2Esqlpp b/asterixdb/asterix-app/src/test= /resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth= -methods/anonymous-no-auth-no-endpoint/test=2E099=2Eddl=2Esqlpp similarity = index 100% copy from asterixdb/asterix-app/src/test/resources/runtimets/que= ries_sqlpp/external-dataset/azure_blob_storage/auth-methods/invalid-no-auth= /test=2E099=2Eddl=2Esqlpp copy to asterixdb/asterix-app/src/test/resources/= runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-methods/an= onymous-no-auth-no-endpoint/test=2E099=2Eddl=2Esqlpp diff --git a/asterixdb= /asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/az= ure_blob_storage/auth-methods/invalid-no-auth/test=2E000=2Eddl=2Esqlpp b/as= terixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dat= aset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E000=2Eddl=2Es= qlpp similarity index 100% rename from asterixdb/asterix-app/src/test/resou= rces/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-metho= ds/invalid-no-auth/test=2E000=2Eddl=2Esqlpp rename to asterixdb/asterix-app= /src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_sto= rage/auth-methods/anonymous-no-auth/test=2E000=2Eddl=2Esqlpp diff --git a/a= sterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-da= taset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E001=2Equery= =2Esqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp= /external-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E= 001=2Equery=2Esqlpp new file mode 100644 index 0000000=2E=2E8ec9cc0 --- /de= v/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sql= pp/external-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test= =2E001=2Equery=2Esqlpp @@ -0,0 +1,21 @@ +/* + * Licensed to the Apache Soft= ware Foundation (ASF) under one + * or more contributor license agreements= =2E See the NOTICE file + * distributed with this work for additional info= rmation + * regarding copyright ownership=2E The ASF licenses this file + = * to you under the Apache License, Version 2=2E0 (the + * "License"); you m= ay not use this file except in compliance + * with the License=2E You may = obtain a copy of the License at + * + * http://www=2Eapache=2Eorg/license= s/LICENSE-2=2E0 + * + * Unless required by applicable law or agreed to in w= riting, + * software distributed under the License is distributed on an + *= "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either ex= press or implied=2E See the License for the + * specific language governin= g permissions and limitations + * under the License=2E + */ + +use test; +s= elect count(*) `count` from test; \ No newline at end of file diff --git a/= asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-d= ataset/azure_blob_storage/auth-methods/invalid-no-auth/test=2E099=2Eddl=2Es= qlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/ext= ernal-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E099= =2Eddl=2Esqlpp similarity index 100% rename from asterixdb/asterix-app/src/= test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/= auth-methods/invalid-no-auth/test=2E099=2Eddl=2Esqlpp rename to asterixdb/a= sterix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azur= e_blob_storage/auth-methods/anonymous-no-auth/test=2E099=2Eddl=2Esqlpp diff= --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/externa= l-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/result=2E001=2E= adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/external-d= ataset/azure_blob_storage/auth-methods/anonymous-no-auth/result=2E001=2Eadm= new file mode 100644 index 0000000=2E=2E187a8cb --- /dev/null +++ b/asteri= xdb/asterix-app/src/test/resources/runtimets/results/external-dataset/azure= _blob_storage/auth-methods/anonymous-no-auth/result=2E001=2Eadm @@ -0,0 +1 = @@ +{ "count": 100 } \ No newline at end of file diff --git a/asterixdb/ast= erix-app/src/test/resources/runtimets/testsuite_external_dataset_azure_blob= _storage=2Exml b/asterixdb/asterix-app/src/test/resources/runtimets/testsui= te_external_dataset_azure_blob_storage=2Exml index 1e302da=2E=2E5c6c1c0 100= 644 --- a/asterixdb/asterix-app/src/test/resources/runtimets/testsuite_exte= rnal_dataset_azure_blob_storage=2Exml +++ b/asterixdb/asterix-app/src/test/= resources/runtimets/testsuite_external_dataset_azure_blob_storage=2Exml @@ = -68,9 +68,14 @@ - - invalid-no-auth - ASX1134: No authe= ntication parameters provided + + anonymous-no= -auth + + + + = + anonymous-no-auth-no-endpoint + ASX1151: No authentication credentials provided, 'blobEndpoin= t' field is required for anonymous access diff --git a/asterixdb/asteri= x-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode=2Ejav= a b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/except= ions/ErrorCode=2Ejava index 4078ea2=2E=2Ee80dbfb 100644 --- a/asterixdb/ast= erix-common/src/main/java/org/apache/asterix/common/exceptions/ErrorCode=2E= java +++ b/asterixdb/asterix-common/src/main/java/org/apache/asterix/common= /exceptions/ErrorCode=2Ejava @@ -235,6 +235,7 @@ CANNOT_DROP_OBJECT_DE= PENDENT_EXISTS(1148), ILLEGAL_FUNCTION_RECURSION(1149), ILLEGAL_F= UNCTION_USE(1150), + NO_AUTH_PROVIDED_ENDPOINT_REQUIRED_FOR_ANONYMOUS_AC= CESS(1151), // Feed errors DATAFLOW_ILLEGAL_STATE(3001), diff -= -git a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproper= ties b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproper= ties index 18898bb=2E=2Ecd4a289 100644 --- a/asterixdb/asterix-common/src/m= ain/resources/asx_errormsg/en=2Eproperties +++ b/asterixdb/asterix-common/s= rc/main/resources/asx_errormsg/en=2Eproperties @@ -237,6 +237,7 @@ 1148 = =3D Cannot drop %1$s %2$s being used by %3$s %4$s 1149 =3D Illegal functio= n recursion 1150 =3D Illegal use of function %1$s +1151 =3D No authenticat= ion credentials provided, '%1$s' field is required for anonymous access = # Feed Errors 3001 =3D Illegal state=2E diff --git a/asterixdb/asterix-ext= ernal-data/src/main/java/org/apache/asterix/external/util/ExternalDataUtils= =2Ejava b/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/= external/util/ExternalDataUtils=2Ejava index 8e94263=2E=2E6fb0271 100644 --= - a/asterixdb/asterix-external-data/src/main/java/org/apache/asterix/extern= al/util/ExternalDataUtils=2Ejava +++ b/asterixdb/asterix-external-data/src/= main/java/org/apache/asterix/external/util/ExternalDataUtils=2Ejava @@ -22,= 17 +22,7 @@ import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExter= nalDataConstants=2EAwsS3=2EACCESS_KEY_ID_FIELD_NAME; import static org=2Ea= pache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAwsS3=2EERROR_M= ETHOD_NOT_IMPLEMENTED; import static org=2Eapache=2Easterix=2Eexternal=2Eu= til=2EExternalDataConstants=2EAwsS3=2ESECRET_ACCESS_KEY_FIELD_NAME; -import= static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2E= AzureBlob=2EACCOUNT_KEY_FIELD_NAME; -import static org=2Eapache=2Easterix= =2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2EACCOUNT_NAME_FIELD= _NAME; -import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDa= taConstants=2EAzureBlob=2EBLOB_ENDPOINT_FIELD_NAME; -import static org=2Eap= ache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2ECONN= ECTION_STRING_ACCOUNT_KEY; -import static org=2Eapache=2Easterix=2Eexternal= =2Eutil=2EExternalDataConstants=2EAzureBlob=2ECONNECTION_STRING_ACCOUNT_NAM= E; -import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataCo= nstants=2EAzureBlob=2ECONNECTION_STRING_BLOB_ENDPOINT; -import static org= =2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob= =2ECONNECTION_STRING_ENDPOINT_SUFFIX; -import static org=2Eapache=2Easterix= =2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2ECONNECTION_STRING_= FIELD_NAME; -import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExter= nalDataConstants=2EAzureBlob=2ECONNECTION_STRING_SHARED_ACCESS_SIGNATURE; -= import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConsta= nts=2EAzureBlob=2EENDPOINT_SUFFIX_FIELD_NAME; -import static org=2Eapache= =2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2ESHARED_A= CCESS_SIGNATURE_FIELD_NAME; +import static org=2Eapache=2Easterix=2Eexterna= l=2Eutil=2EExternalDataConstants=2EAzureBlob=2E*; import static org=2Eapac= he=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EKEY_DELIMITER; im= port static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstant= s=2EKEY_ESCAPE; import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EE= xternalDataConstants=2EKEY_EXTERNAL_SCAN_BUFFER_SIZE; @@ -899,7 +889,7 @@ = String blobEndpoint =3D configuration=2Eget(BLOB_ENDPOINT_FIELD= _NAME); String endpointSuffix =3D configuration=2Eget(ENDPOINT= _SUFFIX_FIELD_NAME); - // Constructor the connection string + = // Construct the connection string // Connection st= ring format: name1=3Dvalue1;name2=3Dvalue2;=2E=2E=2E=2E String= Builder connectionStringBuilder =3D new StringBuilder(); BlobS= erviceClientBuilder builder =3D new BlobServiceClientBuilder(); @@ -926,34 = +916,38 @@ if (authMethodFound) { thr= ow new CompilationException(ErrorCode=2EONLY_SINGLE_AUTHENTICATION_IS_ALLOW= ED); } - authMethodFound =3D true; = // account name + shared access token connectionS= tringBuilder=2Eappend(CONNECTION_STRING_ACCOUNT_NAME)=2Eappend("=3D")=2Eapp= end(accountName) =2Eappend(";")=2Eappend(CONNECTIO= N_STRING_SHARED_ACCESS_SIGNATURE)=2Eappend("=3D") = =2Eappend(sharedAccessSignature)=2Eappend(";"); } - = if (!authMethodFound) { - throw new CompilationException(= ErrorCode=2ENO_AUTH_METHOD_PROVIDED); - } - // Add = blobEndpoint and endpointSuffix if present, adjust any '/' as needed = if (blobEndpoint !=3D null) { connectionStringBuild= er=2Eappend(CONNECTION_STRING_BLOB_ENDPOINT)=2Eappend("=3D")=2Eappend(blobE= ndpoint) =2Eappend(";"); + if (en= dpointSuffix !=3D null) { - String endpointSuffixUpdated= ; + String endpointSuffixSlashCorrected; + = if (blobEndpoint=2EendsWith("/")) { - endpoi= ntSuffixUpdated =3D + endpointSuffixSlashCorrected = =3D endpointSuffix=2EstartsWith("/") ? end= pointSuffix=2Esubstring(1) : endpointSuffix; } else { = - endpointSuffixUpdated =3D endpointSuffix=2EstartsW= ith("/") ? endpointSuffix : "/" + endpointSuffix; + = endpointSuffixSlashCorrected =3D + endpointS= uffix=2EstartsWith("/") ? endpointSuffix : "/" + endpointSuffix; = } connectionStringBuilder=2Eappend(CONNECTI= ON_STRING_ENDPOINT_SUFFIX)=2Eappend("=3D") - =2E= append(endpointSuffixUpdated)=2Eappend(";"); + = =2Eappend(endpointSuffixSlashCorrected)=2Eappend(";"); } = } + // No credentials or endpoint provided + = if (connectionStringBuilder=2Elength() =3D=3D 0) { + t= hrow new CompilationException(ErrorCode=2ENO_AUTH_PROVIDED_ENDPOINT_REQUIRE= D_FOR_ANONYMOUS_ACCESS, + BLOB_ENDPOINT_FIELD_NAME);= + } + try { return builder=2Econn= ectionString(connectionStringBuilder=2EtoString())=2EbuildClient(); = } catch (Exception ex) { -- To view, visit https://asterix-gerrit= =2Eics=2Euci=2Eedu/c/asterixdb/+/11983 To unsubscribe, or for help writing = mail filters, visit https://asterix-gerrit=2Eics=2Euci=2Eedu/settings Gerr= it-Project: asterixdb Gerrit-Branch: cheshire-cat Gerrit-Change-Id: I19a161= a3613371572fa26c786bab690e332fdd08 Gerrit-Change-Number: 11983 Gerrit-Patch= Set: 1 Gerrit-Owner: Hussain Towaileb Gerrit-Messag= eType: newchange --MsdOM2gS6FY= Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Hussain Towaileb has uploaded this change for review=2E=

View Change

[ASTERIXDB-2919] Azure Blob external datasets: add sup=
port to anonymous authentication

- user model changes: no
- stora=
ge format changes: no
- interface changes: no

Details:
- Added=
 support to creating external datasets with no
  credentials (anonymous)=
=2E

Change-Id: I19a161a3613371572fa26c786bab690e332fdd08
---
A=
 asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-=
dataset/azure_blob_storage/auth-methods/anonymous-no-auth-no-endpoint/test=
=2E000=2Eddl=2Esqlpp
C asterixdb/asterix-app/src/test/resources/runtimet=
s/queries_sqlpp/external-dataset/azure_blob_storage/auth-methods/anonymous-=
no-auth-no-endpoint/test=2E099=2Eddl=2Esqlpp
R asterixdb/asterix-app/src=
/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage=
/auth-methods/anonymous-no-auth/test=2E000=2Eddl=2Esqlpp
A asterixdb/ast=
erix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_=
blob_storage/auth-methods/anonymous-no-auth/test=2E001=2Equery=2Esqlpp
R=
 asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-=
dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E099=2Eddl=
=2Esqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/ex=
ternal-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/result=2E0=
01=2Eadm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_=
external_dataset_azure_blob_storage=2Exml
M asterixdb/asterix-common/src=
/main/java/org/apache/asterix/common/exceptions/ErrorCode=2Ejava
M aster=
ixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproperties
M as=
terixdb/asterix-external-data/src/main/java/org/apache/asterix/external/uti=
l/ExternalDataUtils=2Ejava
10 files changed, 81 insertions(+), 24 deleti=
ons(-)

git pull ssh://asterix-gerrit=2Eics=2Euci=2Eedu:29418/aste=
rixdb refs/changes/83/11983/1
diff --git a/asterixdb/asterix-app/sr=
c/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storag=
e/auth-methods/anonymous-no-auth-no-endpoint/test=2E000=2Eddl=2Esqlpp b/ast=
erixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-data=
set/azure_blob_storage/auth-methods/anonymous-no-auth-no-endpoint/test=2E00=
0=2Eddl=2Esqlpp
new file mode 100644
index =
0000000=2E=2Ebd3e340
--- /dev/null
+++ b/as=
terixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external-dat=
aset/azure_blob_storage/auth-methods/anonymous-no-auth-no-endpoint/test=2E0=
00=2Eddl=2Esqlpp
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one<=
/span>
+ * or more contribut=
or license agreements=2E  See the NOTICE file
+ * distributed with this work for additional info=
rmation
+ * regarding=
 copyright ownership=2E  The ASF licenses this file
+ * to you under the Apache License, Versi=
on 2=2E0 (the
+ * &qu=
ot;License"); you may not use this file except in compliance+ * with the License=2E  You m=
ay obtain a copy of the License at
+ *
+ *=
   http://www=2Eapache=2Eorg/licenses/LICENSE-2=2E0
+ *
+ * Unless required by applicable law or agreed to in writi=
ng,
+ * software dist=
ributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES O=
R CONDITIONS OF ANY
+=
 * KIND, either express or implied=2E  See the License for the
+ * specific language governing p=
ermissions and limitations
+ * under the License=2E
+ */
+
+drop dataverse test if e=
xists;
+create datave=
rse test;
+use test;<=
/span>
+
+drop type test if exists;
+create type test as open {=

+};
+
+// bad case: no auth method is provided
+drop dataset test if exists;
<=
span style=3D"color: hsl(120, 100%, 40%);">+CREATE EXTERNAL DATASET test(te=
st) USING AZUREBLOB (
+("container"=3D"playground"),
+("definition"=3D"json-data=
/reviews/single-line/json"),
+("format"=3D"json")
+);
\ No newline at end of=
 file
diff --git a/asterixdb/asterix-app/src/test/resources=
/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-methods/i=
nvalid-no-auth/test=2E099=2Eddl=2Esqlpp b/asterixdb/asterix-app/src/test/re=
sources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-me=
thods/anonymous-no-auth-no-endpoint/test=2E099=2Eddl=2Esqlpp
similarity index 100%
copy from asterixdb/asterix-app/src=
/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage=
/auth-methods/invalid-no-auth/test=2E099=2Eddl=2Esqlpp
copy=
 to asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/extern=
al-dataset/azure_blob_storage/auth-methods/anonymous-no-auth-no-endpoint/te=
st=2E099=2Eddl=2Esqlpp
diff --git a/asterixdb/asterix-app/s=
rc/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_stora=
ge/auth-methods/invalid-no-auth/test=2E000=2Eddl=2Esqlpp b/asterixdb/asteri=
x-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blo=
b_storage/auth-methods/anonymous-no-auth/test=2E000=2Eddl=2Esqlppsimilarity index 100%
rename from asterixdb/asterix-=
app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_=
storage/auth-methods/invalid-no-auth/test=2E000=2Eddl=2Esqlpp
rename to asterixdb/asterix-app/src/test/resources/runtimets/queries_sql=
pp/external-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test=
=2E000=2Eddl=2Esqlpp
diff --git a/asterixdb/asterix-app/src=
/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage=
/auth-methods/anonymous-no-auth/test=2E001=2Equery=2Esqlpp b/asterixdb/aste=
rix-app/src/test/resources/runtimets/queries_sqlpp/external-dataset/azure_b=
lob_storage/auth-methods/anonymous-no-auth/test=2E001=2Equery=2Esqlpp
new file mode 100644
index 0000000=2E=2E8ec9cc0<=
/span>
--- /dev/null
+++ b/asterixdb/asterix-app/s=
rc/test/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_stora=
ge/auth-methods/anonymous-no-auth/test=2E001=2Equery=2Esqlpp
@@ -0,0 +1,21 @@
+/=
*
+ * Licensed to the=
 Apache Software Foundation (ASF) under one
+ * or more contributor license agreements=2E  See t=
he NOTICE file
+ * di=
stributed with this work for additional information
+ * regarding copyright ownership=2E  The =
ASF licenses this file
+ * "License"); you may not =
use this file except in compliance
+ * with the License=2E  You may obtain a copy of the License=
 at
+ *
+ *   http://www=2Eapache=2Eorg/li=
censes/LICENSE-2=2E0
=
+ *
+ * Unless requir=
ed by applicable law or agreed to in writing,
+ * software distributed under the License is dist=
ributed on an
+ * &qu=
ot;AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implie=
d=2E  See the License for the
+ * specific language governing permissions and limitations=

+ * under the License=2E
+ */
+
+use test;
+select count(*) `count` from test;
\ No newline at =
end of file
diff --git a/asterixdb/asterix-app/src/test/res=
ources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/auth-met=
hods/invalid-no-auth/test=2E099=2Eddl=2Esqlpp b/asterixdb/asterix-app/src/t=
est/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/a=
uth-methods/anonymous-no-auth/test=2E099=2Eddl=2Esqlpp
simi=
larity index 100%
rename from asterixdb/asterix-app/src/tes=
t/resources/runtimets/queries_sqlpp/external-dataset/azure_blob_storage/aut=
h-methods/invalid-no-auth/test=2E099=2Eddl=2Esqlpp
rename t=
o asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/external=
-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/test=2E099=2Eddl=
=2Esqlpp
diff --git a/asterixdb/asterix-app/src/test/resour=
ces/runtimets/results/external-dataset/azure_blob_storage/auth-methods/anon=
ymous-no-auth/result=2E001=2Eadm b/asterixdb/asterix-app/src/test/resources=
/runtimets/results/external-dataset/azure_blob_storage/auth-methods/anonymo=
us-no-auth/result=2E001=2Eadm
new file mode 100644index 0000000=2E=2E187a8cb
--- /dev/null
=
+++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/exte=
rnal-dataset/azure_blob_storage/auth-methods/anonymous-no-auth/result=2E001=
=2Eadm
@@ -0,0 +1 @@
+{ "count": 100 }
\ No newline at=
 end of file
diff --git a/asterixdb/asterix-app/src/test/re=
sources/runtimets/testsuite_external_dataset_azure_blob_storage=2Exml b/ast=
erixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_=
azure_blob_storage=2Exml
index 1e302da=2E=2E5c6c1c0 100644<=
/span>
--- a/asterixdb/asterix-app/src/test/resources/runtimets/te=
stsuite_external_dataset_azure_blob_storage=2Exml
+++ b/ast=
erixdb/asterix-app/src/test/resources/runtimets/testsuite_external_dataset_=
azure_blob_storage=2Exml
@@ -68,9 +68,14 @@
       </compilation-unit>
     </test-case>
     <test-case FilePath=3D"external-dataset/azure_b=
lob_storage/auth-methods">
-      <compilation-unit name=3D"invalid-no-auth"&=
gt;
-        <output=
-dir compare=3D"Text">invalid-no-auth</output-dir>
-        <expected-error&=
gt;ASX1134: No authentication parameters provided</expected-error>
+      <compilation-=
unit name=3D"anonymous-no-auth">
+        <output-dir compare=3D"Text"=
>anonymous-no-auth</output-dir>
+      </compilation-unit>
+    </test-case>
+    <test-case FilePath=3D"ext=
ernal-dataset/azure_blob_storage/auth-methods">
+      <compilation-unit name=3D"=
;anonymous-no-auth-no-endpoint">
+        <output-dir compare=3D"Text">a=
nonymous-no-auth-no-endpoint</output-dir>
+        <expected-error>ASX1151: No authe=
ntication credentials provided, 'blobEndpoint' field is required fo=
r anonymous access</expected-error>
       </compi=
lation-unit>
     </test-case>
   =
</test-group>
diff --git a/asterixdb/asterix-common/s=
rc/main/java/org/apache/asterix/common/exceptions/ErrorCode=2Ejava b/asteri=
xdb/asterix-common/src/main/java/org/apache/asterix/common/exceptions/Error=
Code=2Ejava
index 4078ea2=2E=2Ee80dbfb 100644
--- a/asterixdb/asterix-common/src/main/java/org/apache/asterix/common/e=
xceptions/ErrorCode=2Ejava
+++ b/asterixdb/asterix-common/s=
rc/main/java/org/apache/asterix/common/exceptions/ErrorCode=2Ejava@@ -235,6 +235,7 @@
     CANNOT_DROP_OBJECT_DEPENDE=
NT_EXISTS(1148),
     ILLEGAL_FUNCTION_RECURSION(1149),
     ILLEGAL_FUNCTION_USE(1150),
+    NO_AUTH_PROVIDED_ENDPOINT_REQUIRED_FOR_ANONYM=
OUS_ACCESS(1151),
 
     // Feed errors
     DATAFLOW_ILLEGAL_STATE(3001),
diff --git =
a/asterixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproperties =
b/asterixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproperties<=
/span>
index 18898bb=2E=2Ecd4a289 100644
--- a/ast=
erixdb/asterix-common/src/main/resources/asx_errormsg/en=2Eproperties
+++ b/asterixdb/asterix-common/src/main/resources/asx_errormsg/e=
n=2Eproperties
@@ -237,6 +237,7 @@
 1148 =
=3D Cannot drop %1$s %2$s being used by %3$s %4$s
 1149 =3D=
 Illegal function recursion
 1150 =3D Illegal use of functi=
on %1$s
+1151 =3D No =
authentication credentials provided, '%1$s' field is required for a=
nonymous access
 
 # Feed Errors
=
 3001 =3D Illegal state=2E
diff --git a/asterixdb/ast=
erix-external-data/src/main/java/org/apache/asterix/external/util/ExternalD=
ataUtils=2Ejava b/asterixdb/asterix-external-data/src/main/java/org/apache/=
asterix/external/util/ExternalDataUtils=2Ejava
index 8e9426=
3=2E=2E6fb0271 100644
--- a/asterixdb/asterix-external-data=
/src/main/java/org/apache/asterix/external/util/ExternalDataUtils=2Ejava
+++ b/asterixdb/asterix-external-data/src/main/java/org/apach=
e/asterix/external/util/ExternalDataUtils=2Ejava
@@ -22,17 =
+22,7 @@
 import static org=2Eapache=2Easterix=2Eexternal=
=2Eutil=2EExternalDataConstants=2EAwsS3=2EACCESS_KEY_ID_FIELD_NAME;<=
br> import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternal=
DataConstants=2EAwsS3=2EERROR_METHOD_NOT_IMPLEMENTED;
 impo=
rt static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=
=2EAwsS3=2ESECRET_ACCESS_KEY_FIELD_NAME;
-import static org=2Eapache=2Easterix=2Eexternal=2Eutil=
=2EExternalDataConstants=2EAzureBlob=2EACCOUNT_KEY_FIELD_NAME;
-import static org=2Eapache=2Easter=
ix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2EACCOUNT_NAME_FIE=
LD_NAME;
-import static=
 org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBl=
ob=2EBLOB_ENDPOINT_FIELD_NAME;
-import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalD=
ataConstants=2EAzureBlob=2ECONNECTION_STRING_ACCOUNT_KEY;
-import static org=2Eapache=2Easterix=2E=
external=2Eutil=2EExternalDataConstants=2EAzureBlob=2ECONNECTION_STRING_ACC=
OUNT_NAME;
-import stat=
ic org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzure=
Blob=2ECONNECTION_STRING_BLOB_ENDPOINT;
-import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2E=
ExternalDataConstants=2EAzureBlob=2ECONNECTION_STRING_ENDPOINT_SUFFIX;
-import static org=2Eapache=
=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2ECONNECTI=
ON_STRING_FIELD_NAME;
-=
import static org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConsta=
nts=2EAzureBlob=2ECONNECTION_STRING_SHARED_ACCESS_SIGNATURE;
-import static org=2Eapache=2Easterix=
=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureBlob=2EENDPOINT_SUFFIX_FI=
ELD_NAME;
-import stati=
c org=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EAzureB=
lob=2ESHARED_ACCESS_SIGNATURE_FIELD_NAME;
+import static org=2Eapache=2Easterix=2Eexternal=2Euti=
l=2EExternalDataConstants=2EAzureBlob=2E*;
 import static o=
rg=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EKEY_DELIM=
ITER;
 import static org=2Eapache=2Easterix=2Eexternal=2Eut=
il=2EExternalDataConstants=2EKEY_ESCAPE;
 import static org=
=2Eapache=2Easterix=2Eexternal=2Eutil=2EExternalDataConstants=2EKEY_EXTERNA=
L_SCAN_BUFFER_SIZE;
@@ -899,7 +889,7 @@
   =
          String blobEndpoint =3D configuration=2Eget(BLOB_ENDPOINT_FIELD_N=
AME);
             String endpointSuffix =3D configuration=
=2Eget(ENDPOINT_SUFFIX_FIELD_NAME);
 
-            // Constructor the connection s=
tring
+            //=
 Construct the connection string
             // Connection=
 string format: name1=3Dvalue1;name2=3Dvalue2;=2E=2E=2E=2E
=
             StringBuilder connectionStringBuilder =3D new StringBuilder();=

             BlobServiceClientBuilder builder =3D new Blob=
ServiceClientBuilder();
@@ -926,34 +916,38 @@
                 if (authMethodFound) {
                =
     throw new CompilationException(ErrorCode=2EONLY_SINGLE_AUTHENTICATION_=
IS_ALLOWED);
                 }
-                authMethodFound =3D true;<=
br>                 // account name + shared access token
<=
span>                 connectionStringBuilder=2Eappend(CONNECTION_STRING_AC=
COUNT_NAME)=2Eappend("=3D")=2Eappend(accountName)
                         =2Eappend(";")=2Eappend(CONNECTION_STRI=
NG_SHARED_ACCESS_SIGNATURE)=2Eappend("=3D")
     =
                    =2Eappend(sharedAccessSignature)=2Eappend(";"=
);
             }
 
-            if (!authMethodFound) {<=
br>-                throw new Com=
pilationException(ErrorCode=2ENO_AUTH_METHOD_PROVIDED);
-            }
-
             // Add blobEndpoint=
 and endpointSuffix if present, adjust any '/' as needed
=
             if (blobEndpoint !=3D null) {
          =
       connectionStringBuilder=2Eappend(CONNECTION_STRING_BLOB_ENDPOINT)=2E=
append("=3D")=2Eappend(blobEndpoint)
            =
             =2Eappend(";");
+
                 if (endpointSuffix !=
=3D null) {
-          =
          String endpointSuffixUpdated;
+                    String endpointSuffixSlashCorrected=
;
+
 =
                    if (blobEndpoint=2EendsWith("/")) {-                        endpoin=
tSuffixUpdated =3D
+ =
                       endpointSuffixSlashCorrected =3D
   =
                              endpointSuffix=2EstartsWith("/") ? =
endpointSuffix=2Esubstring(1) : endpointSuffix;
           =
          } else {
-   =
                     endpointSuffixUpdated =3D endpointSuffix=2EstartsWith(=
"/") ? endpointSuffix : "/" + endpointSuffix;+                        endpo=
intSuffixSlashCorrected =3D
+                                endpointSuffix=2EstartsWith("/=
") ? endpointSuffix : "/" + endpointSuffix;
=
                     }
                     connectionStrin=
gBuilder=2Eappend(CONNECTION_STRING_ENDPOINT_SUFFIX)=2Eappend("=3D&quo=
t;)
-                  =
          =2Eappend(endpointSuffixUpdated)=2Eappend(";");<=
br>+                           =
 =2Eappend(endpointSuffixSlashCorrected)=2Eappend(";");                 }
             }
 <=
/span>
+            // No cr=
edentials or endpoint provided
+            if (connectionStringBuilder=2Elength() =3D=3D 0) {
+                throw=
 new CompilationException(ErrorCode=2ENO_AUTH_PROVIDED_ENDPOINT_REQUIRED_FO=
R_ANONYMOUS_ACCESS,
+=
                        BLOB_ENDPOINT_FIELD_NAME);
+            }
+
             try {
                 return builder=2EconnectionString(connectionStringBuilde=
r=2EtoString())=2EbuildClient();
             } catch (Exce=
ption ex) {

To view, visit change 11983=2E To unsubscribe, or for help writing mail filters, visit settings=2E

Gerrit-Project: asteri= xdb
Gerrit-Branch: cheshire-cat
=
Gerrit-Change-Id: I19a161a3613371572fa26c786ba= b690e332fdd08
Gerrit-Change-Number: 119= 83
Gerrit-PatchSet: 1
Gerrit-Owner: Hussain Towaileb <hussainht@gmail=2Ecom= >
Gerrit-MessageType: newchange --MsdOM2gS6FY=--