From dev-return-16805-archive-asf-public=cust-asf.ponee.io@reef.apache.org Thu Jun 14 22:34:08 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D36CC18066C for ; Thu, 14 Jun 2018 22:34:07 +0200 (CEST) Received: (qmail 79486 invoked by uid 500); 14 Jun 2018 20:34:05 -0000 Mailing-List: contact dev-help@reef.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@reef.apache.org Delivered-To: mailing list dev@reef.apache.org Received: (qmail 79333 invoked by uid 99); 14 Jun 2018 20:34:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 14 Jun 2018 20:34:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 57D64182E7B for ; Thu, 14 Jun 2018 20:34:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.51 X-Spam-Level: X-Spam-Status: No, score=-109.51 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100, WEIRD_QUOTING=0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id SrQh-wMh-iKE for ; Thu, 14 Jun 2018 20:34:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 404925F59D for ; Thu, 14 Jun 2018 20:34:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id AFCA0E0CA4 for ; Thu, 14 Jun 2018 20:34:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 5C2A721840 for ; Thu, 14 Jun 2018 20:34:00 +0000 (UTC) Date: Thu, 14 Jun 2018 20:34:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: dev@reef.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (REEF-2031) Azure Batch credentials exposed in job-submission-params.json in REEF.NET MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/REEF-2031?page=3Dcom.atlassian.= jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D16512= 951#comment-16512951 ]=20 ASF GitHub Bot commented on REEF-2031: -------------------------------------- motus closed pull request #1469: [REEF-2031]Azure Batch credentials exposed= in job-submission-params.json in REEF.NET URL: https://github.com/apache/reef/pull/1469 =20 =20 =20 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/lang/cs/Org.Apache.REEF.Client/Avro/AzureBatch/AvroAzureBatchJ= obSubmissionParameters.cs b/lang/cs/Org.Apache.REEF.Client/Avro/AzureBatch/= AvroAzureBatchJobSubmissionParameters.cs index 78bfe4a323..6409dc87fe 100644 --- a/lang/cs/Org.Apache.REEF.Client/Avro/AzureBatch/AvroAzureBatchJobSubmi= ssionParameters.cs +++ b/lang/cs/Org.Apache.REEF.Client/Avro/AzureBatch/AvroAzureBatchJobSubmi= ssionParameters.cs @@ -30,7 +30,7 @@ namespace Org.Apache.REEF.Client.Avro.AzureBatch [DataContract(Namespace =3D "org.apache.reef.reef.bridge.client.avro")= ] public sealed class AvroAzureBatchJobSubmissionParameters { - private const string JsonSchema =3D @"{""type"":""record"",""name"= ":""org.apache.reef.reef.bridge.client.avro.AvroAzureBatchJobSubmissionPara= meters"",""doc"":""Job submission parameters used by the Azure Batch runtim= e"",""fields"":[{""name"":""sharedJobSubmissionParameters"",""type"":{""typ= e"":""record"",""name"":""org.apache.reef.reef.bridge.client.avro.AvroJobSu= bmissionParameters"",""doc"":""General cross-language job submission parame= ters shared by all runtimes"",""fields"":[{""name"":""jobId"",""type"":""st= ring""},{""name"":""jobSubmissionFolder"",""type"":""string""}]}},{""name""= :""AzureBatchAccountKey"",""type"":""string""},{""name"":""AzureBatchAccoun= tName"",""type"":""string""},{""name"":""AzureBatchAccountUri"",""type"":""= string""},{""name"":""AzureBatchPoolId"",""type"":""string""},{""name"":""A= zureStorageAccountKey"",""type"":""string""},{""name"":""AzureStorageAccoun= tName"",""type"":""string""},{""name"":""AzureStorageContainerName"",""type= "":""string""},{""name"":""AzureBatchIsWindows"",""type"":""boolean""}]}"; + private const string JsonSchema =3D @"{""type"":""record"",""name"= ":""org.apache.reef.reef.bridge.client.avro.AvroAzureBatchJobSubmissionPara= meters"",""doc"":""Job submission parameters used by the Azure Batch runtim= e"",""fields"":[{""name"":""sharedJobSubmissionParameters"",""type"":{""typ= e"":""record"",""name"":""org.apache.reef.reef.bridge.client.avro.AvroJobSu= bmissionParameters"",""doc"":""General cross-language job submission parame= ters shared by all runtimes"",""fields"":[{""name"":""jobId"",""type"":""st= ring""},{""name"":""jobSubmissionFolder"",""type"":""string""}]}},{""name""= :""AzureBatchAccountName"",""type"":""string""},{""name"":""AzureBatchAccou= ntUri"",""type"":""string""},{""name"":""AzureBatchPoolId"",""type"":""stri= ng""},{""name"":""AzureStorageAccountName"",""type"":""string""},{""name"":= ""AzureStorageContainerName"",""type"":""string""}]}"; =20 /// /// Gets the schema. @@ -49,12 +49,6 @@ public static string Schema [DataMember] public AvroJobSubmissionParameters sharedJobSubmissionParameters {= get; set; } =20 - /// - /// Gets or sets the AzureBatchAccountKey field. - /// - [DataMember] - public string AzureBatchAccountKey { get; set; } - /// /// Gets or sets the AzureBatchAccountName field. /// @@ -73,12 +67,6 @@ public static string Schema [DataMember] public string AzureBatchPoolId { get; set; } =20 - /// - /// Gets or sets the AzureStorageAccountKey field. - /// - [DataMember] - public string AzureStorageAccountKey { get; set; } - /// /// Gets or sets the AzureStorageAccountName field. /// @@ -91,12 +79,6 @@ public static string Schema [DataMember] public string AzureStorageContainerName { get; set; } =20 - /// - /// Gets or sets the AzureBatchIsWindows field. - /// - [DataMember] - public bool AzureBatchIsWindows { get; set; } - /// /// Initializes a new instance of the class. /// diff --git a/lang/cs/Org.Apache.REEF.Client/AzureBatch/Util/JobJarMaker.cs = b/lang/cs/Org.Apache.REEF.Client/AzureBatch/Util/JobJarMaker.cs index 48e62e53d9..4d434d764a 100644 --- a/lang/cs/Org.Apache.REEF.Client/AzureBatch/Util/JobJarMaker.cs +++ b/lang/cs/Org.Apache.REEF.Client/AzureBatch/Util/JobJarMaker.cs @@ -40,11 +40,9 @@ internal sealed class JobJarMaker IResourceArchiveFileGenerator resourceArchiveFileGenerator, DriverFolderPreparationHelper driverFolderPreparationHelper, REEFFileNames fileNames, - [Parameter(typeof(AzureBatchAccountKey))] string azureBatchAcc= ountKey, [Parameter(typeof(AzureBatchAccountName))] string azureBatchAc= countName, [Parameter(typeof(AzureBatchAccountUri))] string azureBatchAcc= ountUri, [Parameter(typeof(AzureBatchPoolId))] string azureBatchPoolId, - [Parameter(typeof(AzureStorageAccountKey))] string azureStorag= eAccountKey, [Parameter(typeof(AzureStorageAccountName))] string azureStora= geAccountName, [Parameter(typeof(AzureStorageContainerName))] string azureSto= rageContainerName) { @@ -53,14 +51,11 @@ internal sealed class JobJarMaker _fileNames =3D fileNames; _avroAzureBatchJobSubmissionParameters =3D new AvroAzureBatchJ= obSubmissionParameters { - AzureBatchAccountKey =3D azureBatchAccountKey, AzureBatchAccountName =3D azureBatchAccountName, AzureBatchAccountUri =3D azureBatchAccountUri, AzureBatchPoolId =3D azureBatchPoolId, - AzureStorageAccountKey =3D azureStorageAccountKey, AzureStorageAccountName =3D azureStorageAccountName, AzureStorageContainerName =3D azureStorageContainerName, - AzureBatchIsWindows =3D true }; } =20 diff --git a/lang/java/reef-bridge-client/src/main/avro/JobSubmissionParame= ters.avsc b/lang/java/reef-bridge-client/src/main/avro/JobSubmissionParamet= ers.avsc index e2a30f39b3..d3bda9f256 100644 --- a/lang/java/reef-bridge-client/src/main/avro/JobSubmissionParameters.av= sc +++ b/lang/java/reef-bridge-client/src/main/avro/JobSubmissionParameters.av= sc @@ -73,14 +73,11 @@ "doc": "Cross-language submission parameters to the Azure Batch runt= ime", "fields": [ { "name": "sharedJobSubmissionParameters", "type": "AvroJobSubmiss= ionParameters" }, - { "name": "AzureBatchAccountKey", "type": "string" }, { "name": "AzureBatchAccountName", "type": "string" }, { "name": "AzureBatchAccountUri", "type": "string" }, { "name": "AzureBatchPoolId", "type": "string" }, - { "name": "AzureStorageAccountKey", "type": "string" }, { "name": "AzureStorageAccountName", "type": "string" }, - { "name": "AzureStorageContainerName", "type": "string" }, - { "name": "AzureBatchIsWindows", "type": "boolean" } + { "name": "AzureStorageContainerName", "type": "string" } ] } ] diff --git a/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bri= dge/client/AzureBatchBootstrapDriverConfigGenerator.java b/lang/java/reef-b= ridge-client/src/main/java/org/apache/reef/bridge/client/AzureBatchBootstra= pDriverConfigGenerator.java index c054722a81..5f09abd02a 100644 --- a/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bridge/cli= ent/AzureBatchBootstrapDriverConfigGenerator.java +++ b/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bridge/cli= ent/AzureBatchBootstrapDriverConfigGenerator.java @@ -18,9 +18,6 @@ */ package org.apache.reef.bridge.client; =20 -import org.apache.avro.io.DecoderFactory; -import org.apache.avro.io.JsonDecoder; -import org.apache.avro.specific.SpecificDatumReader; import org.apache.reef.reef.bridge.client.avro.AvroAzureBatchJobSubmission= Parameters; import org.apache.reef.runtime.common.client.DriverConfigurationProvider; import org.apache.reef.runtime.common.driver.parameters.ClientRemoteIdenti= fier; @@ -28,9 +25,6 @@ =20 import javax.inject.Inject; import java.io.File; -import java.io.FileInputStream; -import java.io.IOException; -import java.io.InputStream; import java.util.logging.Level; import java.util.logging.Logger; =20 @@ -49,15 +43,10 @@ private AzureBatchBootstrapDriverConfigGenerator(final = DriverConfigurationProvid this.driverConfigurationProvider =3D driverConfigurationProvider; } =20 - Configuration getDriverConfigurationFromParams(final String bootstrapJob= ArgsLocation) throws IOException { - - final File bootstrapJobArgsFile =3D new File(bootstrapJobArgsLocation)= .getCanonicalFile(); - - final AvroAzureBatchJobSubmissionParameters azureBatchBootstrapJobArgs= =3D - readAzureBatchJobSubmissionParametersFromFile(bootstrapJobArgsFile= ); - - final String jobId =3D azureBatchBootstrapJobArgs.getSharedJobSubmissi= onParameters().getJobId().toString(); - final File jobFolder =3D new File(azureBatchBootstrapJobArgs + Configuration getDriverConfigurationFromParams( + final AvroAzureBatchJobSubmissionParameters avroAzureBatchJobSubmiss= ionParameters) { + final String jobId =3D avroAzureBatchJobSubmissionParameters.getShared= JobSubmissionParameters().getJobId().toString(); + final File jobFolder =3D new File(avroAzureBatchJobSubmissionParameter= s .getSharedJobSubmissionParameters().getJobSubmissionFolder().toStr= ing()); =20 LOG.log(Level.INFO, "jobFolder {0} jobId {1}.", new Object[]{jobFolder= .toURI(), jobId}); @@ -66,20 +55,4 @@ Configuration getDriverConfigurationFromParams(final Str= ing bootstrapJobArgsLoca jobFolder.toURI(), ClientRemoteIdentifier.NONE, jobId, Constants.DRIVER_CONFIGURATION_WITH_HTTP_AND_NAMESERVER); } - - private AvroAzureBatchJobSubmissionParameters readAzureBatchJobSubmissio= nParametersFromFile(final File file) - throws IOException { - try (final FileInputStream fileInputStream =3D new FileInputStream(fil= e)) { - return readAzureBatchSubmissionParametersFromInputStream(fileInputSt= ream); - } - } - - private static AvroAzureBatchJobSubmissionParameters readAzureBatchSubmi= ssionParametersFromInputStream( - final InputStream inputStream) throws IOException { - final JsonDecoder decoder =3D DecoderFactory.get().jsonDecoder( - AvroAzureBatchJobSubmissionParameters.getClassSchema(), inputStrea= m); - final SpecificDatumReader reade= r =3D new SpecificDatumReader<>( - AvroAzureBatchJobSubmissionParameters.class); - return reader.read(null, decoder); - } } diff --git a/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bri= dge/client/AzureBatchBootstrapREEFLauncher.java b/lang/java/reef-bridge-cli= ent/src/main/java/org/apache/reef/bridge/client/AzureBatchBootstrapREEFLaun= cher.java index 35a1e5942d..4f2a3f66f3 100644 --- a/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bridge/cli= ent/AzureBatchBootstrapREEFLauncher.java +++ b/lang/java/reef-bridge-client/src/main/java/org/apache/reef/bridge/cli= ent/AzureBatchBootstrapREEFLauncher.java @@ -23,10 +23,21 @@ import org.apache.avro.specific.SpecificDatumReader; import org.apache.reef.annotations.audience.Interop; import org.apache.reef.reef.bridge.client.avro.AvroAzureBatchJobSubmission= Parameters; -import org.apache.reef.runtime.azbatch.client.AzureBatchRuntimeConfigurati= on; -import org.apache.reef.runtime.azbatch.client.AzureBatchRuntimeConfigurati= onCreator; +import org.apache.reef.runtime.azbatch.AzureBatchClasspathProvider; +import org.apache.reef.runtime.azbatch.AzureBatchJVMPathProvider; +import org.apache.reef.runtime.azbatch.client.AzureBatchDriverConfiguratio= nProviderImpl; +import org.apache.reef.runtime.azbatch.parameters.AzureBatchAccountName; +import org.apache.reef.runtime.azbatch.parameters.AzureBatchAccountUri; +import org.apache.reef.runtime.azbatch.parameters.AzureBatchPoolId; +import org.apache.reef.runtime.azbatch.parameters.AzureStorageAccountName; +import org.apache.reef.runtime.azbatch.parameters.AzureStorageContainerNam= e; +import org.apache.reef.runtime.azbatch.util.command.CommandBuilder; +import org.apache.reef.runtime.azbatch.util.command.WindowsCommandBuilder; import org.apache.reef.runtime.common.REEFEnvironment; +import org.apache.reef.runtime.common.client.DriverConfigurationProvider; import org.apache.reef.runtime.common.evaluator.PIDStoreStartHandler; +import org.apache.reef.runtime.common.files.RuntimeClasspathProvider; +import org.apache.reef.runtime.common.files.RuntimePathProvider; import org.apache.reef.runtime.common.launch.REEFErrorHandler; import org.apache.reef.runtime.common.launch.REEFMessageCodec; import org.apache.reef.tang.Configuration; @@ -73,9 +84,10 @@ public static void main(final String[] args) throws IOEx= ception, InjectionExcept throw fatal(message, new IllegalArgumentException(message)); } =20 - final File partialConfigFile =3D new File(args[0]); + final AvroAzureBatchJobSubmissionParameters avroAzureBatchJobSubmissio= nParameters =3D + readAvroJobSubmissionParameters(new File(args[0])); final AzureBatchBootstrapDriverConfigGenerator azureBatchBootstrapDriv= erConfigGenerator =3D - TANG.newInjector(generateConfigurationFromJobSubmissionParameters(= partialConfigFile)) + TANG.newInjector(generateConfiguration(avroAzureBatchJobSubmission= Parameters)) .getInstance(AzureBatchBootstrapDriverConfigGenerator.class); =20 final Configuration launcherConfig =3D @@ -87,7 +99,8 @@ public static void main(final String[] args) throws IOExc= eption, InjectionExcept .build(); =20 try (final REEFEnvironment reef =3D REEFEnvironment.fromConfiguration( - azureBatchBootstrapDriverConfigGenerator.getDriverConfigurationFro= mParams(args[0]), launcherConfig)) { + azureBatchBootstrapDriverConfigGenerator.getDriverConfigurationFro= mParams( + avroAzureBatchJobSubmissionParameters), launcherConfig)) { reef.run(); } catch (final InjectionException ex) { throw fatal("Unable to configure and start REEFEnvironment.", ex); @@ -98,33 +111,35 @@ public static void main(final String[] args) throws IO= Exception, InjectionExcept System.exit(0); // TODO[REEF-1715]: Should be able to exit cleanly at = the end of main() } =20 - private static Configuration generateConfigurationFromJobSubmissionParam= eters(final File params) throws IOException { - + private static AvroAzureBatchJobSubmissionParameters readAvroJobSubmissi= onParameters( + final File paramsFile) throws IOException { final AvroAzureBatchJobSubmissionParameters avroAzureBatchJobSubmissio= nParameters; - - try (final FileInputStream fileInputStream =3D new FileInputStream(par= ams)) { + try (final FileInputStream fileInputStream =3D new FileInputStream(par= amsFile)) { final JsonDecoder decoder =3D DecoderFactory.get().jsonDecoder( AvroAzureBatchJobSubmissionParameters.getClassSchema(), fileInpu= tStream); final SpecificDatumReader rea= der =3D new SpecificDatumReader<>(AvroAzureBatchJobSubmissionParameters.= class); avroAzureBatchJobSubmissionParameters =3D reader.read(null, decoder)= ; } + return avroAzureBatchJobSubmissionParameters; + } =20 - return AzureBatchRuntimeConfigurationCreator - .getOrCreateAzureBatchRuntimeConfiguration(avroAzureBatchJobSubmis= sionParameters.getAzureBatchIsWindows()) - .set(AzureBatchRuntimeConfiguration.AZURE_BATCH_ACCOUNT_NAME, + private static Configuration generateConfiguration( + final AvroAzureBatchJobSubmissionParameters avroAzureBatchJobSubmiss= ionParameters) { + return TANG.newConfigurationBuilder() + .bindImplementation(DriverConfigurationProvider.class, AzureBatchD= riverConfigurationProviderImpl.class) + .bindImplementation(RuntimeClasspathProvider.class, AzureBatchClas= spathProvider.class) + .bindImplementation(RuntimePathProvider.class, AzureBatchJVMPathPr= ovider.class) + .bindImplementation(CommandBuilder.class, WindowsCommandBuilder.cl= ass) + .bindNamedParameter(AzureBatchAccountName.class, avroAzureBatchJobSubmissionParameters.getAzureBatchAccountName= ().toString()) - .set(AzureBatchRuntimeConfiguration.AZURE_BATCH_ACCOUNT_KEY, - avroAzureBatchJobSubmissionParameters.getAzureBatchAccountKey(= ).toString()) - .set(AzureBatchRuntimeConfiguration.AZURE_BATCH_ACCOUNT_URI, + .bindNamedParameter(AzureBatchAccountUri.class, avroAzureBatchJobSubmissionParameters.getAzureBatchAccountUri(= ).toString()) - .set(AzureBatchRuntimeConfiguration.AZURE_BATCH_POOL_ID, + .bindNamedParameter(AzureBatchPoolId.class, avroAzureBatchJobSubmissionParameters.getAzureBatchPoolId().to= String()) - .set(AzureBatchRuntimeConfiguration.AZURE_STORAGE_ACCOUNT_NAME, + .bindNamedParameter(AzureStorageAccountName.class, avroAzureBatchJobSubmissionParameters.getAzureStorageAccountNa= me().toString()) - .set(AzureBatchRuntimeConfiguration.AZURE_STORAGE_ACCOUNT_KEY, - avroAzureBatchJobSubmissionParameters.getAzureStorageAccountKe= y().toString()) - .set(AzureBatchRuntimeConfiguration.AZURE_STORAGE_CONTAINER_NAME, + .bindNamedParameter(AzureStorageContainerName.class, avroAzureBatchJobSubmissionParameters.getAzureStorageContainer= Name().toString()) .build(); } =20 ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. =20 For queries about this service, please contact Infrastructure at: users@infra.apache.org > Azure Batch credentials exposed in job-submission-params.json in REEF.NET > ------------------------------------------------------------------------- > > Key: REEF-2031 > URL: https://issues.apache.org/jira/browse/REEF-2031 > Project: REEF > Issue Type: Improvement > Components: REEF.NET Driver > Reporter: Chenxi Zhao > Assignee: Chenxi Zhao > Priority: Major > > When running REEF .NET to submit a job to AzureBatch, it generates=C2=A0j= ob-submission-params.json file which carries configuration data from .NET c= lient to Java bootstrap launcher. > =C2=A0 > Azure Batch REEF .Java is using "AZ_BATCH_AUTH_TOKEN_ENV" for Azure Batch= credential and shared access token for BlobStorage. > =C2=A0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)