From common-dev-return-103174-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Wed Sep 18 15:04:27 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 45035180634 for ; Wed, 18 Sep 2019 17:04:27 +0200 (CEST) Received: (qmail 27191 invoked by uid 500); 18 Sep 2019 15:04:25 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 27179 invoked by uid 99); 18 Sep 2019 15:04:24 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Sep 2019 15:04:24 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 29BC9C1DDE for ; Wed, 18 Sep 2019 15:04:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.51 X-Spam-Level: ***** X-Spam-Status: No, score=5.51 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, KAM_NUMSUBJECT=0.5, MISSING_HEADERS=1.207, NORMAL_HTTP_TO_IP=0.001, NUMERIC_HTTP_ADDR=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=cloudera.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 3FznpQGyvz2S for ; Wed, 18 Sep 2019 15:04:22 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.195; helo=mail-qk1-f195.google.com; envelope-from=stevel@cloudera.com; receiver= Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id E3D02BC509 for ; Wed, 18 Sep 2019 15:04:21 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id w2so8406080qkf.2 for ; Wed, 18 Sep 2019 08:04:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:cc; bh=PYxIfD+1qB4EFwgNj0QlxVjSsZX6dPQpeAYQ1Z/qqkI=; b=iiHz/vOopX/oYLIbn5lMxeZPYWJpc6cudN8rAf01fT/vkAVud3g0mq6fD6H76pXASh XrEcumqtbxRrxHZEcuLuPKm7tcUmhZhmrFyUlURaCZYjsQlwF3VPho2Ila95H9/jidO3 r89kmbVCG/GzGfiF8i2DhbrO6l8+W949nQ2uwUt8Te4QeZYU+lDhrUo6dffZNBInP042 QpzdtiI8YtoomUyPjLDdsnCcP8HGuWoN98hmsjb2769wKiwoTKzOJpw4QttykMdwBEC/ BQJXaYQZACJ5uz9nco3tvCWPJJ8Nae2jFe+2ddalKedAJ0LfaR/HmmVAL6cuK61PcD1o 7MiQ== X-Gm-Message-State: APjAAAUYUnXbC9h94DMQNceInCWk+KW0G/O5h1k8kyqjhHfrlMuvvM2v Xyw/47f2nq4+c5uNMPTrk6joTRGEbEBFHMABw2rLsw== X-Received: by 2002:ae9:f403:: with SMTP id y3mt4296134qkl.148.1568819061277; Wed, 18 Sep 2019 08:04:21 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Steve Loughran Date: Wed, 18 Sep 2019 16:03:45 +0100 Message-ID: Subject: Re: [VOTE] Release Apache Hadoop 3.2.1 - RC0 Cc: Hdfs-dev , yarn-dev , mapreduce-dev , Hadoop Common Content-Type: multipart/alternative; boundary="000000000000de18250592d5267b" --000000000000de18250592d5267b Content-Type: text/plain; charset="UTF-8" > > +1 binding. > > One caveat: warn people that guava is now at 27.0 -and that if you run > with an older version of Guava things will inevitably break. > > > steps to validate > ================== > > * downloaded src and binary artifacts > * after import of KEYS and trusting Rohith's key, validate GPG signatures > * test basic hadoop fs commands against s3a with s3guard and abfds > > > > > Validating S3A connector > ======================== > > * grabbed the latest build of my cloudstore diagnostics JAR > https://github.com/steveloughran/cloudstore/releases/tag/tag_2019-09-13 > * and set an env var to it: > set -gx CLOUDSTORE cloudstore/target/cloudstore-0.1-SNAPSHOT.jar > > bin/hadoop jar $CLOUDSTORE storediag s3a://hwdev-steve-ireland-new > > > > Diagnostics for filesystem s3a://hwdev-steve-ireland-new/ > ========================================================= > > S3A FileSystem connector > ASF Filesystem Connector to Amazon S3 Storage and compatible stores > > https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html > > Hadoop information > ================== > > Hadoop 3.2.1 > Compiled by rohithsharmaks on 2019-09-10T15:56Z > Compiled with protoc 2.5.0 > From source with checksum 776eaf9eee9c0ffc370bcbc1888737 > > Required Classes > ================ > > All these classes must be on the classpath > > class: org.apache.hadoop.fs.s3a.S3AFileSystem > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: com.amazonaws.services.s3.AmazonS3 > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.amazonaws.ClientConfiguration > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > > Optional Classes > ================ > > These classes are needed in some versions of Hadoop. > And/or for optional features to work. > > class: com.amazonaws.services.dynamodbv2.AmazonDynamoDB > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: com.fasterxml.jackson.annotation.JacksonAnnotation > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar > class: com.fasterxml.jackson.core.JsonParseException > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar > class: com.fasterxml.jackson.databind.ObjectMapper > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar > class: org.joda.time.Interval > Not found on classpath: org.joda.time.Interval > class: org.apache.hadoop.fs.s3a.s3guard.S3Guard > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.commit.staging.StagingCommitter > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.Invoker > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > class: org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-aws-3.2.1.jar > > then some classes which aren't in 3.2 and so which I didn't expect to > see. > > class: org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens > Not found on classpath: > org.apache.hadoop.fs.s3a.auth.delegation.S3ADelegationTokens > class: com.amazonaws.services.s3.model.SelectObjectContentRequest > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.375.jar > class: org.apache.hadoop.fs.s3a.select.SelectInputStream > Not found on classpath: > org.apache.hadoop.fs.s3a.select.SelectInputStream > class: org.apache.hadoop.fs.s3a.impl.RenameOperation > Not found on classpath: > org.apache.hadoop.fs.s3a.impl.RenameOperation > > > > + the command then executed basic list/read/write operations; all good. > > > > Validating abfs connector > ========================= > > * set -gx HADOOP_OPTIONAL_TOOLS hadoop-azure > > > > > Diagnostics for filesystem abfs://container@someone.dfs.core.windows.net/ > > ============================================================================ > > Azure Abfs connector > ASF Filesystem Connector to Microsoft Azure ABFS Storage > https://hadoop.apache.org/docs/current/hadoop-azure/index.html > > Hadoop information > ================== > > Hadoop 3.2.1 > Compiled by rohithsharmaks on 2019-09-10T15:56Z > Compiled with protoc 2.5.0 > From source with checksum 776eaf9eee9c0ffc370bcbc1888737 > > Environment Variables > ===================== > > HADOOP_HOME = "/Users/stevel/hadoop/candidates/hadoop-3.2.1" > HADOOP_CONF_DIR = "/Users/stevel/hadoop-3.2.1/etc/hadoop" > HADOOP_OPTIONAL_TOOLS = > "hadoop-azure,hadoop-aws,hadoop-azure-datalake,hadoop-openstack" > HADOOP_SHELL_SCRIPT_DEBUG = (unset) > HADOOP_TOOLS_HOME = (unset) > HADOOP_TOOLS_OPTIONS = (unset) > HDP_VERSION = (unset) > SPARK_HOME = (unset) > SPARK_CONF_DIR = (unset) > PYSPARK_DRIVER_PYTHON = (unset) > SPARK_SCALA_VERSION = (unset) > YARN_CONF_DIR = (unset) > HADOOP_TOKEN_FILE_LOCATION = (unset) > HADOOP_TOKEN = (unset) > > > > Hadoop Options > ============== > > fs.defaultFS = "file:///" [core-default.xml] > fs.default.name = "file:///" > hdp.version = (unset) > yarn.resourcemanager.address = "0.0.0.0:8032" [yarn-default.xml] > yarn.resourcemanager.webapp.address = "0.0.0.0:8088" [yarn-default.xml] > yarn.resourcemanager.webapp.https.address = "0.0.0.0:8090" > [yarn-default.xml] > mapreduce.input.fileinputformat.list-status.num-threads = "1" > [mapred-default.xml] > > > Required Classes > ================ > > All these classes must be on the classpath > > class: com.fasterxml.jackson.annotation.JsonProperty > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-annotations-2.9.8.jar > class: com.google.common.base.Preconditions > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/guava-27.0-jre.jar > class: com.fasterxml.jackson.core.JsonFactory > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-core-2.9.8.jar > class: com.fasterxml.jackson.databind.ObjectReader > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/jackson-databind-2.9.8.jar > class: com.microsoft.azure.storage.StorageErrorCode > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/azure-storage-7.0.0.jar > class: org.apache.http.client.utils.URIBuilder > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/common/lib/httpclient-4.5.6.jar > class: org.wildfly.openssl.OpenSSLProvider > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/wildfly-openssl-1.0.7.Final.jar > class: org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem > > file:/Users/stevel/hadoop-3.2.1/share/hadoop/tools/lib/hadoop-azure-3.2.1.jar > > -again a full filesystem list/read/write/delete sequence was executed; > again all good. > > > ------------------------------------------- > > spark > ===== > > > (branch master; version 3.0.0-SNAPSHOT) > > After editing my maven settings to avoid our in-house nexus server, I did > a clean build with the profiles: > > -Dhadoop.version=3.2.1 -Phive -Phive-thriftserver -Pyarn -Phadoop-3.2 > -Phadoop-cloud -Psnapshots-and-staging > > > Downloading from ASF Staging: > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/3.2.1/hadoop-project-3.2.1.pom > Downloaded from ASF Staging: > https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/3.2.1/hadoop-project-3.2.1.pom > (70 kB at 72 kB/s) > > > For the record, three issues surfaced there > * spark now demands mvn 3.6.2, so upgrade time there > * for some reason the build decided to download all the other artifacts > that maven central has ever hosted > * spark's build hangs if you pass in the -T 1C argument to maven. Worth > remembering. > * and of couse you have to rebuild the next day. > > Spark compilation completed: I did not run its test suites. > > spark cloud integration tests > ============================= > > Once spark was built, I went with my downstream test suite. > https://github.com/hortonworks-spark/cloud-integration > > Here here I got burned at runtime by the guava change; the module was set > up to use guava 0.19, and Configuration.set() failed. > > S3ACommitBulkDataSuite: > com.cloudera.spark.cloud.s3.commit.S3ACommitBulkDataSuite *** ABORTED *** > java.lang.RuntimeException: Unable to load a Suite class that was > discovered in the runpath: > com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V > at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$2(CloudSuiteTrait.scala:145) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$2$adapted(CloudSuiteTrait.scala:144) > at scala.Option.foreach(Option.scala:274) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$1(CloudSuiteTrait.scala:144) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.$anonfun$overlayConfiguration$1$adapted(CloudSuiteTrait.scala:143) > at scala.collection.immutable.List.foreach(List.scala:392) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.overlayConfiguration(CloudSuiteTrait.scala:143) > at > com.cloudera.spark.cloud.common.CloudSuiteTrait.overlayConfiguration$(CloudSuiteTrait.scala:142) > at > com.cloudera.spark.cloud.common.CloudSuite$.overlayConfiguration(CloudSuite.scala:41) > ... > > This is one of those things which was inevitable. I do think we'll need to > warn people in the release notes. > > (I also discovered someone has deleted a test source data bucket of mine > so some tests failed; not something I fixed today.) > > With the guava tests, other than a failure due to someone deleting a > bucket of mine containing test source data, all tests against s3a and abfs > were happy. > > > --000000000000de18250592d5267b--