From common-issues-return-225644-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org Fri Jun 25 11:59:56 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id 2598E180181 for ; Fri, 25 Jun 2021 13:59:56 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id 45496628BF for ; Fri, 25 Jun 2021 11:59:55 +0000 (UTC) Received: (qmail 85954 invoked by uid 500); 25 Jun 2021 11:59:54 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 85942 invoked by uid 99); 25 Jun 2021 11:59:54 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jun 2021 11:59:54 +0000 From: =?utf-8?q?GitBox?= To: common-issues@hadoop.apache.org Subject: =?utf-8?q?=5BGitHub=5D_=5Bhadoop=5D_mehakmeet_opened_a_new_pull_request_=233?= =?utf-8?q?144=3A_HADOOP-17774=2E_bytesRead_FS_statistic_showing_twice_the_c?= =?utf-8?q?orrect_value_in_S3A?= Message-ID: Date: Fri, 25 Jun 2021 11:59:54 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit mehakmeet opened a new pull request #3144: URL: https://github.com/apache/hadoop/pull/3144 Test command: ```mvn clean verify -Dparallel-tests -DtestsThreadCount=4 -Dscale``` Region: ap-south-1 ``` INFO] Results: [INFO] [WARNING] Tests run: 568, Failures: 0, Errors: 0, Skipped: 5 ``` ``` [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut [ERROR] ITestS3AMiscOperationCost.testGetContentSummaryRoot:96->AbstractS3ACostTest.verifyMetrics:376->lambda$testGetContentSummaryRoot$1:96->getContentSummary:140 » TestTimedOut [INFO] [ERROR] Tests run: 1460, Failures: 0, Errors: 2, Skipped: 462 ``` ``` [ERROR] Errors: [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:267 » TestTimedOut [INFO] [ERROR] Tests run: 151, Failures: 2, Errors: 1, Skipped: 28 ``` Seeing these errors: ``` [ERROR] Failures: [ERROR] ITestS3AContractRootDir.testListEmptyRootDirectory:82->AbstractContractRootDirectoryTest.testListEmptyRootDirectory:196->Assert.fail:89 Deleted file: unexpectedly found s3a://mehakmeet-singh-data/user as S3AFileStatus{path=s3a://mehakmeet-singh-data/user; isDirectory=true; modification_time=0; access_time=0; owner=mehakmeet.singh; group=mehakmeet.singh; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRmEmptyRootDirNonRecursive:101->Assert.fail:89 After 20 attempts: listing after rm /* not empty final [00] S3AFileStatus{path=s3a://mehakmeet-singh-data/user; isDirectory=true; modification_time=0; access_time=0; owner=mehakmeet.singh; group=mehakmeet.singh; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag=null versionId=null ``` have seen these errors intermittently, due to some issue with DynamoDB table. @steveloughran suggested ```hadoop org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable -force s3a://example-bucket/``` But that fails with an error: ``` 2021-06-25 16:18:43,464 INFO service.AbstractService: Service PurgeS3GuardDynamoTable failed in state STARTED -1: Filesystem has no metadata store: s3a://mehakmeet-singh-data at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.failure(AbstractS3GuardDynamoDBDiagnostic.java:115) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.require(AbstractS3GuardDynamoDBDiagnostic.java:94) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.bindStore(AbstractS3GuardDynamoDBDiagnostic.java:157) at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardDynamoDBDiagnostic.bindFromCLI(AbstractS3GuardDynamoDBDiagnostic.java:147) at org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable.serviceStart(PurgeS3GuardDynamoTable.java:123) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194) at org.apache.hadoop.service.launcher.ServiceLauncher.coreServiceLaunch(ServiceLauncher.java:619) at org.apache.hadoop.service.launcher.ServiceLauncher.launchService(ServiceLauncher.java:494) at org.apache.hadoop.fs.s3a.s3guard.DumpS3GuardDynamoTable.serviceMain(DumpS3GuardDynamoTable.java:517) at org.apache.hadoop.fs.s3a.s3guard.PurgeS3GuardDynamoTable.main(PurgeS3GuardDynamoTable.java:205) 2021-06-25 16:18:43,467 INFO util.ExitUtil: Exiting with status -1: Filesystem has no metadata store: s3a://mehakmeet-singh-data ``` Would like the reviewers to run the aws test suite once in their setup while reviewing as well. CC: @steveloughran @mukund-thakur @bogthe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org For additional commands, e-mail: common-issues-help@hadoop.apache.org