hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Work logged] (HADOOP-16848) add initial S3A layering + async init
Date Fri, 25 Sep 2020 13:43:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-16848?focusedWorklogId=491105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491105
]

ASF GitHub Bot logged work on HADOOP-16848:
-------------------------------------------

                Author: ASF GitHub Bot
            Created on: 25/Sep/20 13:42
            Start Date: 25/Sep/20 13:42
    Worklog Time Spent: 10m 
      Work Description: steveloughran commented on pull request #1839:
URL: https://github.com/apache/hadoop/pull/1839#issuecomment-698228801


   mockito test failure in async store launch
   ```
   NFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.255 s - in org.apache.hadoop.fs.s3a.TestDataBlocks
   [INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.401 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
   [INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
   [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.236 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.TestListing
   [ERROR] testProvidedFileStatusIteratorEnd(org.apache.hadoop.fs.s3a.TestListing)  Time elapsed:
1.063 s  <<< ERROR!
   java.lang.NullPointerException
   	at org.apache.hadoop.fs.impl.FutureIOSupport.awaitFutureQuietly(FutureIOSupport.java:250)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.awaitStoreLive(S3AFileSystem.java:4828)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.store(S3AFileSystem.java:4838)
   	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:420)
   	at org.apache.hadoop.fs.s3a.AbstractS3AMockTest.setup(AbstractS3AMockTest.java:62)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
   	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
   	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
   	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
   	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
   ```
   
   viewing this PR as a PoC, I think the async startup code needlessly complicates life. skipping
bucket existence probes is enough to make for a fast launch.
   
   Proposed:
   
   1. pull out the request factory into its own PR
   1. Add support for requester pays
   1. Unit tests to verify request pays and encryption options passed in everywhere


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 491105)
    Time Spent: 1h 10m  (was: 1h)

> add initial S3A layering + async init
> -------------------------------------
>
>                 Key: HADOOP-16848
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16848
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.3.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Split the S3A code into layers
> * S3AFileSystem
> * S3AStore + Impl
> * RawS3A + Impl
> S3AFS will create the others and start in order: DelegationTokens, RawS3A, S3AStore,
Metastore
> this will involve wrapping all access of DTs, s3client, Metastore to block until that
layer is complete, or raise an exception if instantiation of it/predecessor failed.
> New layers will all be subclasses of Service, split into Interface and Impl, so we can
manage the init/start/stop lifecycle with existing code



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message