[ https://issues.apache.org/jira/browse/HADOOP-16848?focusedWorklogId=491105&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-491105
]
ASF GitHub Bot logged work on HADOOP-16848:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 25/Sep/20 13:42
Start Date: 25/Sep/20 13:42
Worklog Time Spent: 10m
Work Description: steveloughran commented on pull request #1839:
URL: https://github.com/apache/hadoop/pull/1839#issuecomment-698228801
mockito test failure in async store launch
```
NFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.255 s - in org.apache.hadoop.fs.s3a.TestDataBlocks
[INFO] Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.401 s - in org.apache.hadoop.fs.s3a.TestS3AAWSCredentialsProvider
[INFO] Running org.apache.hadoop.fs.s3a.s3guard.TestAuthoritativePath
[ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.236 s <<<
FAILURE! - in org.apache.hadoop.fs.s3a.TestListing
[ERROR] testProvidedFileStatusIteratorEnd(org.apache.hadoop.fs.s3a.TestListing) Time elapsed:
1.063 s <<< ERROR!
java.lang.NullPointerException
at org.apache.hadoop.fs.impl.FutureIOSupport.awaitFutureQuietly(FutureIOSupport.java:250)
at org.apache.hadoop.fs.s3a.S3AFileSystem.awaitStoreLive(S3AFileSystem.java:4828)
at org.apache.hadoop.fs.s3a.S3AFileSystem.store(S3AFileSystem.java:4838)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:420)
at org.apache.hadoop.fs.s3a.AbstractS3AMockTest.setup(AbstractS3AMockTest.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
```
viewing this PR as a PoC, I think the async startup code needlessly complicates life. skipping
bucket existence probes is enough to make for a fast launch.
Proposed:
1. pull out the request factory into its own PR
1. Add support for requester pays
1. Unit tests to verify request pays and encryption options passed in everywhere
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 491105)
Time Spent: 1h 10m (was: 1h)
> add initial S3A layering + async init
> -------------------------------------
>
> Key: HADOOP-16848
> URL: https://issues.apache.org/jira/browse/HADOOP-16848
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Split the S3A code into layers
> * S3AFileSystem
> * S3AStore + Impl
> * RawS3A + Impl
> S3AFS will create the others and start in order: DelegationTokens, RawS3A, S3AStore,
Metastore
> this will involve wrapping all access of DTs, s3client, Metastore to block until that
layer is complete, or raise an exception if instantiation of it/predecessor failed.
> New layers will all be subclasses of Service, split into Interface and Impl, so we can
manage the init/start/stop lifecycle with existing code
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org
|