hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-14220) Enhance S3GuardTool with bucket-info and set-capacity commands, tests
Date Fri, 15 Sep 2017 14:26:00 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-14220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-14220:
------------------------------------
    Attachment: HADOOP-14220-012.patch

HADOOP-14220 patch 012
* S3GuardTool bucked-info creates a new FS instance, so won't pick up an old one in a test
run
* LocalMetadataStore.toString() improved
* Some more logging in an assertion failure in {{AbstractS3GuardToolTestBase.testPruneCommand}}.
Not seen the error again though.

Tested. yes, without s3guard, with s3guard local and s3guard dynamo., Saw one test failure,
once, in testPruneCommand. Added some more diagnostics so if it surfaces again we can see
a bit more as to what happened
{code}
Skipped: 0, Time elapsed: 54.485 sec <<< FAILURE! - in org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDBtestPruneCommandConf(org.apache.hadoop.fs.s3a.s3guard.ITestS3GuardToolDynamoDB)
 Time elapsed: 6.305 sec  <<< FAILURE! java.lang.AssertionError: Pruned children
count expected:<1> but was:<0>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:743)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.junit.Assert.assertEquals(Assert.java:555)
	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommand(AbstractS3GuardToolTestBase.java:195)
	at org.apache.hadoop.fs.s3a.s3guard.AbstractS3GuardToolTestBase.testPruneCommandConf(AbstractS3GuardToolTestBase.java:215)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}

> Enhance S3GuardTool with bucket-info and set-capacity commands, tests
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-14220
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14220
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-beta1
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-14220-006.patch, HADOOP-14220-008.patch, HADOOP-14220-009.patch,
HADOOP-14220-010.patch, HADOOP-14220-012.patch, HADOOP-14220-HADOOP-13345-001.patch, HADOOP-14220-HADOOP-13345-002.patch,
HADOOP-14220-HADOOP-13345-003.patch, HADOOP-14220-HADOOP-13345-004.patch, HADOOP-14220-HADOOP-13345-005.patch
>
>
> Add a diagnostics command to s3guard which does whatever we need to diagnose problems
for a specific (named) s3a url. This is something which can be attached to bug reports as
well as used by developers.
> * Properties to log (with provenance attribute, which can track bucket overrides: s3guard
metastore setup, autocreate, capacity,.... 
> * table present/absent
> * # of keys in DDB table for that bucket?
> * any other stats?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message