hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rushabh S Shah (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12907) Allow read-only access to reserved raw for non-superusers
Date Fri, 08 Dec 2017 15:16:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283706#comment-16283706
] 

Rushabh S Shah edited comment on HDFS-12907 at 12/8/17 3:15 PM:
----------------------------------------------------------------

Test failures from #001 patch.
It fixed the actual test failures from previous patch.
I have no idea when  EC related test failures will be fixed.
{noformat}

[INFO] Running org.apache.hadoop.fs.TestUnbuffer
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.3 s <<<
FAILURE! - in org.apache.hadoop.fs.TestUnbuffer
[ERROR] testUnbufferException(org.apache.hadoop.fs.TestUnbuffer)  Time elapsed: 0.125 s  <<<
FAILURE!
java.lang.AssertionError: Expected test to throw (an instance of java.lang.UnsupportedOperationException
and exception with message a string containing "this stream org.apache.hadoop.fs.FSInputStream$$EnhancerByMockitoWithCGLIB$$67df4153
does not support unbuffering")
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.rules.ExpectedException.failDueToMissingException(ExpectedException.java:184)
	at org.junit.rules.ExpectedException.access$100(ExpectedException.java:85)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:170)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)

[INFO] Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.369 s - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[INFO] Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.555 s - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
[INFO] Running org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.583 s - in org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 180.385 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 166.602 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 268.573 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 226.743 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
[INFO] Running org.apache.hadoop.hdfs.TestErasureCodingPolicies
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.64 s - in org.apache.hadoop.hdfs.TestErasureCodingPolicies
[INFO] Running org.apache.hadoop.hdfs.TestFileChecksum
[INFO] Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 293.067 s - in org.apache.hadoop.hdfs.TestFileChecksum
[INFO] Running org.apache.hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.744 s - in org.apache.hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestUnbuffer.testUnbufferException Expected test to throw (an instance of java.lang.UnsupportedOperationException
and exception with message a string containing "this stream org.apache.hadoop.fs.FSInputStream$$EnhancerByMockitoWithCGLIB$$67df4153
does not support unbuffering")
[INFO] 
[ERROR] Tests run: 145, Failures: 1, Errors: 0, Skipped: 16
{noformat}
{{TestUnbuffer#testUnbufferException}} should be fixed by HADOOP-15056.

[~daryn]: can you please give your blessings ?



was (Author: shahrs87):
{noformat}

[INFO] Running org.apache.hadoop.fs.TestUnbuffer
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.3 s <<<
FAILURE! - in org.apache.hadoop.fs.TestUnbuffer
[ERROR] testUnbufferException(org.apache.hadoop.fs.TestUnbuffer)  Time elapsed: 0.125 s  <<<
FAILURE!
java.lang.AssertionError: Expected test to throw (an instance of java.lang.UnsupportedOperationException
and exception with message a string containing "this stream org.apache.hadoop.fs.FSInputStream$$EnhancerByMockitoWithCGLIB$$67df4153
does not support unbuffering")
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.rules.ExpectedException.failDueToMissingException(ExpectedException.java:184)
	at org.junit.rules.ExpectedException.access$100(ExpectedException.java:85)
	at org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:170)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407)

[INFO] Running org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.369 s - in org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
[INFO] Running org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
[INFO] Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.555 s - in org.apache.hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
[INFO] Running org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.583 s - in org.apache.hadoop.hdfs.server.namenode.TestSecurityTokenEditLog
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 180.385 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 166.602 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 268.573 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
[INFO] Running org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
[WARNING] Tests run: 18, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 226.743 s - in
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
[INFO] Running org.apache.hadoop.hdfs.TestErasureCodingPolicies
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.64 s - in org.apache.hadoop.hdfs.TestErasureCodingPolicies
[INFO] Running org.apache.hadoop.hdfs.TestFileChecksum
[INFO] Tests run: 30, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 293.067 s - in org.apache.hadoop.hdfs.TestFileChecksum
[INFO] Running org.apache.hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 33.744 s - in org.apache.hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy

[INFO] Results:
[INFO] 
[ERROR] Failures: 
[ERROR]   TestUnbuffer.testUnbufferException Expected test to throw (an instance of java.lang.UnsupportedOperationException
and exception with message a string containing "this stream org.apache.hadoop.fs.FSInputStream$$EnhancerByMockitoWithCGLIB$$67df4153
does not support unbuffering")
[INFO] 
[ERROR] Tests run: 145, Failures: 1, Errors: 0, Skipped: 16
{noformat}
{{TestUnbuffer#testUnbufferException}} should be fixed by HADOOP-15056.

[~daryn]: can you please give your blessings ?


> Allow read-only access to reserved raw for non-superusers
> ---------------------------------------------------------
>
>                 Key: HDFS-12907
>                 URL: https://issues.apache.org/jira/browse/HDFS-12907
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.0
>            Reporter: Daryn Sharp
>            Assignee: Rushabh S Shah
>         Attachments: HDFS-12907.001.patch, HDFS-12907.patch
>
>
> HDFS-6509 added a special /.reserved/raw path prefix to access the raw file contents
of EZ files.  In the simplest sense it doesn't return the FE info in the {{LocatedBlocks}}
so the dfs client doesn't try to decrypt the data.  This facilitates allowing tools like distcp
to copy raw bytes.
> Access to the raw hierarchy is restricted to superusers.  This seems like an overly broad
restriction designed to prevent non-admins from munging the EZ related xattrs.  I believe
we should relax the restriction to allow non-admins to perform read-only operations.  Allowing
non-superusers to easily read the raw bytes will be extremely useful for regular users, esp.
for enabling webhdfs client-side encryption.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message