hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: In-Memory Reference FS implementations
Date Fri, 07 Mar 2014 09:47:23 GMT
On 7 March 2014 01:35, Jay Vyas <jayunit100@gmail.com> wrote:

> Thanks steve.  So i guess the conclusion is
> 1) Wait on HADOOP-9361.

"help" with is a better plan. I really don't look at it that often. I'll
try and get it ready to review this weekend

> 2) There definitively cannot be a strict contract for a single HCFS, based
> on your examples shown.
> HDFS effectively defines the behaviour, though it is a defacto
specification. All the tests and docs we can do can formalise the
behaviour, and show other filesystems (including file://) where they are
inconsistent. Failure modes are intractable -it is everyone's right to fail

What worries me is that there are some unintentional behaviours
-epiphenomena- that we aren't aware of, but which downstream code depends

mkdirs() being an atomic is an example -nobody made a decision to do that,
it just fell out of holding locks efficiently in the NN. Does anything
depend on int? Hopefully not -as if it does, that's something HDFS needs to
keep forever -so there'd better be a test for it.

Any others? I don't know -and that worries me. Filename, dir size and file
size assumptions are things that caused problems in the swift object store,
code also assumes that rmdir is O(1) and not O(n), so teardown of tests on
directories with many small files caused timeouts against throttled Swift

> In the meantime ill audit existing test coverage, and let me know if i can
> lend a hand in the cleanup process.

That'll be great

NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message