hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gil Vernik (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-12091) Issues with directories handling
Date Tue, 16 Jun 2015 08:13:01 GMT
Gil Vernik created HADOOP-12091:

             Summary: Issues with directories handling
                 Key: HADOOP-12091
                 URL: https://issues.apache.org/jira/browse/HADOOP-12091
             Project: Hadoop Common
          Issue Type: Improvement
          Components: fs/swift
            Reporter: Gil Vernik
            Assignee: Gil Vernik

OpenStack Swift doesn't have notion of directories. In Swift everything is object, stored
in some container, that belongs to account.
Current implementation contains many code that handle directories structure in Swift, in particular
functions with code that zero length object is a directory.
While it might be true for certain cases, there are also many cases where "directory" handling
cases problems and highly reduce performance.
For example, if Swift's container has dozens of objects and one of them has 0 length size,
than Swift driver thinks it's a directory and report it to
upper layer as directory. In consequence, this leads to various exceptions and crashes in
client side / upper Hadoop layer.

The propose of this Jira topic is to make directories handling in driver as an optional and
configurable. The driver will behave the same, but there
will be a configurable option that will disable directories handling and so everything will
be objects, even those with 0 length size.

This will cover cases, where clients doesn't care about directories structures.

This message was sent by Atlassian JIRA

View raw message