hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Fabbri (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7240) Object store in HDFS
Date Wed, 22 Nov 2017 22:42:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16263485#comment-16263485
] 

Aaron Fabbri commented on HDFS-7240:
------------------------------------

Thanks for taking notes, and thank you for the lively discussion.  A lot of valid concerns
on all sides here. A couple of minor corrections:
{quote}AaronF: Ozone is a lot of new code and Hadoop already has so much code.. {quote}
My concerns particularly are around things that affect stability and operability. Not concerned
about lines of code, more about "how manageable" is the codebase in terms of getting a stable
release, and supporting in production.

{quote}
shallow data copy is practical only if within same project and daemon otherwise have deal
with security setting and coordinations across daemons.
{quote}
We can factor common code, if any, to a shared dependency. I don't see how which repository
code lives in really affects fast copy between storage systems.  I can think of ways to do
it both within a JVM process consisting of code from multiple git repositories, and via IPC
(hand off ownership of a file to another process--not even talking about fancy stuff like
shmem).

{quote}
The opponents will raise the same issue as today: show feature parity 
{quote}

I get your concern, but I didn't hear anyone say feature parity. I only heard "integrate with
HDFS".  Even integrated with HDFS, there is still a high bar of "utility" to pass, IMO, to
justify a very large patch which affects production code.

We all want stable, scalable HDFS. Nobody opposes that ideal. 

I'm not sure trying to evolve HDFS to scale is a better approach than being separate with
maybe some shared, well-factored dependencies.  The latter, IMO, could result in better code
and dramatically less risk to HDFS.

Appreciate all your hard work thus far and appreciate the challenges you guys face here. I
hope you can understand my perspective as well.

> Object store in HDFS
> --------------------
>
>                 Key: HDFS-7240
>                 URL: https://issues.apache.org/jira/browse/HDFS-7240
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Jitendra Nath Pandey
>            Assignee: Jitendra Nath Pandey
>         Attachments: HDFS Scalability and Ozone.pdf, HDFS-7240.001.patch, HDFS-7240.002.patch,
HDFS-7240.003.patch, HDFS-7240.003.patch, HDFS-7240.004.patch, HDFS-7240.005.patch, HDFS-7240.006.patch,
MeetingMinutes.pdf, Ozone-architecture-v1.pdf, Ozonedesignupdate.pdf, ozone_user_v0.pdf
>
>
> This jira proposes to add object store capabilities into HDFS. 
> As part of the federation work (HDFS-1052) we separated block storage as a generic storage
layer. Using the Block Pool abstraction, new kinds of namespaces can be built on top of the
storage layer i.e. datanodes.
> In this jira I will explore building an object store using the datanode storage, but
independent of namespace metadata.
> I will soon update with a detailed design document.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message