hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6634) inotify in HDFS
Date Tue, 08 Jul 2014 22:30:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14055611#comment-14055611
] 

Andrew Wang commented on HDFS-6634:
-----------------------------------

Hi James,

I took a quick look through the doc, had a few high-level comments:

* Would be good to specify the usecases concretely. Would help us evaluate which design is
better.
* Might also be good to give some brief background about what inotify is, how it's used in
user apps historically, any differences between Linux and us in terms of usecases
* Do we want the full-on inotify API (which means subscribing to certain events on certain
paths), or just exposing the entire edits stream to superusers? The latter would be easier
to do, might hit a bunch of our usecases. A full inotify-like API could be done later as an
enhancement.
* I think full-inotify is hard to do with just QJM, for the reasons you mentioned. One additional
thing I'd add is that QJM just sees bytes, so is unable to interpret things like a path.
* If we wanted to offload this from the NN, it should be possible to have a standalone mode
sorta like the SbNN, which follows the edit stream but exposes this new API.
* This might be an excuse to PB the edit log format, though there might be performance concerns
there.

Overall, I'm leaning toward putting this in the NN. It'd be cool to do the non-voting quorum
member for better latency, but I doubt that our latency requirements are that stringent. We
could also add a "long poll" API to QJM if this does become an issue, which would be handy
both for this and the SbNN (since it'd mean fewer edits to replay on failover, and make stale
reads less stale).

> inotify in HDFS
> ---------------
>
>                 Key: HDFS-6634
>                 URL: https://issues.apache.org/jira/browse/HDFS-6634
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client, namenode, qjm
>            Reporter: James Thomas
>            Assignee: James Thomas
>         Attachments: inotify-intro.pdf
>
>
> Design a mechanism for applications like search engines to access the HDFS edit stream.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message