hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6634) inotify in HDFS
Date Tue, 19 Aug 2014 22:49:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14102990#comment-14102990
] 

Colin Patrick McCabe commented on HDFS-6634:
--------------------------------------------

I took a look at inotify-design.3.pdf

bq. Please see inotify-intro.\[latest\].pdf for background and a longer-term vision for the
system.

It's more traditional to include this kind of information in the design doc.  It seems like
it should just be a paragraph or two.  We probably want to mention the Solr and Hive use-cases
in the motivation paragraph, at least in passing.

I like the event-based design-- it will allow us to change the edit log without breaking existing
inotify clients.  Great idea.

Can we have a call like {{getMinimumAvailableEvents}}?  This would return the difference between
the last known transaction ID that the server has, and the next transaction ID we're reading.
 That way clients could know if they are falling behind (failing to keep up with the number
of events the NN is generating).  Obviously this would be an underestimate of the events that
might be out there (it only reveals what txid the NN was at when we last talked to it).

I agree with the superuser-only decision and the pull model.  Good rationales.

Future work: this section might better be titled "potential future work"?

> inotify in HDFS
> ---------------
>
>                 Key: HDFS-6634
>                 URL: https://issues.apache.org/jira/browse/HDFS-6634
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: hdfs-client, namenode, qjm
>            Reporter: James Thomas
>            Assignee: James Thomas
>         Attachments: HDFS-6634.2.patch, HDFS-6634.3.patch, HDFS-6634.4.patch, HDFS-6634.5.patch,
HDFS-6634.patch, inotify-design.2.pdf, inotify-design.3.pdf, inotify-design.pdf, inotify-intro.2.pdf,
inotify-intro.pdf
>
>
> Design a mechanism for applications like search engines to access the HDFS edit stream.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message