hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8940) Support for large-scale multi-tenant inotify service
Date Thu, 23 Jun 2016 06:38:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345892#comment-15345892

Ming Ma commented on HDFS-8940:

[~zhz], thanks for the follow-up. Shortly after posting the design, we de-prioritized the
work and haven't done anything. It is great you can pick it up.

Regarding using Kafka, your suggestion of building an independent app is a good idea. At least
you can try it out. Our case might be somewhat different from yours, where we have large number
of clients pulling infrequently changed files; thus the question if Kafka can scale to handle
broadcast to large number of consumers.

After you get it working, it might be useful to abstract the underlying notification channel,
so that others can use other systems besides Kafka if necessary.

To quantify the benefit, maybe we can measure the write performance, # of blocks created per

When applications use the service, is it going to be similar to the existing DFSInotifyEventInputStream?

> Support for large-scale multi-tenant inotify service
> ----------------------------------------------------
>                 Key: HDFS-8940
>                 URL: https://issues.apache.org/jira/browse/HDFS-8940
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>         Attachments: Large-Scale-Multi-Tenant-Inotify-Service.pdf
> HDFS-6634 provides the core inotify functionality. We would like to extend that to provide
a large-scale service that ten of thousands of clients can subscribe to.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message