hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11299) Support multiple Datanode File IO hooks
Date Fri, 06 Jan 2017 23:49:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15806199#comment-15806199

Arpit Agarwal commented on HDFS-11299:

Thanks for this improvement [~hanishakoneru]. Comments below:

# FaultInjectorFileIoEvents#beforeMetadataOp and FaultInjectorFileIoEvents#beforeFileIo can
return void.
# We can just remove FileIoProvider#getStatistics, also FaultInjectorFileIoEvents#getStatistics
and Datanode#getFileIoProviderStatistics.
# We should keep the DiskChecker invocation out of ProfilingFileIoEvents. One way is to add
a failure handler in FileIoProvider which invokes diskchecker then invokes ProfilingFileIoEvents#onFailure.
# Minor: The config key names can be simplified. e.g. {{dfs.datanode.fileio.events.enabled.profiling}}
==> {{dfs.datanode.enable.fileio.profiling}}.

Looks good otherwise.

> Support multiple Datanode File IO hooks
> ---------------------------------------
>                 Key: HDFS-11299
>                 URL: https://issues.apache.org/jira/browse/HDFS-11299
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs
>            Reporter: Hanisha Koneru
>            Assignee: Hanisha Koneru
>         Attachments: HDFS-11299.000.patch
> HDFS-10958 introduces instrumentation hooks around DataNode disk IO and HDFS-10959 adds
support for profiling hooks to expose latency statistics. 
> Instead of choosing only one hook using Config parameters, we want to add two separate
hooks - one for profiling and one for fault injection. The fault injection hook will be useful
for testing purposes. 
> This jira only introduces support for fault injection hook. The implementation for that
will come later on.
> Also, now Default and Counting FileIOEvents would not be needed as we can control enabling
the profiling and fault injection hooks using config parameters.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message