hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-708) A stress-test tool for HDFS.
Date Thu, 01 Apr 2010 19:09:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12852509#action_12852509
] 

Konstantin Shvachko commented on HDFS-708:
------------------------------------------

Steve> Might be good for this to be independent of hdfs. That way, we can stress any filesystem.


DFSIO uses generic FileSystem API, so it should be able to stress any fs implementing it.
I remember somebody actually used it to benchmark MR over Lustre.

Joshua> 
1. What is the difference between running a "create" set of jobs vs. "generating in a previous
job"?
2. I agree, it's a good thing to have.
3. These are good generalizations of the framework. We should consider them as possible improvements
of the tool in the future.

> A stress-test tool for HDFS.
> ----------------------------
>
>                 Key: HDFS-708
>                 URL: https://issues.apache.org/jira/browse/HDFS-708
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: test, tools
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.22.0
>
>         Attachments: SLiveTest.pdf
>
>
> It would be good to have a tool for automatic stress testing HDFS, which would provide
IO-intensive load on HDFS cluster.
> The idea is to start the tool, let it run overnight, and then be able to analyze possible
failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message