hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-708) A stress-test tool for HDFS.
Date Tue, 13 Apr 2010 23:55:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856696#action_12856696
] 

Konstantin Shvachko commented on HDFS-708:
------------------------------------------

I am a bit confused about your iterations and inner loops.
In the document attached here there is one loop and on each iteration of this loop the individual
task firstly chooses a random operation. The operation is chosen randomly with a configurable
ratio {{slive.op.<op>.pct}}. In the uniform distribution the ration/100 is the probably
of generating this operation. In non-uniform cases this probability is skewed by the distribution
factor. So on each iteration you produce probabilities for each operation, and then generate
one according to them. And then execute the op. So it is not about how many operation of a
type you execute on an iteration (as you state it). You execute only one op. It is about what
is the probability of generating a particular operation.
I wouldn't worry about distributions now. Lets assume there is only uniform distribution for
now. We can add distributions later.

> A stress-test tool for HDFS.
> ----------------------------
>
>                 Key: HDFS-708
>                 URL: https://issues.apache.org/jira/browse/HDFS-708
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: test, tools
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.22.0
>
>         Attachments: SLiveTest.pdf
>
>
> It would be good to have a tool for automatic stress testing HDFS, which would provide
IO-intensive load on HDFS cluster.
> The idea is to start the tool, let it run overnight, and then be able to analyze possible
failures.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message