hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang Xu (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-708) A stress-test tool for HDFS.
Date Tue, 26 Jan 2010 15:42:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805065#action_12805065
] 

Wang Xu commented on HDFS-708:
------------------------------

I post our design illustration here.
http://gnawux.info/hadoop/2010/01/a-simple-hdfs-performance-test-tool/

And I will post the code on google code or other place tomorrow.

In our test program, synchronizer is a server written in python, it accepts the request of
test program running in test nodes. Having received requests from all nodes, it admits them
start pressure simultaneously.

The test program is written in Java, and it starts several threads to write or read with DFSClient.
All the pressure thread record the data it has written in a variable and the main thread of
the test program collect them periodically, then written into a XML file.

Analyzing the xml output file, we can tell the performance of reading and writing.

In our test program, it supports read only, write only and read-write. And it can be set as
read files writen by itself or random files.

> A stress-test tool for HDFS.
> ----------------------------
>
>                 Key: HDFS-708
>                 URL: https://issues.apache.org/jira/browse/HDFS-708
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: test, tools
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Shvachko
>             Fix For: 0.22.0
>
>
> It would be good to have a tool for automatic stress testing HDFS, which would provide
IO-intensive load on HDFS cluster.
> The idea is to start the tool, let it run overnight, and then be able to analyze possible
failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message