hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From HarishKashyap TS <HarishKashyap...@infosys.com>
Subject RE: HDFS single node cluster vs. NTFS performance comparison
Date Wed, 23 Sep 2009 15:11:07 GMT
Hi All,

I have completed a performance testing activity of HDFS single node vs. NTFS file systems.
Modified versions of SLG tools provided by Hadoop has been utilized for this activity. Under
similar environment conditions, performance of the two file systems has been compared across
various file operations.
>From our tests, statistics related to the amount of overhead introduced by HDFS can be
For E.g. If number of file created is considers as a metric, then, local file system (NTFS)
performs 30% better when compared to HDFS.

We are planning to publish an article on this. Suggestions about the technical forums, where
the publication of this article would be appropriate, will be of great help.

Thanks a lot for your inputs and time.

Harish Kashyap

From: Aaron Kimball [mailto:aaron@cloudera.com]
Sent: Wednesday, September 23, 2009 1:49 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: HDFS single node cluster vs. NTFS performance comparison

To my knowledge, nobody's benchmarked this in a rigorous fashion. It's virtually certain,
though, that on the same machine, NTFS would perform faster. HDFS does not directly write
to the disk driver, it uses the local filesystem of the node on which it's installed. So any
HDFS writes would themselves be channeled through NTFS and then down to the disk. The read
path, of course, would go through NTFS first and then via HDFS out to the client.

So, HDFS can only add overhead. How much overhead is probably not a published number.

- Aaron
On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS <HarishKashyap_TS@infosys.com<mailto:HarishKashyap_TS@infosys.com>>

Hi All,

Has performance testing and comparison of HDFS single node cluster vs. NTFS file systems been
performed? Any sample results of HDFS single node vs. NTFS performance comparison available?

Your input/feedback regarding this would be very helpful.


Harish Kashyap

View raw message