hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Urso <anthony.u...@gmail.com>
Subject Re: HDFS single node cluster vs. NTFS performance comparison
Date Thu, 24 Sep 2009 03:40:08 GMT
The Annals of Improbable Research may be interested.  I believe they
recently published a study comparing apples and oranges.

Cheers,
Anthony

On Wed, Sep 23, 2009 at 8:11 AM, HarishKashyap TS
<HarishKashyap_TS@infosys.com> wrote:
> Hi All,
>
>
>
> I have completed a performance testing activity of HDFS single node vs. NTFS
> file systems. Modified versions of SLG tools provided by Hadoop has been
> utilized for this activity. Under similar environment conditions,
> performance of the two file systems has been compared across various file
> operations.
>
> From our tests, statistics related to the amount of overhead introduced by
> HDFS can be obtained.
>
> For E.g. If number of file created is considers as a metric, then, local
> file system (NTFS) performs 30% better when compared to HDFS.
>
>
>
> We are planning to publish an article on this. Suggestions about the
> technical forums, where the publication of this article would be
> appropriate, will be of great help.
>
>
>
> Aaron,
>
> Thanks a lot for your inputs and time.
>
>
>
> Regards,
>
> Harish Kashyap
>
>
>
> From: Aaron Kimball [mailto:aaron@cloudera.com]
> Sent: Wednesday, September 23, 2009 1:49 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: HDFS single node cluster vs. NTFS performance comparison
>
>
>
> To my knowledge, nobody's benchmarked this in a rigorous fashion. It's
> virtually certain, though, that on the same machine, NTFS would perform
> faster. HDFS does not directly write to the disk driver, it uses the local
> filesystem of the node on which it's installed. So any HDFS writes would
> themselves be channeled through NTFS and then down to the disk. The read
> path, of course, would go through NTFS first and then via HDFS out to the
> client.
>
> So, HDFS can only add overhead. How much overhead is probably not a
> published number.
>
> - Aaron
>
> On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS
> <HarishKashyap_TS@infosys.com> wrote:
>
> Hi All,
>
>
>
> Has performance testing and comparison of HDFS single node cluster vs. NTFS
> file systems been performed? Any sample results of HDFS single node vs. NTFS
> performance comparison available?
>
>
>
> Your input/feedback regarding this would be very helpful.
>
>
>
> Regards,
>
> Harish Kashyap
>
>
>
>

Mime
View raw message