Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of anthony.urso@gmail.com
 designates 209.85.216.190 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=QYJOckJpPvW3t6/Di3/m5EvlKxL5tgwhHVfwycDylKLbdnwcWNkJUIqQun18M/WuAr
         T9tRatdrlH9s4qTx8FQMsWvritkzBS1y7r0lUJrVTJMz9Q8YTFz9EKb29cGwbHO1AYwy
         huAAqgcot0TYccbg5qAjL7ISwi5BraZuU07uI=
MIME-Version: 1.0
In-Reply-To: 
 <F5CAE06FA663E14FBF21A6C3C6FE09F71315F2F8AD@BLRKECMBX09.ad.infosys.com>
References: 
 <F5CAE06FA663E14FBF21A6C3C6FE09F71310F9327C@BLRKECMBX09.ad.infosys.com>
	 <d6d7c4410909221318n341698ccnc9eddd54eadf7404@mail.gmail.com>
	 <F5CAE06FA663E14FBF21A6C3C6FE09F71315F2F8AD@BLRKECMBX09.ad.infosys.com>
Date: Wed, 23 Sep 2009 20:40:08 -0700
Message-ID: <ba72f77f0909232040k1959785bua683511c7efbe1da@mail.gmail.com>
Subject: Re: HDFS single node cluster vs. NTFS performance comparison
From: Anthony Urso <anthony.urso@gmail.com>
To: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

The Annals of Improbable Research may be interested.  I believe they
recently published a study comparing apples and oranges.

Cheers,
Anthony

On Wed, Sep 23, 2009 at 8:11 AM, HarishKashyap TS
<HarishKashyap_TS@infosys.com> wrote:
> Hi All,
>
>
>
> I have completed a performance testing activity of HDFS single node vs. NTFS
> file systems. Modified versions of SLG tools provided by Hadoop has been
> utilized for this activity. Under similar environment conditions,
> performance of the two file systems has been compared across various file
> operations.
>
> From our tests, statistics related to the amount of overhead introduced by
> HDFS can be obtained.
>
> For E.g. If number of file created is considers as a metric, then, local
> file system (NTFS) performs 30% better when compared to HDFS.
>
>
>
> We are planning to publish an article on this. Suggestions about the
> technical forums, where the publication of this article would be
> appropriate, will be of great help.
>
>
>
> Aaron,
>
> Thanks a lot for your inputs and time.
>
>
>
> Regards,
>
> Harish Kashyap
>
>
>
> From: Aaron Kimball [mailto:aaron@cloudera.com]
> Sent: Wednesday, September 23, 2009 1:49 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: HDFS single node cluster vs. NTFS performance comparison
>
>
>
> To my knowledge, nobody's benchmarked this in a rigorous fashion. It's
> virtually certain, though, that on the same machine, NTFS would perform
> faster. HDFS does not directly write to the disk driver, it uses the local
> filesystem of the node on which it's installed. So any HDFS writes would
> themselves be channeled through NTFS and then down to the disk. The read
> path, of course, would go through NTFS first and then via HDFS out to the
> client.
>
> So, HDFS can only add overhead. How much overhead is probably not a
> published number.
>
> - Aaron
>
> On Tue, Sep 22, 2009 at 7:25 AM, HarishKashyap TS
> <HarishKashyap_TS@infosys.com> wrote:
>
> Hi All,
>
>
>
> Has performance testing and comparison of HDFS single node cluster vs. NTFS
> file systems been performed? Any sample results of HDFS single node vs. NTFS
> performance comparison available?
>
>
>
> Your input/feedback regarding this would be very helpful.
>
>
>
> Regards,
>
> Harish Kashyap
>
>
>
>