hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stas Oskin <stas.os...@gmail.com>
Subject Re: Recommended file-system for DataNode
Date Fri, 09 Oct 2009 01:15:07 GMT
Hi.

I head about this option before, but never actually tried it.

There is also another option, called "relatime", which described as being
more compatible then noatime.
Can anyone comment on this?

Regards.

2009/10/8 Edward Capriolo <edlinuxguru@gmail.com>

> On Thu, Oct 8, 2009 at 4:00 PM, Jason Venner <jason.hadoop@gmail.com>
> wrote:
> > noatime is absolutely essential, I forget to mention it, because it is
> > automatic now for me.
> >
> > I have a fun story about atime, I have some Solaris machines with ZFS
> file
> > systems, and I was doing a find on a 6 level hashed directory tree with
> > 250000 leaf nodes.
> >
> > The find on a cold idle file system was running slowly, and the machine
> was
> > writing at 5-10MB/sec, solaris lets you toggle atime at runtime,
> > when I turned it off, the writes went to 0, and the find drastically
> speeded
> > up.
> >
> > This is very representative of a datanode with many blocks.
> >
> >
> >
> > On Thu, Oct 8, 2009 at 12:43 PM, Tom Wheeler <tomwheel@gmail.com> wrote:
> >
> >> I've used XFS on Silicon Graphics machines and JFS on AIX systems --
> >> both were quite fast and extremely reliable, though this long predates
> >> my use of Hadoop.
> >>
> >> To your question, I recently came across a blog that compares
> >> performance of several Linux filesystems:
> >>
> >>   http://log.amitshah.net/2009/04/re-comparing-file-systems.html
> >>
> >> I'd consider his results anecdotal unless the tests reflect the actual
> >> workload of a datanode, but since he's made the code available, you
> >> could probably adapt it yourself to get a better measure.
> >>
> >> On Thu, Oct 8, 2009 at 2:26 PM, Stas Oskin <stas.oskin@gmail.com>
> wrote:
> >> > Hi.
> >> >
> >> > Thanks for the info.
> >> >
> >> > What about JFS, any idea how well it compares to XFS?
> >> >
> >> > From what I read, JFS is considered more stable then XFS, but less
> >> > performing, so I wonder if this true.
> >> >
> >> > Also, Ext4 is around the corner and was recently accepted into kernel,
> so
> >> I
> >> > wonder if anyone knows about this one.
> >>
> >> --
> >> Tom Wheeler
> >> http://www.tomwheeler.com/
> >>
> >
> >
> >
> > --
> > Pro Hadoop, a book to guide you from beginner to hadoop mastery,
> > http://www.amazon.com/dp/1430219424?tag=jewlerymall
> > www.prohadoopbook.com a community for Hadoop Professionals
> >
>
> The good news is its not like you are stuck into the file system you
> pick. Assuming you use the normal replication level 3, you can pull
> out a datanode, format it's disk with any FS you want and then stick
> it back into the cluster. Hadoop should not care after all.  Not
> suggesting this...but you could theoretically run each node with a
> different file system, look at the performance and say "THIS is the
> one for me"
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message