hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan A. Pendleton" ...@geekdom.net>
Subject Re: Some queries about stability and reliability
Date Mon, 14 Aug 2006 18:59:26 GMT
The rotting issue is with blocks disappearing. It has gotten considerably
more rare, but I still find that, occasionally (every 2-3 weeks) files will
turn up with missing blocks, even though no detected errors are in progress
(datanodes may have *been* missing, but all datanodes are running, etc.).

If it happened regularly enough to be characterizable, I would have filed a
bug... it's certainly not the first time I've mentioned it. But, it happens
enough to make me not count on DFS as a long-term reliable data storage
environment, yet. The "fsck" utility is very helpful in these concerns,
certainly, but it can't bring data back into existence that is no longer
locatable on the cluster.

On 8/14/06, Konstantin Shvachko <shv@yahoo-inc.com> wrote:
>
> Bryan, could you describe in more details how your files "rot"?
> Do you loose entire files or blocks? How do you detect missing files?
> How regular is "regular basis"?
> Should we have a jira issue for that?
>
> I think Jagadeesh' q#3 was about running a spare name node on the same
> cluster,
> rather than running overlapping clusters.
>
> On the logging issue. I think we should change the default logging level,
> which is INFO at the moment.
>
> Thanks,
> Konstantin
>
>
> Bryan A. Pendleton wrote:
>
> > I'm not sure I agree with Konstantin's assurances on reliability, just
> > yet.
> > I've been running a persistent cluster of hadoop since early this
> > year, and
> > have repeatedly had data "rot". Things have gotten better lately, but
> > it's
> > not clear that all of the necessary tests and ruggedness testing have
> > been
> > done to stick this into a live production environment, especially
> without
> > additional backups in other storage systems. Still, a small portion of
> my
> > files "rot" on a regular basis.
> >
> > To 1: Yes, with a replication of 3, any 2 nodes can fail with no loss of
> > data. If the failures happen slowly enough, more failures are tolerated,
> > supposing that there's time and space for the re-replications to take
> > place.
> > I once had 5 of my ~30 nodes fail across a weekend, with no data
> becoming
> > unavailable for ongoing processes. There are currently no active tests
> > that
> > blocks which are reported present can actually be read from disk,
> though,
> > unless attempts to read the data are actually made.
> >
> > To 2: Not at this time. However, if you temporarily turn replication
> *up*
> > for a large subset of your data, wait for things to become quiescent,
> > then
> > turn replication back - you will effectively redistribute your free
> > space.
> > Be aware that things might be "wonky" during heavy up-replications.
> > I've had
> > map jobs fail because the replication load made normal block accesses
> > take
> > long enough to start losing tasks.
> >
> > To 3: Using the current state of the code, you could run overlapping DFS
> > clusters. For each namenode you run, you'd want to use different storage
> > directories on the cluster nodes.
> >
> > I don't know if you're likely to run into namenode performance
> > problems just
> > yet, though. It is worth noting that, for a very busy cluster, you'll
> > want
> > to periodically restart the namenode, to incorporate the change long
> into
> > the disk-stored image. Otherwise, the edits log might grow forever.
> > Probably
> > take a lot longer than running out of space from your over-full logs,
> > though.
> >
> > On 8/10/06, Jagadeesh <jagadeesh.vn@gmail.com> wrote:
> >
> >>
> >> Dear Konstantin Shvachko,
> >>
> >> Thanks for your reply.
> >>
> >> I have anyway decided to try Hadoop for our application and I
> >> successfully
> >> connected 22 nodes and it's working fine so far. I had one issue though
> >> where the master node generated 5 log files of size 125GB each in 5
> >> days and
> >> resulted in crashing the server. Anyway I have changed the properties
> in
> >> log4j and fixed it.
> >>
> >> I have a few more queries and really appreciate if you can give me an
> >> answer to those.
> >>
> >> 1. If I set the replication level to 3 or 4, will I be able to access
> >> the
> >> files even if one or two slave nodes go down.
> >>
> >> 2. If one of the slave nodes run out of disk space, will hadoop perform
> >> any defragmentation process by moving some data blocks onto other
> nodes?
> >>
> >> 3. Can I run multiple master nodes with the same set of slaves and
> >> then by
> >> modifying the code have the master nodes communicating to each other
> >> informing the data chunks stored in slave nodes? By this way we can run
> >> multiple master nodes and provide options for load balancing and
> >> clustering.
> >> You would be the right person to suggest an approach and I can work
> >> on it.
> >>
> >> Please let me know your thougts...
> >>
> >> Thanks and Best
> >> Jugs
> >>
> >> -----Original Message-----
> >>     From: "Konstantin Shvachko"<shv@yahoo-inc.com>
> >>     Sent: 8/10/06 11:27:13 PM
> >>     To: "hadoop-user@lucene.apache.org"<hadoop-user@lucene.apache.org>
> >>     Subject: Re: Some queries about stability and reliability
> >>
> >>     Hi Jagadeesh,
> >>
> >>     >I am very much new to Hadoop and would like to know some details
> >> about the
> >>     >reliability and stability. I am developing flickr kind of an
> >> application for
> >>     >storing and sharing movies and would like to use Hadoop as my
> >> storage
> >>     >backend. I am planning to put in atleast 100 nodes and would
> >> like to
> >> know
> >>     >more about the product. I will appreciate if you could answer
> >> some of
> >> my
> >>     >queries.
> >>     >
> >>     >
> >>     This is a very interesting application for Hadoop.
> >>     Did you have any progress with the system?
> >>
> >>     >1. Is the product matured enough for using in an application like
> >> this?
> >>     >
> >>     >
> >>     Yes.
> >>
> >>     >2. Has somebody tested it using atleast 100 nodes?
> >>     >
> >>     >
> >>     Yes, there are even larger installations.
> >>
> >>     >3. Can I have multiple master nodes in Hadoop to do load balancing
> >> and
> >>     >fail-overs?
> >>     >
> >>     >
> >>     Not yet.
> >>
> >>     >4. What is the maximum number of simultaneous connections
> >> possible in
> >>     >Hadoop?
> >>     >
> >>     >
> >>     Hadoop is designed to support and actually supports high volume of
> >>     simultaneous connections.
> >>     E.g., on a 100 node cluster an extensive map-reduce job can
> generate
> >> 400
> >>     concurrent connections.
> >>
> >>     Creation time and date is not implemented for DFS files.
> >>     Do you have a good application for ctime?
> >>
> >>     Thank you,
> >>
> >>     --Konstantin
> >>
> >>
> >
> >
>
>


-- 
Bryan A. P. Pendleton
Ph: (877) geek-1-bp

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message