hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Any related paper on how to resolve hadoop SPOF issue?
Date Tue, 30 Aug 2011 03:15:41 GMT
Andy,

We did run HBase on Ceph but were not able to get a stable performance for
sustained periods. It would give good throughput for a while and the client
would crash soon thereafter. However, this was more than a year ago and a
re-evaluation should be done before commenting further.

-Amandeep

On Mon, Aug 29, 2011 at 8:09 PM, Alex Nelson <ajnelson@cs.ucsc.edu> wrote:

> Hi Andy,
>
> When I ran Hadoop on Ceph, I was able to complete a run of Terasort.  We
> didn't run HBase.
>
> After I submitted a patch to the Ceph mailing list, Noah Watkins (CC'd)
> took over developing the file system class in Hadoop's code base.  I don't
> know what's become of the work since then.  The JIRA is here:
> https://issues.apache.org/jira/browse/HADOOP-6779
>
> The last I heard, which was a while ago, there was a snag with the URI
> scheme (ceph:// vs. file:// or something similar) which had a subtle but
> challenging complexity.  Noah is more able to answer questions at this
> point.  He has posted Java RADOS bindings to the Ceph mailing list recently,
> but I'm not sure what he has been up to aside from that.
>
> --Alex
>
>
> On Aug 27, 2011, at 02:07 , Andrew Purtell wrote:
>
> > Hi Alex,
> >
> >> UC Santa Cruz contributed a ;login: article describing replacing HDFS
> with
> >> Ceph.  (I was one of the authors.)  One of the key architectural
> advantages of
> >> Ceph over HDFS is that Ceph distributes its metadata service over
> multiple
> >> metadata servers.
> >
> > Were you able to maintain stable operation with Ceph at a high load? Did
> you try running HBase?
> >
> > Best regards,
> >
> >
> >        - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
> >
> >
> > ----- Original Message -----
> >> From: Alex Nelson <ajnelson@cs.ucsc.edu>
> >> To: George Kousiouris <gkousiou@mail.ntua.gr>
> >> Cc: common-dev@hadoop.apache.org; common-user@hadoop.apache.org; Carlos
> Maltzahn <carlosm@soe.ucsc.edu>
> >> Sent: Friday, August 26, 2011 2:44 AM
> >> Subject: Re: Any related paper on how to resolve hadoop SPOF issue?
> >>
> >> Hi George,
> >>
> >> UC Santa Cruz contributed a ;login: article describing replacing HDFS
> with
> >> Ceph.  (I was one of the authors.)  One of the key architectural
> advantages of
> >> Ceph over HDFS is that Ceph distributes its metadata service over
> multiple
> >> metadata servers.
> >>
> >> I hope that helps.
> >>
> >> --Alex
> >>
> >>
> >> On Aug 25, 2011, at 02:54 , George Kousiouris wrote:
> >>
> >>>
> >>> Hi,
> >>>
> >>> many thanks!!
> >>>
> >>> I also found this one, which has some related work also with other
> efforts,
> >> in case it is helpful for someone else:
> >>>
> >>
> https://ritdml.rit.edu/bitstream/handle/1850/13321/ATalwalkarThesis1-2011.pdf?sequence=1
> >>>
> >>>
> >>> BR,
> >>> George
> >>>
> >>> On 8/25/2011 12:46 PM, Nan Zhu wrote:
> >>>> Hope it helps
> >>>>
> >>>> http://www.springerlink.com/content/h17r882710314147/
> >>>>
> >>>> Best,
> >>>>
> >>>> Nan
> >>>>
> >>>> On Thu, Aug 25, 2011 at 5:43 PM, George
> >> Kousiouris<gkousiou@mail.ntua.gr>wrote:
> >>>>
> >>>>> Hi guys,
> >>>>>
> >>>>> We are currently in the process of writing a paper regarding hadoop
> >> and we
> >>>>> would like to reference any attempt to remove the single point of
> >> failure of
> >>>>> the Namenode. We have found in various presentations some efforts
> >> (like
> >>>>> dividing the namespace between more than one namenodes if i
> >> remember
> >>>>> correctly) but the search for a concrete paper on the issue came
to
> >> nothing.
> >>>>>
> >>>>> Is anyone aware or has participated in such an effort?
> >>>>>
> >>>>> Thanks,
> >>>>> George
> >>>>>
> >>>>> --
> >>>>>
> >>>>> ---------------------------
> >>>>>
> >>>>> George Kousiouris
> >>>>> Electrical and Computer Engineer
> >>>>> Division of Communications,
> >>>>> Electronics and Information Engineering
> >>>>> School of Electrical and Computer Engineering
> >>>>> Tel: +30 210 772 2546
> >>>>> Mobile: +30 6939354121
> >>>>> Fax: +30 210 772 2569
> >>>>> Email: gkousiou@mail.ntua.gr
> >>>>> Site: http://users.ntua.gr/gkousiou/
> >>>>>
> >>>>> National Technical University of Athens
> >>>>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>> --
> >>>
> >>> ---------------------------
> >>>
> >>> George Kousiouris
> >>> Electrical and Computer Engineer
> >>> Division of Communications,
> >>> Electronics and Information Engineering
> >>> School of Electrical and Computer Engineering
> >>> Tel: +30 210 772 2546
> >>> Mobile: +30 6939354121
> >>> Fax: +30 210 772 2569
> >>> Email: gkousiou@mail.ntua.gr
> >>> Site: http://users.ntua.gr/gkousiou/
> >>>
> >>> National Technical University of Athens
> >>> 9 Heroon Polytechniou str., 157 73 Zografou, Athens, Greece
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message