Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A2BE210FE5 for ; Wed, 16 Oct 2013 18:58:27 +0000 (UTC) Received: (qmail 69279 invoked by uid 500); 16 Oct 2013 18:58:21 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 68772 invoked by uid 500); 16 Oct 2013 18:58:17 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 68764 invoked by uid 99); 16 Oct 2013 18:58:16 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 18:58:16 +0000 X-ASF-Spam-Status: No, hits=4.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_BL_SPAMCOP_NET,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [106.10.151.209] (HELO nm32-vm2.bullet.mail.sg3.yahoo.com) (106.10.151.209) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 18:58:09 +0000 Received: from [106.10.166.117] by nm32.bullet.mail.sg3.yahoo.com with NNFMP; 16 Oct 2013 18:57:46 -0000 Received: from [106.10.151.254] by tm6.bullet.mail.sg3.yahoo.com with NNFMP; 16 Oct 2013 18:57:46 -0000 Received: from [127.0.0.1] by omp1003.mail.sg3.yahoo.com with NNFMP; 16 Oct 2013 18:57:46 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 305115.69081.bm@omp1003.mail.sg3.yahoo.com Received: (qmail 96240 invoked by uid 60001); 16 Oct 2013 18:57:46 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.co.in; s=s1024; t=1381949866; bh=llQUIDLfk4ji40csAUK0hTv/4FnT7ujjDmZUWPnlodY=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=v6mv15hvwejP4GuvRcQHDeRCr9+mls/81sHmaLI8GLkxWc8MCqzkBJWPNyqxO86EZ3m/J8cC/x90xvyMFMwUx+NEDrkafmrIaBLz4XfaxNLoKAsTYxMtFdt30Svmz7KdyOa1Y1oxpKYFnYjStQz9VvXf4+7zsOVx8AE6qr+qI8M= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.co.in; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type; b=uh8sIlKdXIRhpLws/Ota/9M/ocUE4OQBapbwscgPoG4DvE7NAoiwIagdCSfRaQwXKOMQaKKKrtv/oP5MxT5fpSCslAm4xGuDQ6nOYNPWQuirPdDvUd/LmmM8JFUB1Qi2Jr64+dbgos8nDlTTYd4DD+EFKUCuUpoLgSuUQ+B+JqY=; X-YMail-OSG: sB1XBVkVM1nzATltYqa5oD23Ok_Jp5paG7dvdxtC6KeF4qm 3H8RmNTuN5mmvHoD6aBn3vgw8X4w_BdtXT.4y2plXPudHQ7HKLIoLMBkuPwn ggMTV9FtrncLuUcV2Wvz_XCDTHcjPI9oAeea8aQMmhzDOWlQVq4d3qGZtaKU E0lAW1bxCzWi0XbiOQ3CANisP3GFqi7Q18yxlTRSuyZdCKbZgJUr1UGxMpF5 YklECIcdUSPKCvWPvtad8PPmU2e4k2FSjYoJOqpU3Jg2.G9wPt3prXLbdUyQ 1w5C3fWgpmEwai.3rJZiHMWR4z0Sc5MtYOIGuCHsDxYsbsNCXhM.jQrWphU4 N_ofRkGx1gm55X52.E2HWTJ_gdCNKJ1HqyLJKAp93UyzV2Y8EajZIxTGTijo ITV_fTVKqBtYeqeOJDSBcWCTneFpjMYazp1HIhBQG6RlnXOgGS.CIZeRRcpQ etD36rCiBUxSH6u3nLYvnT_2RJJ5IlMaq0tMFlBbD1OuC8iMtjSnl3r.UTZP soBJIgTsabmZNdHMH.b85Z1DmlUdRGZX_C6musL4rFG0aaEIQOps4rwgLBEy Gdp6dX_raD8LvMJyLKXT4pkv3uC.66wf2Lnmp2yT2O.RlZZoRWX3x9_11dyV Bu6JnEjElQ.mBxMxY8OE2TDDE5SdsXaBIwhRvLWQzTL07dtTrKAdzYO1J5Rs pEklWYT1xctpXYwgE1BEy7gN9wC7aVYJ5xZhdq4IR4u6.N35mpOkkIvRb1g. 7.IfV8x6nErowOaMRTKAW74IrcJ_iRWsPxW0qJ.Sy_F4T6FhQ6t4q1ohpxz9 FFZiyZpXv9VbgE8m9exsQkb_ovKGYUG4WCNtZ236VRsXYZh685rweoIl9A1. CbTmMCjDmTggAb7uklgBdFGSeI2POjch7LS5gJDBJobly2GHVweaubGHD.Q0 GZ5wAfMuJRSE0DbwX7RG4Csik6jLvz426fc24LmHZbCKXz2SwnB126vLXUP6 MCcNedc3aOsq0bMVgw_3gbxZvR0Z5V0Zo2V0TGdJ8RVKYeFJbi5B7qjGJmrO fkujRbUtGsBhsjSeWxTMuVySdUej4vLxNwtr0_GvtFTIuX10eEdsU Received: from [69.191.241.59] by web190101.mail.sg3.yahoo.com via HTTP; Thu, 17 Oct 2013 02:57:45 SGT X-Rocket-MIMEInfo: 002.001,VGhhbmtzIGZvciBzaGFyaW5nIHRoZSBleHBlcmllbmNlIEFsZXguIEkga2luZCBvZiBhbnRpY2lwYXRlZCB0aGUga2luZCBvZiBpc3N1ZXMgeW91IG1lbnRpb25lZCBoZXJlIGJ1dCBqdXN0IHdhbnRlZCB0byBtYWtlIHN1cmUgSSBleHBsb3JlIGFsbCBwb3NzaWJsZSBvcHRpb25zCsKgClJlZ2FyZHMsCkRoYXZhbAoKCgpPbiBXZWRuZXNkYXksIDE2IE9jdG9iZXIgMjAxMyAxOjM0IFBNLCBhbGV4IGJvaHIgPGFsZXhqYm9ockBnbWFpbC5jb20.IHdyb3RlOgogCkhpIERoYXZhbCwKU29ycnkganVzdCBzYXcgdGgBMAEBAQE- X-Mailer: YahooMailWebService/0.8.160.587 References: <1375984416.41725.YahooMailNeo@web190104.mail.sg3.yahoo.com> <1375997716.81432.YahooMailNeo@web190102.mail.sg3.yahoo.com> <1377110211.49599.YahooMailNeo@web190106.mail.sg3.yahoo.com> Message-ID: <1381949865.95419.YahooMailNeo@web190101.mail.sg3.yahoo.com> Date: Thu, 17 Oct 2013 02:57:45 +0800 (SGT) From: Dhaval Shah Reply-To: Dhaval Shah Subject: Re: Hosting Hadoop To: "user@hadoop.apache.org" In-Reply-To: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="2138131439-1232228761-1381949865=:95419" X-Virus-Checked: Checked by ClamAV on apache.org --2138131439-1232228761-1381949865=:95419 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable Thanks for sharing the experience Alex. I kind of anticipated the kind of i= ssues you mentioned here but just wanted to make sure I explore all possibl= e options=0A=A0=0ARegards,=0ADhaval=0A=0A=0A=0AOn Wednesday, 16 October 201= 3 1:34 PM, alex bohr wrote:=0A =0AHi Dhaval,=0ASorry = just saw this email (oops) so might not be relevant - but:=0AWe didn't enco= unter too much Funky issues that we were worried about regarding random res= ource constraints or random outages that might happen when sharing a physic= al box with unknown neighbors.=0A=0ABut overall we feel the virtualization = is robbing us of significant CPU, and more importantly they don't have idea= l instance types. =A0The M1.xlarges are too small storage wise (we ended up= paying for more CPU than we needed to get the amount of storage we needed)= and the hs1.8xlarge are too big - they have 24 drives and it feels like we= lose a good amount of CPU controlling IO across all those drives, and we n= ow have significantly more storage than we need in order to get enough CPU = to keep our SLAs.=0A=0AFor initial set-up - AWS is way quicker than owning = hardware. =A0But if you already have hardware, moving to AWS I think will i= ncrease your monthly bills to get comparable performance.=0A=0A=0A=0AOn Wed= , Aug 21, 2013 at 11:36 AM, Dhaval Shah wrote= :=0A=0AAlex, did you run into funky issues with EC2/EMR? The kind of issues= that would come up because its a virtualized environment? We currently own= our hardware and are just trying to do an ROI analysis on whether moving t= o Amazon can reduce our admin costs. Currently administering a Hadoop clust= er is a bit expensive (in terms of man hours spent trying to replace disks = and so on) and we are exploring whether its possible to avoid some of those= costs=0A>=A0=0A>Regards,=0A>Dhaval=0A>=0A>=0A>=0A>________________________= ________=0A> From: alex bohr =0A>To: user@hadoop.apach= e.org =0A>Cc: Dhaval Shah =0A>Sent: Monday, 1= 2 August 2013 1:41 PM=0A>Subject: Re: Hosting Hadoop=0A> =0A>=0A>=0A>I've h= ad good experience running a large hadoop cluster on EC2 instances. =A0Afte= r almost 1 year we haven't had any significant down time, just lost a small= # of data nodes. =A0=0A>I don't think EMR is an ideal solution if your clu= ster will be running 24/7.=0A>=0A>=0A>But for running a large cluster, I do= n't see how you it's more cost efficient to run in the cloud than to own th= e hardware and we're trying to move off the cloud onto our own hardware. = =A0Can I ask why you're looking to move to the cloud?=0A>=0A>=0A>=0A>On Fri= , Aug 9, 2013 at 10:42 AM, Nitin Pawar wrote:=0A>= =0A>check altiscale as well=0A>>=0A>>=0A>>=0A>>On Fri, Aug 9, 2013 at 3:05 = AM, Dhaval Shah wrote:=0A>>=0A>>Thanks for th= e list Marcos. I will go through the slides/links. I think that's helpful= =0A>>>=A0=0A>>>Regards,=0A>>>Dhaval=0A>>>=0A>>>=0A>>>=0A>>>________________= ________________=0A>>> From: Marcos Luis Ortiz Valmaseda =0A>>>To: Dhaval Shah =0A>>>Cc: user@= hadoop.apache.org =0A>>>Sent: Thursday, 8 August 2013 4:50 PM=0A>>>Subject:= Re: Hosting Hadoop=0A>>> =0A>>>=0A>>>Well, all depends, because many compa= nies use Cloud Computing=0A>>>platforms like Amazon EMR. Vmware, Rackscpace= Cloud for Hadoop=0A>>>hosting:=0A>>>http://aws.amazon.com/elasticmapreduce= =0A>>>http://www.vmware.com/company/news/releases/vmw-mapr-hadoop-062013.ht= ml=0A>>>http://bitrefinery.com/services/hadoop-hosting=0A>>>http://www.joye= nt.com/products/compute-service/features/hadoop=0A>>>=0A>>>There a lot of c= ompanies using HBase hosted in Cloud. The last=0A>>>HBaseCon was full of gr= eat use-cases:=0A>>>HBase at=0A Pinterest:=0A>>>http://www.hbasecon.com/ses= sions/apache-hbase-operations-at-pinterest/=0A>>>=0A>>>HBase at Groupon=0A>= >>http://www.hbasecon.com/sessions/apache-hbase-at-groupon/=0A>>>=0A>>>A gr= eat talk by Benoit for Networking design for HBase:=0A>>>http://www.hbaseco= n.com/sessions/scalable-network-designs-for-apache-hbase/=0A>>>=0A>>>Using = Coprocessors to Index Columns in an Elasticsearch Cluster=0A>>>http://www.h= basecon.com/sessions/using-coprocessors-to-index-columns/=0A>>>=0A>>>2013/8= /8, Dhaval Shah :=0A>>>> We are exploring the = possibility of hosting Hadoop outside of our data=0A>>>> centers. I am awar= e that Hadoop in general isn't exactly designed to run on=0A>>>> virtual ha= rdware. So a few questions:=0A>>>> 1. Are there any providers out there who= would host Hadoop on dedicated=0A>>>> physical hardware?=0A>>>> 2. Has any= one had success hosting Hadoop on virtualized hardware where 100%=0A>>>> up= time and performance/stability are very important (we use HBase as a real= =0A>>>> time database and it needs to be up all the time)?=0A>>>>=0A>>>> Th= anks,=0A>>>> Dhaval=0A>>>=0A>>>=0A>>>-- =0A>>>Marcos Ortiz Valmaseda=0A>>>P= roduct Manager at PDVSA=0A>>>http://about.me/marcosortiz=0A>>>=0A>>>=0A>>>= =0A>>=0A>>=0A>>=0A>>-- =0A>>Nitin Pawar=0A>>=0A>=0A>=0A> --2138131439-1232228761-1381949865=:95419 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for= sharing the experience Alex. I kind of anticipated the kind of issues you = mentioned here but just wanted to make sure I explore all possible options<= /span>
 
Regards,
Dhaval
<= div class=3D"yahoo_quoted" style=3D"display: block;">

On= Wednesday, 16 October 2013 1:34 PM, alex bohr <alexjbohr@gmail.com> = wrote:
Hi Dhaval,
Sorry just saw this email (oo= ps) so might not be relevant - but:
We didn't encounter too much = Funky issues that we were worried about regarding random resource constraints or random= outages that might happen when sharing a physical box with unknown neighbo= rs.
=0A

But overall we feel the virt= ualization is robbing us of significant CPU, and more importantly they don'= t have ideal instance types.  The M1.xlarges are too small storage wis= e (we ended up paying for more CPU than we needed to get the amount of stor= age we needed) and the hs1.8xlarge are too big - they have 24 drives and it= feels like we lose a good amount of CPU controlling IO across all those dr= ives, and we now have significantly more storage than we need in order to g= et enough CPU to keep our SLAs.
=0A

= For initial set-up - AWS is way quicker than owning hardware.  But if = you already have hardware, moving to AWS I think will increase your monthly= bills to get comparable performance.
=0A

On Wed, Aug 21, 2013 at 11:36 AM, Dhaval Shah <prince_mithibai@yahoo.co.in> wrote:
=0A
Alex, = did you run into funky issues with EC2/EMR? The kind of issues that would c= ome up because its a virtualized environment? We currently own our hardware= and are just trying to do an ROI analysis on whether moving to Amazon can = reduce our admin costs. Currently administering a Hadoop cluster is a bit e= xpensive (in terms of man hours spent trying to replace disks and so on) an= d we are exploring whether its possible to avoid some of those costs
= =0A
 
Regards,
Dhaval<= /div>

=0A
=0AI don't think EMR is an ideal solu= tion if your cluster will be running 24/7.
=0A

=
But for running a large cluster, I don't see how you it's more c= ost efficient to run in the cloud than to own the hardware and we're trying= to move off the cloud onto our own hardware.  Can I ask why you're lo= oking to move to the cloud?
=0A=0A


On Fri, Aug 9, 2013 at 10:42 AM, Nitin Pawar <niti= npawar432@gmail.com> wrote:
=0A
check altiscale as well


=0AOn Fri, Aug 9, 2013 at 3:05 AM, Dhaval Shah <= span dir=3D"ltr"><prince_mithibai@yahoo.co.in> wrote:
=0A
Thanks for the list M= arcos. I will go through the slides/links. I think that's helpful=0A=0A=0A
 
Regards,
= Dhaval

=0A
From: Marcos Luis Ortiz Valmaseda <marcosluis2186@gmail.com>
=0A=0A=0A To:<= /span> Dhaval Shah <prince_mithibai@yahoo.co.in>
Cc: user@hadoop.apache.org
=0A=0A=0A Sent: T= hursday, 8 August 2013 4:50 PM
Subject: Re: Hosting Hadoop

Well, all depends, becaus= e many companies use Cloud Computing
=0A=0A=0Aplatforms l= ike Amazon EMR. Vmware, Rackscpace Cloud for Hadoop
hosti= ng:
http://aws.amazon.com/elast= icmapreduce
http://www.vmware.com/company/news/releases/vmw-mapr-hado= op-062013.html
=0A=0A=0Ahttp://bitrefinery.com/services/hadoop-hosting
<= a rel=3D"nofollow" shape=3D"rect" target=3D"_blank" href=3D"http://www.joye= nt.com/products/compute-service/features/hadoop">http://www.joyent.com/prod= ucts/compute-service/features/hadoop
=0A=0A=0A
There a lot of companies using HBase hosted in Cloud. The last<= br clear=3D"none">HBaseCon was full of great use-cases:
H= Base at=0A Pinterest:
http://www.hbasecon.com/sessions/apache-hbase-operat= ions-at-pinterest/

HBase at Groupo= n
=0Ahttp://= www.hbasecon.com/sessions/apache-hbase-at-groupon/
= =0A=0A
A great talk by Benoit for Networking design for H= Base:
http://www.hbasecon.com/sessions/scalable-network-designs-for-a= pache-hbase/
=0A=0A=0A
Using Coproc= essors to Index Columns in an Elasticsearch Cluster
http://www.hbasecon.com= /sessions/using-coprocessors-to-index-columns/
=0A=0A= =0A
2013/8/8, Dhaval Shah <prince_mithibai@yahoo.co.in= >:
> We are exploring the possibility of hosting Ha= doop outside of our data
=0A> centers. I am aware that= Hadoop in general isn't exactly designed to run on
=0A= =0A> virtual hardware. So a few questions:
> 1. Are= there any providers out there who would host Hadoop on dedicated
> physical hardware?
> 2. Has anyone had = success hosting Hadoop on virtualized hardware where 100%
=0A=0A=0A> uptime and performance/stability are very important (we use = HBase as a real
> time database and it needs to be up = all the time)?
>
> Thanks,
> Dhaval


--
Marcos Ortiz Valmaseda
=0A=0A= =0AProduct Manager at PDVSA
http://about.m= e/marcosortiz


<= /div>


=0A

=0A
=0A--
Nitin Pawar
=0A
=0A

=


=



--2138131439-1232228761-1381949865=:95419--