Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 078BF17740 for ; Mon, 2 Feb 2015 20:03:47 +0000 (UTC) Received: (qmail 88581 invoked by uid 500); 2 Feb 2015 20:03:25 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 88472 invoked by uid 500); 2 Feb 2015 20:03:25 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 88458 invoked by uid 99); 2 Feb 2015 20:03:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 20:03:25 +0000 X-ASF-Spam-Status: No, hits=2.5 required=5.0 tests=FREEMAIL_REPLY,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of daemeonr@gmail.com designates 209.85.223.182 as permitted sender) Received: from [209.85.223.182] (HELO mail-ie0-f182.google.com) (209.85.223.182) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 20:03:00 +0000 Received: by mail-ie0-f182.google.com with SMTP id ar1so20063248iec.13 for ; Mon, 02 Feb 2015 12:01:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=hCbSHkvMmz7VaJMy3B6j8poBZxS7MtDBdcAoaEChajg=; b=rGL9KsDCNzHz3fkrj4tMaYgY0d0oEOXviCctnhXLGoogg+FtscFEB7wa6+VaxcaTK7 jokQ/TFkHEPC2rInXq4/V2NUFazGM7b8Em0xajwLgasyRrbMTuuPIWUSZRfRMGy/JTTZ UhgrQaxSCkpeoQtvPpsfzrvs4VOwm9dCGZCxlaL+9CwBJpUm+DM8fw+yonCnJL8v2knu 0PPeAJbOqbbNwluv0niRYmAfXiIoqGZReFfFN3CwkH24WFxMOKsp0OxXOFRGovUoIxU0 TuRRsily7OWBXJsb0d1rqWNOpirm+olTz5p7VGo3hrDrsWzHz045ftweeHxYO4R7MZn3 L9vg== X-Received: by 10.107.16.169 with SMTP id 41mr4854201ioq.33.1422907288053; Mon, 02 Feb 2015 12:01:28 -0800 (PST) MIME-Version: 1.0 Received: by 10.50.245.44 with HTTP; Mon, 2 Feb 2015 12:00:57 -0800 (PST) In-Reply-To: <001901d03ed8$b6452ff0$22cf8fd0$@visolve.com> References: <54C6065D.3010201@tubitak.gov.tr> <000601d03eca$3eacc3d0$bc064b70$@visolve.com> <815CE7F1-7928-4FD6-9D70-84C2079A127B@gmail.com> <001901d03ed8$b6452ff0$22cf8fd0$@visolve.com> From: daemeon reiydelle Date: Mon, 2 Feb 2015 12:00:57 -0800 Message-ID: Subject: Re: Multiple separate Hadoop clusters on same physical machines To: user@hadoop.apache.org Cc: Ashish Kumar9 , =?UTF-8?Q?Harun_Re=C5=A1it_Zafer?= Content-Type: multipart/alternative; boundary=001a113edac473ecd4050e206bde X-Virus-Checked: Checked by ClamAV on apache.org --001a113edac473ecd4050e206bde Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Fantastic! I was delighted to have recently worked for a large search engine company that has moved significant components of their hadoop to to docker containers on Ubunto, seeing amazing performance/density improvements. And yes, the build process is really picky. Thanks SO much! *.......* *=E2=80=9CLife should not be a journey to the grave with the intention of a= rriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D - Hunter ThompsonDaemeon C.= M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support wrote: > Hello Ashish, > > > > Alexander reference is great. Adding to that, you can also find latest > Hadoop docker image from below > > > > https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/ > > > > Set up is as simple as that > > 1. Install Docker > > 2. Build and Pull above Image > > 3. Start Container (if any errors, let us know) > > > > Hope it helps!!! > > > > Thanks and Regards, > S.RagavendraGanesh > Hadoop Support Team > ViSolve Inc.|www.visolve.com > > > > *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com] > *Sent:* Monday, February 02, 2015 3:07 PM > *To:* user@hadoop.apache.org > *Cc:* Harun Re=C5=A1it Zafer > > *Subject:* Re: Multiple separate Hadoop clusters on same physical machine= s > > > > > http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-do= cker/ > > > > Ambari based, but works quite well > > > > > > On 02 Feb 2015, at 10:33, Ashish Kumar9 wrote: > > > > Is there any good reference material available to follow to test docker > and hadoop integration . > > > > > From: > To: "'Harun Re=C5=9Fit Zafer'" , < > user@hadoop.apache.org> > Date: 02/02/2015 02:57 PM > Subject: RE: Multiple separate Hadoop clusters on same physical > machines > ------------------------------ > > > > > Hello Harun, > > Your question is very interesting and will be useful for future Hadoop > setups for startup/individuals too. > > Normally for testing purposes, we prefer you to use pseudo-distributed > environments (i.e. installation of all cluster files in single node). You > can refer few links which will guide you through the whole process below > for reference: > > > https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distribut= ed-environment > > Individual Pseudo Distributed Cluster Implementation: > > http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installatio= n/ > http://hbase.apache.org/book.html#quickstart_pseudo > and please check for others. > > From our 20 years of Server & its related Industrial experience, we > recommend you to use VM/Instances for production & Business Critical > environment. Other way around, if you are developing some products relate= d > to Hadoop, you can use docker & other related resources for development. = As > shipment to production will become stress free with the use of these tool= s > with cluster environment setup. > > Feel free to ask for further queries. > > Thanks and Regards, > S.RagavendraGanesh > Hadoop Support Team > ViSolve Inc.|www.visolve.com > > > > *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com > ] > *Sent:* Monday, February 02, 2015 12:56 PM > *To:* user@hadoop.apache.org > *Subject:* Re: Multiple separate Hadoop clusters on same physical machine= s > > start several vms and install hadoop on each vm > keywords: kvm, QEMU > > On Mon, Jan 26, 2015 at 1:18 AM, Harun Re=C5=9Fit Zafer < > harun.zafer@tubitak.gov.tr> wrote: > Hi everyone, > > We have set up and been playing with Hadoop 1.2.x and its friends (Hbase, > pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe > different versions) and ecosystem on physical machines (virtualization is > not an option) from different perspectives. > > As a bunch of developer we would like to work in parallel. We want every > team member play with his/her own cluster. However we have limited amount > of servers (strong machines though). > > So the question is, by changing port numbers, environment variables and > other configuration parameters, is it possible to setup several independe= nt > clusters on same physical machines. Is there any constraints? What are th= e > possible difficulties we are to face? > > Thanks in advance > > -- > Harun Re=C5=9Fit Zafer > T=C3=9CB=C4=B0TAK B=C4=B0LGEM BTE > Bulut Bili=C5=9Fim ve B=C3=BCy=C3=BCk Veri Analiz Sistemleri B=C3=B6l=C3= =BCm=C3=BC > T +90 262 675 3268 > W http://www.hrzafer.com > > > > --001a113edac473ecd4050e206bde Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Fantastic! I was delighted to have rec= ently worked for a large search engine company that has moved significant c= omponents of their hadoop to to docker containers on Ubunto, seeing amazing= performance/density improvements. And yes, the build process is really pic= ky. Thanks SO much!


.......
=E2=80=9CLife shou= ld not be a journey to the grave with the intention of arriving safely in a
pretty and well preserved body, but rather to skid in broadside in a cloud of smoke,
thoroughly used up, totally worn out, and loudly proclaiming =E2=80=9CWow! What a Ride!=E2=80=9D
- = Hunter Thompson

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
L= ondon (+44) (0) 20 8144 9872

On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Suppo= rt <hadoop.support@visolve.com> wrote:

Hello Ashish,=

=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0

Alexander reference is great.= Adding to that, you can also find latest Hadoop docker image from below=

=C2=A0

https:= //registry.hub.docker.com/u/sequenceiq/hadoop-docker/

=C2=A0

Set up is as simple as that

= 1.=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 Install Docke= r

2.=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 B= uild and Pull above Image

3.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <= u>Start Container (if any errors, let us know)=

=C2=A0

Hope it helps!!!

= =C2=A0

Thanks and Re= gards,
S.RagavendraGanesh
Hadoop Support Team
ViSolve Inc.|www.= visolve.com

=C2=A0

From: Alexander Alten-Lorenz [mail= to:wget.null@gmail= .com]
Sent: Monday, February 02, 2015 3:07 PM
To: = user@hadoop.apa= che.org
Cc: Harun Re=C5=A1it Zafer


Subject: Re: Multiple separate Hadoop clusters on same p= hysical machines

Ambari based, but works quite well

=C2=A0

<= u>=C2=A0

On 02 Feb 2015, at 10:33, Ashish Ku= mar9 <ashishk4@= in.ibm.com> wrote:

= =C2=A0

Is there any good reference= material available to follow to test docker and hadoop integration .




From: =C2=A0 =C2=A0 =C2=A0 =C2=A0<= span style=3D"font-size:7.5pt;font-family:"Arial",sans-serif"><= ;hadoop.sup= port@visolve.com>
To: =C2=A0 =C2=A0 =C2=A0 = =C2=A0"'Harun Re=C5=9Fit Zafer'" <harun.zafer@tubitak.gov.tr= >, <u= ser@hadoop.apache.org>
Date: =C2=A0 =C2=A0 = =C2=A0 =C2=A002/02/2015 02:57 PM
Subject: =C2= =A0 =C2=A0 =C2=A0 =C2=A0RE: Multiple separate Hadoop clusters on same = physical machines


<= br>
Hello Harun,
=C2=A0 =
Your question is very interesting and will be useful for= future Hadoop setups for startup/individuals too.
=C2=A0
Normally for testing purposes, we pref= er you to use pseudo-distributed environments (i.e. installation of all clu= ster files in single node). You can refer few links which will guide you th= rough the whole process below for reference:
=C2= =A0
ht= tps://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-e= nvironment
=C2=A0
I= ndividual Pseudo Distributed Cluster Implementation:
=C2=A0
http://= www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/=
http://hbase.apache.org/book.html#quickstart= _pseudo
and please check for others. <= br>=C2=A0
From our 20 years of Ser= ver & its related Industrial experience, we recommend you to use VM/Ins= tances for production & Business Critical environment. Other way around= , if you are developing some products related to Hadoop, you can use docker= & other related resources for development. As shipment to production w= ill become stress free with the use of these tools with cluster environment= setup.
=C2=A0
Feel fre= e to ask for further queries.
=C2=A0
<= span style=3D"font-size:10.0pt;font-family:"Calibri",sans-serif;c= olor:#004080">Thanks and Regards,
S.Ragav= endraGanesh
Hadoop Support Team
ViSolve Inc.|www.visolve.com
=C2=A0=
=C2=A0
=C2=A0 <= br>From: Alexander Pivovarov [mailto:apivovarov@gmail.com= ]
Sent: Monday, February 02, 2015 12:56 PM
To: user@hadoop.apache.or= g
Subject: Re: Multiple separate Hadoop clusters on same phys= ical machines
=C2=A0
start several vms and install hadoop on= each vm
keywords: kvm, QEMU
=C2=A0
On Mon, Jan 26, 2015 at 1:1= 8 AM, Harun Re=C5=9Fit Zafer <harun.zafer@tubitak.gov.tr> wrote:
Hi ever= yone,

We have set up and been playing with Hadoop 1.2.x and its frie= nds (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop (= maybe different versions) and ecosystem on physical machines (virtualizatio= n is not an option) from different perspectives.

As a bunch of devel= oper we would like to work in parallel. We want every team member play with= his/her own cluster. However we have limited amount of servers (strong mac= hines though).

So the question is, by changing port numbers, environ= ment variables and other configuration parameters, is it possible to setup = several independent clusters on same physical machines. Is there any constr= aints? What are the possible difficulties we are to face?

Thanks in = advance

--
Harun Re=C5=9Fit Zafer<= br>T=C3=9CB=C4=B0TAK B=C4=B0LGEM BTE
Bulut Bili=C5=9Fim ve B=C3=BCy=C3= =BCk Veri Analiz Sistemleri B=C3=B6l=C3=BCm=C3=BC
T +90 262 675 3268
W =C2=A0http://www.hrzafer.com
=C2=A0

=

=C2=A0


--001a113edac473ecd4050e206bde--