Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0DB1817B04 for ; Mon, 2 Feb 2015 09:27:03 +0000 (UTC) Received: (qmail 98447 invoked by uid 500); 2 Feb 2015 09:26:56 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 98319 invoked by uid 500); 2 Feb 2015 09:26:56 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98293 invoked by uid 99); 2 Feb 2015 09:26:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 09:26:46 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of hadoop.support@visolve.com designates 173.11.74.250 as permitted sender) Received: from [173.11.74.250] (HELO mail.visolve.com) (173.11.74.250) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Feb 2015 09:26:21 +0000 Received: from ViMF (mail.visolve.com [173.11.74.250]) by mail.visolve.com (Postfix) with ESMTPA id 74E3C4A0C; Mon, 2 Feb 2015 01:25:55 -0800 (PST) Received: from abts-tn-static-154.230.165.122.airtelbroadband.in ([122.165.230.154] helo=visolvePC) by ViMF with ESMTP (ASSP 1.9); 2 Feb 2015 01:25:53 -0800 From: To: =?UTF-8?Q?'Harun_Re=C5=9Fit_Zafer'?= , References: <54C6065D.3010201@tubitak.gov.tr> In-Reply-To: Subject: RE: Multiple separate Hadoop clusters on same physical machines Date: Mon, 2 Feb 2015 14:55:51 +0530 Message-ID: <000601d03eca$3eacc3d0$bc064b70$@visolve.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0007_01D03EF8.587330A0" X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHcnec5gBYew4bpB/3+vW+Zj4MauwIVQxQHnLOs5oA= Content-Language: en-us X-Virus-Checked: Checked by ClamAV on apache.org This is a multipart message in MIME format. ------=_NextPart_000_0007_01D03EF8.587330A0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello Harun, =20 Your question is very interesting and will be useful for future Hadoop = setups for startup/individuals too. =20 Normally for testing purposes, we prefer you to use pseudo-distributed = environments (i.e. installation of all cluster files in single node). = You can refer few links which will guide you through the whole process = below for reference: =20 https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distribut= ed-environment =20 Individual Pseudo Distributed Cluster Implementation: =20 http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installatio= n/ http://hbase.apache.org/book.html#quickstart_pseudo and please check for others. =20 >From our 20 years of Server & its related Industrial experience, we = recommend you to use VM/Instances for production & Business Critical = environment. Other way around, if you are developing some products = related to Hadoop, you can use docker & other related resources for = development. As shipment to production will become stress free with the = use of these tools with cluster environment setup.=20 =20 Feel free to ask for further queries. =20 Thanks and Regards, S.RagavendraGanesh Hadoop Support Team ViSolve Inc.| www.visolve.com =20 =20 =20 From: Alexander Pivovarov [mailto:apivovarov@gmail.com]=20 Sent: Monday, February 02, 2015 12:56 PM To: user@hadoop.apache.org Subject: Re: Multiple separate Hadoop clusters on same physical machines =20 start several vms and install hadoop on each vm keywords: kvm, QEMU =20 On Mon, Jan 26, 2015 at 1:18 AM, Harun Re=C5=9Fit Zafer = > wrote: Hi everyone, We have set up and been playing with Hadoop 1.2.x and its friends = (Hbase, pig, hive etc.) on 7 physical servers. We want to test Hadoop = (maybe different versions) and ecosystem on physical machines = (virtualization is not an option) from different perspectives. As a bunch of developer we would like to work in parallel. We want every = team member play with his/her own cluster. However we have limited = amount of servers (strong machines though). So the question is, by changing port numbers, environment variables and = other configuration parameters, is it possible to setup several = independent clusters on same physical machines. Is there any = constraints? What are the possible difficulties we are to face? Thanks in advance --=20 Harun Re=C5=9Fit Zafer T=C3=9CB=C4=B0TAK B=C4=B0LGEM BTE Bulut Bili=C5=9Fim ve B=C3=BCy=C3=BCk Veri Analiz Sistemleri = B=C3=B6l=C3=BCm=C3=BC T +90 262 675 3268 =20 W http://www.hrzafer.com =20 ------=_NextPart_000_0007_01D03EF8.587330A0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello Harun,

 

Your question is very interesting and will be useful for future Hadoop = setups for startup/individuals too.

 

Normally for testing purposes, we prefer you to use pseudo-distributed = environments (i.e. installation of all cluster files in single node). = You can refer few links which will guide you through the whole process = below for reference:

 

https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distribu= ted-environment

 

Individual Pseudo Distributed Cluster = Implementation:

 

http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installati= on/

http://hbase.apache.org/book.html#quickstart_pseudo

and please check for others.

 

From our 20 years of Server & its related Industrial experience, we = recommend you to use VM/Instances for production & Business Critical = environment. Other way around, if you are developing some products = related to Hadoop, you can use docker & other related resources for = development. As shipment to production will become stress free with the = use of these tools with cluster environment setup. =

 

Feel free to ask for further queries.

 

Thanks and Regards,
S.RagavendraGanesh

Hadoop Support Team

ViSolve Inc.|www.visolve.com

 

 

 

From:<= /b> = Alexander Pivovarov [mailto:apivovarov@gmail.com]
Sent: = Monday, February 02, 2015 12:56 PM
To: = user@hadoop.apache.org
Subject: Re: Multiple separate Hadoop = clusters on same physical machines

 

start several vms and install hadoop on each = vm

keywords: kvm, = QEMU

 

On Mon, = Jan 26, 2015 at 1:18 AM, Harun Re=C5=9Fit Zafer <harun.zafer@tubitak.gov.tr> = wrote:

Hi = everyone,

We have set up and been playing with Hadoop 1.2.x and = its friends (Hbase, pig, hive etc.) on 7 physical servers. We want to = test Hadoop (maybe different versions) and ecosystem on physical = machines (virtualization is not an option) from different = perspectives.

As a bunch of developer we would like to work in = parallel. We want every team member play with his/her own cluster. = However we have limited amount of servers (strong machines = though).

So the question is, by changing port numbers, = environment variables and other configuration parameters, is it possible = to setup several independent clusters on same physical machines. Is = there any constraints? What are the possible difficulties we are to = face?

Thanks in advance

--
Harun Re=C5=9Fit Zafer
T=C3=9CB=C4=B0TAK B=C4=B0LGEM BTE
Bulut Bili=C5=9Fim ve B=C3=BCy=C3=BCk Veri Analiz = Sistemleri B=C3=B6l=C3=BCm=C3=BC
T = +90 262 675 3268
W  = http://www.hrzafer.com

<= /div>

 

------=_NextPart_000_0007_01D03EF8.587330A0--