hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From daemeon reiydelle <daeme...@gmail.com>
Subject Re: Multiple separate Hadoop clusters on same physical machines
Date Mon, 02 Feb 2015 20:00:57 GMT
Fantastic! I was delighted to have recently worked for a large search
engine company that has moved significant components of their hadoop to to
docker containers on Ubunto, seeing amazing performance/density
improvements. And yes, the build process is really picky. Thanks SO much!



*.......*






*“Life should not be a journey to the grave with the intention of arriving
safely in apretty and well preserved body, but rather to skid in broadside
in a cloud of smoke,thoroughly used up, totally worn out, and loudly
proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA
(+1) 415.501.0198London (+44) (0) 20 8144 9872*

On Mon, Feb 2, 2015 at 3:09 AM, Hadoop Support <hadoop.support@visolve.com>
wrote:

> Hello Ashish,
>
>
>
> Alexander reference is great. Adding to that, you can also find latest
> Hadoop docker image from below
>
>
>
> https://registry.hub.docker.com/u/sequenceiq/hadoop-docker/
>
>
>
> Set up is as simple as that
>
> 1.       Install Docker
>
> 2.       Build and Pull above Image
>
> 3.       Start Container (if any errors, let us know)
>
>
>
> Hope it helps!!!
>
>
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Alten-Lorenz [mailto:wget.null@gmail.com]
> *Sent:* Monday, February 02, 2015 3:07 PM
> *To:* user@hadoop.apache.org
> *Cc:* Harun Rešit Zafer
>
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
>
>
>
> http://blog.sequenceiq.com/blog/2014/06/19/multinode-hadoop-cluster-on-docker/
>
>
>
> Ambari based, but works quite well
>
>
>
>
>
> On 02 Feb 2015, at 10:33, Ashish Kumar9 <ashishk4@in.ibm.com> wrote:
>
>
>
> Is there any good reference material available to follow to test docker
> and hadoop integration .
>
>
>
>
> From:        <hadoop.support@visolve.com>
> To:        "'Harun Reşit Zafer'" <harun.zafer@tubitak.gov.tr>, <
> user@hadoop.apache.org>
> Date:        02/02/2015 02:57 PM
> Subject:        RE: Multiple separate Hadoop clusters on same physical
> machines
> ------------------------------
>
>
>
>
> Hello Harun,
>
> Your question is very interesting and will be useful for future Hadoop
> setups for startup/individuals too.
>
> Normally for testing purposes, we prefer you to use pseudo-distributed
> environments (i.e. installation of all cluster files in single node). You
> can refer few links which will guide you through the whole process below
> for reference:
>
>
> https://districtdatalabs.silvrback.com/creating-a-hadoop-pseudo-distributed-environment
>
> Individual Pseudo Distributed Cluster Implementation:
>
> http://www.thegeekstuff.com/2012/02/hadoop-pseudo-distributed-installation/
> http://hbase.apache.org/book.html#quickstart_pseudo
> and please check for others.
>
> From our 20 years of Server & its related Industrial experience, we
> recommend you to use VM/Instances for production & Business Critical
> environment. Other way around, if you are developing some products related
> to Hadoop, you can use docker & other related resources for development. As
> shipment to production will become stress free with the use of these tools
> with cluster environment setup.
>
> Feel free to ask for further queries.
>
> Thanks and Regards,
> S.RagavendraGanesh
> Hadoop Support Team
> ViSolve Inc.|www.visolve.com
>
>
>
> *From:* Alexander Pivovarov [mailto:apivovarov@gmail.com
> <apivovarov@gmail.com>]
> *Sent:* Monday, February 02, 2015 12:56 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Multiple separate Hadoop clusters on same physical machines
>
> start several vms and install hadoop on each vm
> keywords: kvm, QEMU
>
> On Mon, Jan 26, 2015 at 1:18 AM, Harun Reşit Zafer <
> harun.zafer@tubitak.gov.tr> wrote:
> Hi everyone,
>
> We have set up and been playing with Hadoop 1.2.x and its friends (Hbase,
> pig, hive etc.) on 7 physical servers. We want to test Hadoop (maybe
> different versions) and ecosystem on physical machines (virtualization is
> not an option) from different perspectives.
>
> As a bunch of developer we would like to work in parallel. We want every
> team member play with his/her own cluster. However we have limited amount
> of servers (strong machines though).
>
> So the question is, by changing port numbers, environment variables and
> other configuration parameters, is it possible to setup several independent
> clusters on same physical machines. Is there any constraints? What are the
> possible difficulties we are to face?
>
> Thanks in advance
>
> --
> Harun Reşit Zafer
> TÜBİTAK BİLGEM BTE
> Bulut Bilişim ve Büyük Veri Analiz Sistemleri Bölümü
> T +90 262 675 3268
> W  http://www.hrzafer.com
>
>
>
>

Mime
View raw message