hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ellis Miller <outlaw...@gmail.com>
Subject Re: Hadoop Development on cloud in a secure and economical way.
Date Wed, 22 May 2013 06:33:13 GMT
Configure private cloud: install VMWare / VirtualBox / KVM on internal
server / cluster and levearge either Cloudera Hadoop (free version) or
Hortonworks (Yahoo introduced Hortonworks and where Cloudera is exceptional
but proprietary  Hortonworks requires some configuration and tuning of
Hadoop in their promise to keep their Hadoop offering entirely open

Both have Hadoop appliances which can be imported into VMWare / VirtualBox.
Couple this with, perhaps, using Amazon Web Services for production hosting
and all development can be done in-house with Eclipse and appropriate
Source Control Management.

With Amazon don't have to worry so much about your proprietary code being
stolen as the biggest concern (depending on your HA requirements) have been
several high-profile outages where their servers in VA (when you subscribe
to Amazon Web Services and create a Hadoop cluster, for example, it asks
you to specify the geographical region).

Nice thing about using Virtualization in-house for development, test,
integration environments your developers can code and perform initial
testing in Dev using VMWare / VirtualBox to then take a snapshot and export
the Dev environment then import into production. VWare and VirtualBox both
provide proprietary means to accelerate this process streamlining an Agile
/ Extreme Programming software development framework while simplifying the
release cycle.

Personally, if I were going to do it on the cheap and keep even production
in-house would leverage x86 Linux servers (running Ubuntu, Fedora, Centos,
or even OpenSuSE) which natively support KVM (linux based Virtualization
which is free) to host Hortonworks (Hadoop) leveraging the Unix
Administrators and Network Engineering monkey's to securing the Production
server which would have to be configured to accept public / external
connections (assuming you are hosting an application which is Software as a
Service for your customers) while not compromising security.

Have worked at 2 firms in the past several years where we did the very same
and it worked exceptionally well just takes time to get your IT staff up to
speed on Virtualization, Cluster administration and synchronizing Virtual
Machine backups with the promotion of releases from Dev to Test to

On Tue, May 21, 2013 at 11:11 PM, Sai Sai <saigraph@yahoo.in> wrote:

> Is it possible to do Hadoop development on cloud in a secure and
> economical way without worrying about our source being taken away. We
> would like to have Hadoop and eclipse installed on a vm in cloud and our
> developers will log into the cloud on a daily basis and work on the cloud.
> Like this we r hoping if we develop any product we will minimize our source
> being taken away by the devs or others and is secure. Please let me know,
> any  suggestions  u may have.
> Thanks,
> Sai

Ellis R. Miller


Mundo Nulla Fides


View raw message