hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wadood.Chaudh...@Instinet.com
Subject RE: Hadoop Multi-node cluster on Windows
Date Thu, 05 Jun 2014 07:03:58 GMT
Thanks. I will try that. By security do you mean firewalls?








----- Message from "Nitin Pawar" <nitinpawar432@gmail.com> on 06/05/2014 02:45:12 AM
-----
To: user@hadoop.apache.org
Subject: Re: Hadoop Multi-node cluster on Windows

if you have installed a single node setup on windows then it should be easy
to move to multinode.

Never tried this myself on windows but this is what I do on linux and it
should work on windows as well.

In the current node, instead of having anything as localhost or 127.0.0.1
.. point it to real hostname or static ip address
make an image out of it and launch similar instances
Launch new instances using that image.
disable windows security on all nodes
make sure all nodes can communicate each  other over network
After that login to each node and start services there and it should come
up as distributed node.


On Thu, Jun 5, 2014 at 11:59 AM, <Wadood.Chaudhary@instinet.com> wrote:

> Thanks for looking into it. I have used Horton but that is a third party
> tool and should not be required when Apache has started supporting Windows.
> Here is the Wiki article on Windows from Apache's.
> https://wiki.apache.org/hadoop/Hadoop2OnWindows
> I am able to setup in Pseudo distributed mode using the instructions
> above. Unfortunately, the article ends there saying that instructions for
> Multi-cluster will be added later. No mention that you cannot do it under
> Windows or you must use Horton.
>
> Wadood
>
>
> [image: Inactive hide details for Nitin Pawar ---06/05/2014 02:09:29
> AM---Currently as per my knowledge only Hortonwork's distribution]Nitin
> Pawar ---06/05/2014 02:09:29 AM---Currently as per my knowledge only
> Hortonwork's distribution comes for windows.
>
> From: Nitin Pawar <nitinpawar432@gmail.com>
> To: user@hadoop.apache.org,
> Date: 06/05/2014 02:09 AM
>
> Subject: Re: Hadoop Multi-node cluster on Windows
> ------------------------------
>
>
>
> Currently as per my knowledge only Hortonwork's distribution comes for
> windows.
>
> As such hadoop does not need ssh to work but ssh in linux was needed for
> the automated deployments.
>
> If you want to avoid that then, I would say setup a single machine with
> all installation where all packages and correct configurations are present.
> then make an image out of it and build other nodes.
>
> But then you will need to login to all nodes and start all services
> manually.
>
> Hortonwork's HDP 2.1 on windows step by step manual install guide is here
>
> *http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-Win-latest/bk_releasenotes_HDP-Win/content/index.html*
> <http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-Win-latest/bk_releasenotes_HDP-Win/content/index.html>
>
>
> On Thu, Jun 5, 2014 at 11:29 AM, <*Wadood.Chaudhary@instinet.com*
> <Wadood.Chaudhary@instinet.com>> wrote:
>
>    Thanks but that is almost a 5 year old tutorial, written when Hadoop
>    was not natively supported on Windows. As of Hadoop 2.x, Windows is
>    supported and setup is much simpler. No Cygwin required.
>
>    [image: Inactive hide details for Ted Yu ---06/04/2014 10:05:15
>    PM---See also http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%]Ted
>    Yu ---06/04/2014 10:05:15 PM---See also
>    *http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html*
>    <http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html>
>
>    From: Ted Yu <*yuzhihong@gmail.com* <yuzhihong@gmail.com>>
>    To: "*common-user@hadoop.apache.org* <common-user@hadoop.apache.org>" <
>    *user@hadoop.apache.org* <user@hadoop.apache.org>>,
>    Date: 06/04/2014 10:05 PM
>
>
>    Subject: Re: Hadoop Multi-node cluster on Windows
>    ------------------------------
>
>
>
>
>    See also
>    *http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html*
>    <http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html>
>
>
>    On Wed, Jun 4, 2014 at 6:43 PM, <*Wadood.Chaudhary@instinet.com*
>    <Wadood.Chaudhary@instinet.com>> wrote:
>       Thanks. Linux cluster requires SSH etc. for every thing including
>       single node. In Windows until now I have been able to setup to pseudo
>       distributed level without SSH or CygWin. Someone was saying that some
>       Windows service needs to be installed for communication across machines.
>       Is that true?
>
>
>
>
>
>
>
>
>       ----- Message from "Olivier Renault" on 06/04/2014 09:36:17 PM
>       -----
>    *To:*
>    *user@hadoop.apache.org* <user@hadoop.apache.org>
>    *Subject:*
>    RE: Hadoop Multi-node cluster on Windows
>    HDP run natively on Linux and Windows. It's the same code base. You
>       can install a windows cluster in the same way as you do for Linux.
>
>       Olivier
>
>       On 4 Jun 2014 18:23, <*Wadood.Chaudhary@instinet.com*
>       <Wadood.Chaudhary@instinet.com>> wrote:
>
>          Yes, but Horton runs Hadoop under Linux Virtual Machine. I am
>          talking of pure and simple Hadoop.
>
>
>
>
>
>
>
>
>          ----- Message from "Ted Yu" on 06/04/2014 09:18:51 PM -----
>
>
>
>    *To:*
>
>
>    "*common-user@hadoop.apache.org* <common-user@hadoop.apache.org>"
>
>
>    *Subject:*
>
>
>    Re: Hadoop Multi-node cluster on Windows
>
>
>
>    Have you seen this ?
>
>
>    *http://hortonworks.com/blog/install-hadoop-windows-hortonworks-data-platform-2-0/*
>    <http://hortonworks.com/blog/install-hadoop-windows-hortonworks-data-platform-2-0/>
>
>    Cheers
>
>
>    On Wed, Jun 4, 2014 at 5:47 PM, <*Wadood.Chaudhary@instinet.com*
>    <Wadood.Chaudhary@instinet.com>> wrote:
>       Has anyone successfully implemented Multi-node cluster under
>       Windows using the latest version of Hadoop. We were able to run Pseudo
>       distributed. Does it still needs SSH under Windows, or something installed
>       as service.
>
>       [image: Inactive hide details for ch huang ---06/04/2014 08:40:38
>       PM---hi,mailist: my company signed a new IDC , i need m]ch huang
>       ---06/04/2014 08:40:38 PM---hi,mailist:               my company signed a
>       new IDC , i need move the total 50T data
>
>       From: ch huang <*justlooks@gmail.com* <justlooks@gmail.com>>
>       To: *user@hadoop.apache.org* <user@hadoop.apache.org>,
>       Date: 06/04/2014 08:40 PM
>       Subject: issue about move data between two hdoop cluster
>       ------------------------------
>
>
>
>       hi,mailist:
>                     my company signed a new IDC , i need move the total
>       50T data from old hadoop cluster to the new cluster in new location ,how to
>       do it?
>
>
> *
>       =========================================================================================================
>       *
>
>       *<<<< Disclaimer >>>>*
>
>       *This message is intended solely for use by the named addressee(s).
>       If you receive this transmission in error, please immediately notify the
>       sender and destroy this message in its entirety, whether in electronic or
>       hard copy format. Any unauthorized use (and reliance thereon), copying,
>       disclosure, retention, or distribution of this transmission or the material
>       in this transmission is forbidden. We reserve the right to monitor and
>       archive electronic communications. This material does not constitute an
>       offer or solicitation with respect to the purchase or sale of any security.
>       It should not be construed to contain any recommendation regarding any
>       security or strategy. Any views expressed are those of the individual
>       sender, except where the message states otherwise and the sender is
>       authorized to state them to be the views of any such entity. This
>       communication is provided on an “as is” basis. It contains material that is
>       owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>       and may not, in whole or in part, be (i) copied, photocopied or duplicated
>       in any form, by any means, or (ii) redistributed, posted, published,
>       excerpted, or quoted without Instinet Incorporated's prior written consent.
>       Please access the following link for important information and
>       instructions: *
>       *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>       <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>
>       *Securities products and services are provided by locally
>       registered brokerage subsidiaries of Instinet Incorporated: Instinet
>       Australia Pty Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the
>       Australian Securities & Investments Commission; Instinet Canada Limited,
>       member IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by
>       the Securities and Futures Commission of Hong Kong; Instinet Singapore
>       Services Private Limited, regulated by the Monetary Authority of Singapore,
>       trading member of The Singapore Exchange Securities Trading Private Limited
>       and clearing member of The Central Depository (Pte) Limited; and Instinet,
>       LLC, member SIPC. *
>
>
>
> *
>       =========================================================================================================
>       *
>
>
>
>
>
>
> *
>    =========================================================================================================
>    *
>
>    *<<<< Disclaimer >>>>*
>
>    *This message is intended solely for use by the named addressee(s). If
>    you receive this transmission in error, please immediately notify the
>    sender and destroy this message in its entirety, whether in electronic or
>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>    disclosure, retention, or distribution of this transmission or the material
>    in this transmission is forbidden. We reserve the right to monitor and
>    archive electronic communications. This material does not constitute an
>    offer or solicitation with respect to the purchase or sale of any security.
>    It should not be construed to contain any recommendation regarding any
>    security or strategy. Any views expressed are those of the individual
>    sender, except where the message states otherwise and the sender is
>    authorized to state them to be the views of any such entity. This
>    communication is provided on an “as is” basis. It contains material that is
>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>    in any form, by any means, or (ii) redistributed, posted, published,
>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>    Please access the following link for important information and
>    instructions: *
>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>
>    *Securities products and services are provided by locally registered
>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>    Securities & Investments Commission; Instinet Canada Limited, member
>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>    member of The Singapore Exchange Securities Trading Private Limited and
>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>    member SIPC. *
>
>
>
> *
>    =========================================================================================================
>    *
>
>
>    CONFIDENTIALITY NOTICE
>    NOTICE: This message is intended for the use of the individual or
>    entity to which it is addressed and may contain information that is
>    confidential, privileged and exempt from disclosure under applicable law.
>    If the reader of this message is not the intended recipient, you are hereby
>    notified that any printing, copying, dissemination, distribution,
>    disclosure or forwarding of this communication is strictly prohibited. If
>    you have received this communication in error, please contact the sender
>    immediately and delete it from your system. Thank You.
>
>
> *
>    =========================================================================================================
>    *
>
>    *<<<< Disclaimer >>>>*
>
>    *This message is intended solely for use by the named addressee(s). If
>    you receive this transmission in error, please immediately notify the
>    sender and destroy this message in its entirety, whether in electronic or
>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>    disclosure, retention, or distribution of this transmission or the material
>    in this transmission is forbidden. We reserve the right to monitor and
>    archive electronic communications. This material does not constitute an
>    offer or solicitation with respect to the purchase or sale of any security.
>    It should not be construed to contain any recommendation regarding any
>    security or strategy. Any views expressed are those of the individual
>    sender, except where the message states otherwise and the sender is
>    authorized to state them to be the views of any such entity. This
>    communication is provided on an “as is” basis. It contains material that is
>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>    in any form, by any means, or (ii) redistributed, posted, published,
>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>    Please access the following link for important information and
>    instructions: *
>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>
>    *Securities products and services are provided by locally registered
>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>    Securities & Investments Commission; Instinet Canada Limited, member
>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>    member of The Singapore Exchange Securities Trading Private Limited and
>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>    member SIPC. *
>
>
>
> *
>    =========================================================================================================
>    *
>
>
>
>
>
> *
>    =========================================================================================================
>    *
>
>    *<<<< Disclaimer >>>>*
>
>    *This message is intended solely for use by the named addressee(s). If
>    you receive this transmission in error, please immediately notify the
>    sender and destroy this message in its entirety, whether in electronic or
>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>    disclosure, retention, or distribution of this transmission or the material
>    in this transmission is forbidden. We reserve the right to monitor and
>    archive electronic communications. This material does not constitute an
>    offer or solicitation with respect to the purchase or sale of any security.
>    It should not be construed to contain any recommendation regarding any
>    security or strategy. Any views expressed are those of the individual
>    sender, except where the message states otherwise and the sender is
>    authorized to state them to be the views of any such entity. This
>    communication is provided on an “as is” basis. It contains material that is
>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>    in any form, by any means, or (ii) redistributed, posted, published,
>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>    Please access the following link for important information and
>    instructions: *
>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>
>    *Securities products and services are provided by locally registered
>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>    Securities & Investments Commission; Instinet Canada Limited, member
>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>    member of The Singapore Exchange Securities Trading Private Limited and
>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>    member SIPC. *
>
>
>
> *
>    =========================================================================================================
>    *
>
>
>
>
>
>
> --
> Nitin Pawar
>
>
> =========================================================================================================
>
>
> *<<<< Disclaimer >>>>*
>
> This message is intended solely for use by the named addressee(s). If you
> receive this transmission in error, please immediately notify the sender
> and destroy this message in its entirety, whether in electronic or hard
> copy format. Any unauthorized use (and reliance thereon), copying,
> disclosure, retention, or distribution of this transmission or the material
> in this transmission is forbidden. We reserve the right to monitor and
> archive electronic communications. This material does not constitute an
> offer or solicitation with respect to the purchase or sale of any security.
> It should not be construed to contain any recommendation regarding any
> security or strategy. Any views expressed are those of the individual
> sender, except where the message states otherwise and the sender is
> authorized to state them to be the views of any such entity. This
> communication is provided on an “as is” basis. It contains material that is
> owned by Instinet Incorporated, its subsidiaries or its or their licensors,
> and may not, in whole or in part, be (i) copied, photocopied or duplicated
> in any form, by any means, or (ii) redistributed, posted, published,
> excerpted, or quoted without Instinet Incorporated's prior written consent.
> Please access the following link for important information and
> instructions:
> http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt
>
> Securities products and services are provided by locally registered
> brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
> Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
> Securities & Investments Commission; Instinet Canada Limited, member
> IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
> Securities and Futures Commission of Hong Kong; Instinet Singapore Services
> Private Limited, regulated by the Monetary Authority of Singapore, trading
> member of The Singapore Exchange Securities Trading Private Limited and
> clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
> member SIPC.
>
>
>
> =========================================================================================================
>
>
>


-- 
Nitin Pawar



=========================================================================================================
 
<<<< Disclaimer >>>>   
This message is intended solely for use by the named addressee(s). If you receive this transmission
in error, please immediately notify the sender and destroy this message in its entirety, whether
in electronic or hard copy format. Any unauthorized use (and reliance thereon), copying, disclosure,
retention, or distribution of this transmission or the material in this transmission is forbidden.
We reserve the right to monitor and archive electronic communications. This material does
not constitute an offer or solicitation with respect to the purchase or sale of any security.
It should not be construed to contain any recommendation regarding any security or strategy.
Any views expressed are those of the individual sender, except where the message states otherwise
and the sender is authorized to state them to be the views of any such entity. This communication
is provided on an “as is” basis. It contains material that is owned by Instinet Incorporated,
its subsidiaries or its or their licensors, and may not, in whole or in part, be (i) copied,
photocopied or duplicated in any form, by any means, or (ii) redistributed, posted, published,
excerpted, or quoted without Instinet Incorporated's prior written consent. Please access
the following link for important information and instructions:  http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt
  
Securities products and services are provided by locally registered brokerage subsidiaries
of Instinet Incorporated: Instinet Australia Pty Limited (ACN: 131 253 686 AFSL No: 327834),
regulated by the Australian Securities & Investments Commission; Instinet Canada Limited,
member IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the Securities and
Futures Commission of Hong Kong; Instinet Singapore Services Private Limited, regulated by
the Monetary Authority of Singapore, trading member of The Singapore Exchange Securities Trading
Private Limited and clearing member of The Central Depository (Pte) Limited; and Instinet,
LLC, member SIPC.  


=========================================================================================================
 
Mime
View raw message