hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nitin Pawar <nitinpawar...@gmail.com>
Subject Re: Hadoop Multi-node cluster on Windows
Date Thu, 05 Jun 2014 07:05:30 GMT
yes

like linux has selinux and iptables. There might be something which might
affect on windows as well.


On Thu, Jun 5, 2014 at 12:33 PM, <Wadood.Chaudhary@instinet.com> wrote:

> Thanks. I will try that. By security do you mean firewalls?
>
>
>
>
>
>
>
>
> ----- Message from "Nitin Pawar"  on 06/05/2014 02:45:12 AM -----
>  *To:* user@hadoop.apache.org  *Subject:* Re: Hadoop Multi-node cluster
> on Windows
> if you have installed a single node setup on windows then it should be
> easy to move to multinode.
>
> Never tried this myself on windows but this is what I do on linux and it
> should work on windows as well.
>
> In the current node, instead of having anything as localhost or 127.0.0.1
> .. point it to real hostname or static ip address
> make an image out of it and launch similar instances
> Launch new instances using that image.
> disable windows security on all nodes
> make sure all nodes can communicate each  other over network
> After that login to each node and start services there and it should come
> up as distributed node.
>
>
> On Thu, Jun 5, 2014 at 11:59 AM, <Wadood.Chaudhary@instinet.com> wrote:
>
>> Thanks for looking into it. I have used Horton but that is a third party
>> tool and should not be required when Apache has started supporting Windows.
>> Here is the Wiki article on Windows from Apache's.
>> https://wiki.apache.org/hadoop/Hadoop2OnWindows
>> I am able to setup in Pseudo distributed mode using the instructions
>> above. Unfortunately, the article ends there saying that instructions for
>> Multi-cluster will be added later. No mention that you cannot do it under
>> Windows or you must use Horton.
>>
>> Wadood
>>
>>
>> [image: Inactive hide details for Nitin Pawar ---06/05/2014 02:09:29
>> AM---Currently as per my knowledge only Hortonwork's distribution]Nitin
>> Pawar ---06/05/2014 02:09:29 AM---Currently as per my knowledge only
>> Hortonwork's distribution comes for windows.
>>
>> From: Nitin Pawar <nitinpawar432@gmail.com>
>> To: user@hadoop.apache.org,
>> Date: 06/05/2014 02:09 AM
>>
>> Subject: Re: Hadoop Multi-node cluster on Windows
>> ------------------------------
>>
>>
>>
>> Currently as per my knowledge only Hortonwork's distribution comes for
>> windows.
>>
>> As such hadoop does not need ssh to work but ssh in linux was needed for
>> the automated deployments.
>>
>> If you want to avoid that then, I would say setup a single machine with
>> all installation where all packages and correct configurations are present.
>> then make an image out of it and build other nodes.
>>
>> But then you will need to login to all nodes and start all services
>> manually.
>>
>> Hortonwork's HDP 2.1 on windows step by step manual install guide is here
>>
>> *http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-Win-latest/bk_releasenotes_HDP-Win/content/index.html*
>> <http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1-Win-latest/bk_releasenotes_HDP-Win/content/index.html>
>>
>>
>> On Thu, Jun 5, 2014 at 11:29 AM, <*Wadood.Chaudhary@instinet.com*
>> <Wadood.Chaudhary@instinet.com>> wrote:
>>
>>    Thanks but that is almost a 5 year old tutorial, written when Hadoop
>>    was not natively supported on Windows. As of Hadoop 2.x, Windows is
>>    supported and setup is much simpler. No Cygwin required.
>>
>>    [image: Inactive hide details for Ted Yu ---06/04/2014 10:05:15
>>    PM---See also http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%]Ted
>>    Yu ---06/04/2014 10:05:15 PM---See also
>>    *http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html*
>>    <http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html>
>>
>>    From: Ted Yu <*yuzhihong@gmail.com* <yuzhihong@gmail.com>>
>>    To: "*common-user@hadoop.apache.org* <common-user@hadoop.apache.org>"
>>    <*user@hadoop.apache.org* <user@hadoop.apache.org>>,
>>    Date: 06/04/2014 10:05 PM
>>
>>
>>    Subject: Re: Hadoop Multi-node cluster on Windows
>>    ------------------------------
>>
>>
>>
>>
>>    See also
>>    *http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html*
>>    <http://ebiquity.umbc.edu/Tutorials/Hadoop/05%20-%20Setup%20SSHD.html>
>>
>>
>>    On Wed, Jun 4, 2014 at 6:43 PM, <*Wadood.Chaudhary@instinet.com*
>>    <Wadood.Chaudhary@instinet.com>> wrote:
>>       Thanks. Linux cluster requires SSH etc. for every thing including
>>       single node. In Windows until now I have been able to setup to pseudo
>>       distributed level without SSH or CygWin. Someone was saying that some
>>       Windows service needs to be installed for communication across machines.
>>       Is that true?
>>
>>
>>
>>
>>
>>
>>
>>
>>       ----- Message from "Olivier Renault" on 06/04/2014 09:36:17 PM
>>       -----
>>    *To:*
>>    *user@hadoop.apache.org* <user@hadoop.apache.org>
>>    *Subject:*
>>    RE: Hadoop Multi-node cluster on Windows
>>    HDP run natively on Linux and Windows. It's the same code base. You
>>       can install a windows cluster in the same way as you do for Linux.
>>
>>       Olivier
>>
>>       On 4 Jun 2014 18:23, <*Wadood.Chaudhary@instinet.com*
>>       <Wadood.Chaudhary@instinet.com>> wrote:
>>
>>          Yes, but Horton runs Hadoop under Linux Virtual Machine. I am
>>          talking of pure and simple Hadoop.
>>
>>
>>
>>
>>
>>
>>
>>
>>          ----- Message from "Ted Yu" on 06/04/2014 09:18:51 PM -----
>>
>>
>>
>>    *To:*
>>
>>
>>    "*common-user@hadoop.apache.org* <common-user@hadoop.apache.org>"
>>
>>
>>    *Subject:*
>>
>>
>>    Re: Hadoop Multi-node cluster on Windows
>>
>>
>>
>>    Have you seen this ?
>>
>>
>>    *http://hortonworks.com/blog/install-hadoop-windows-hortonworks-data-platform-2-0/*
>>    <http://hortonworks.com/blog/install-hadoop-windows-hortonworks-data-platform-2-0/>
>>
>>    Cheers
>>
>>
>>    On Wed, Jun 4, 2014 at 5:47 PM, <*Wadood.Chaudhary@instinet.com*
>>    <Wadood.Chaudhary@instinet.com>> wrote:
>>       Has anyone successfully implemented Multi-node cluster under
>>       Windows using the latest version of Hadoop. We were able to run Pseudo
>>       distributed. Does it still needs SSH under Windows, or something installed
>>       as service.
>>
>>       [image: Inactive hide details for ch huang ---06/04/2014 08:40:38
>>       PM---hi,mailist: my company signed a new IDC , i need m]ch huang
>>       ---06/04/2014 08:40:38 PM---hi,mailist:               my company signed a
>>       new IDC , i need move the total 50T data
>>
>>       From: ch huang <*justlooks@gmail.com* <justlooks@gmail.com>>
>>       To: *user@hadoop.apache.org* <user@hadoop.apache.org>,
>>       Date: 06/04/2014 08:40 PM
>>       Subject: issue about move data between two hdoop cluster
>>       ------------------------------
>>
>>
>>
>>       hi,mailist:
>>                     my company signed a new IDC , i need move the total
>>       50T data from old hadoop cluster to the new cluster in new location ,how to
>>       do it?
>>
>>
>> *
>>       =========================================================================================================
>>       *
>>
>>       *<<<< Disclaimer >>>>*
>>
>>       *This message is intended solely for use by the named
>>       addressee(s). If you receive this transmission in error, please immediately
>>       notify the sender and destroy this message in its entirety, whether in
>>       electronic or hard copy format. Any unauthorized use (and reliance
>>       thereon), copying, disclosure, retention, or distribution of this
>>       transmission or the material in this transmission is forbidden. We reserve
>>       the right to monitor and archive electronic communications. This material
>>       does not constitute an offer or solicitation with respect to the purchase
>>       or sale of any security. It should not be construed to contain any
>>       recommendation regarding any security or strategy. Any views expressed are
>>       those of the individual sender, except where the message states otherwise
>>       and the sender is authorized to state them to be the views of any such
>>       entity. This communication is provided on an “as is” basis. It contains
>>       material that is owned by Instinet Incorporated, its subsidiaries or its or
>>       their licensors, and may not, in whole or in part, be (i) copied,
>>       photocopied or duplicated in any form, by any means, or (ii) redistributed,
>>       posted, published, excerpted, or quoted without Instinet Incorporated's
>>       prior written consent. Please access the following link for important
>>       information and instructions: *
>>       *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>>       <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>>
>>       *Securities products and services are provided by locally
>>       registered brokerage subsidiaries of Instinet Incorporated: Instinet
>>       Australia Pty Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the
>>       Australian Securities & Investments Commission; Instinet Canada Limited,
>>       member IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by
>>       the Securities and Futures Commission of Hong Kong; Instinet Singapore
>>       Services Private Limited, regulated by the Monetary Authority of Singapore,
>>       trading member of The Singapore Exchange Securities Trading Private Limited
>>       and clearing member of The Central Depository (Pte) Limited; and Instinet,
>>       LLC, member SIPC. *
>>
>>
>>
>> *
>>       =========================================================================================================
>>       *
>>
>>
>>
>>
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>    *<<<< Disclaimer >>>>*
>>
>>    *This message is intended solely for use by the named addressee(s).
>>    If you receive this transmission in error, please immediately notify the
>>    sender and destroy this message in its entirety, whether in electronic or
>>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>>    disclosure, retention, or distribution of this transmission or the material
>>    in this transmission is forbidden. We reserve the right to monitor and
>>    archive electronic communications. This material does not constitute an
>>    offer or solicitation with respect to the purchase or sale of any security.
>>    It should not be construed to contain any recommendation regarding any
>>    security or strategy. Any views expressed are those of the individual
>>    sender, except where the message states otherwise and the sender is
>>    authorized to state them to be the views of any such entity. This
>>    communication is provided on an “as is” basis. It contains material that is
>>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>>    in any form, by any means, or (ii) redistributed, posted, published,
>>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>>    Please access the following link for important information and
>>    instructions: *
>>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>>
>>    *Securities products and services are provided by locally registered
>>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>>    Securities & Investments Commission; Instinet Canada Limited, member
>>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>>    member of The Singapore Exchange Securities Trading Private Limited and
>>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>>    member SIPC. *
>>
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>
>>    CONFIDENTIALITY NOTICE
>>    NOTICE: This message is intended for the use of the individual or
>>    entity to which it is addressed and may contain information that is
>>    confidential, privileged and exempt from disclosure under applicable law.
>>    If the reader of this message is not the intended recipient, you are hereby
>>    notified that any printing, copying, dissemination, distribution,
>>    disclosure or forwarding of this communication is strictly prohibited. If
>>    you have received this communication in error, please contact the sender
>>    immediately and delete it from your system. Thank You.
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>    *<<<< Disclaimer >>>>*
>>
>>    *This message is intended solely for use by the named addressee(s).
>>    If you receive this transmission in error, please immediately notify the
>>    sender and destroy this message in its entirety, whether in electronic or
>>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>>    disclosure, retention, or distribution of this transmission or the material
>>    in this transmission is forbidden. We reserve the right to monitor and
>>    archive electronic communications. This material does not constitute an
>>    offer or solicitation with respect to the purchase or sale of any security.
>>    It should not be construed to contain any recommendation regarding any
>>    security or strategy. Any views expressed are those of the individual
>>    sender, except where the message states otherwise and the sender is
>>    authorized to state them to be the views of any such entity. This
>>    communication is provided on an “as is” basis. It contains material that is
>>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>>    in any form, by any means, or (ii) redistributed, posted, published,
>>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>>    Please access the following link for important information and
>>    instructions: *
>>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>>
>>    *Securities products and services are provided by locally registered
>>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>>    Securities & Investments Commission; Instinet Canada Limited, member
>>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>>    member of The Singapore Exchange Securities Trading Private Limited and
>>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>>    member SIPC. *
>>
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>
>>
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>    *<<<< Disclaimer >>>>*
>>
>>    *This message is intended solely for use by the named addressee(s).
>>    If you receive this transmission in error, please immediately notify the
>>    sender and destroy this message in its entirety, whether in electronic or
>>    hard copy format. Any unauthorized use (and reliance thereon), copying,
>>    disclosure, retention, or distribution of this transmission or the material
>>    in this transmission is forbidden. We reserve the right to monitor and
>>    archive electronic communications. This material does not constitute an
>>    offer or solicitation with respect to the purchase or sale of any security.
>>    It should not be construed to contain any recommendation regarding any
>>    security or strategy. Any views expressed are those of the individual
>>    sender, except where the message states otherwise and the sender is
>>    authorized to state them to be the views of any such entity. This
>>    communication is provided on an “as is” basis. It contains material that is
>>    owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>>    and may not, in whole or in part, be (i) copied, photocopied or duplicated
>>    in any form, by any means, or (ii) redistributed, posted, published,
>>    excerpted, or quoted without Instinet Incorporated's prior written consent.
>>    Please access the following link for important information and
>>    instructions: *
>>    *http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt*
>>    <http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt>
>>
>>    *Securities products and services are provided by locally registered
>>    brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>>    Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>>    Securities & Investments Commission; Instinet Canada Limited, member
>>    IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>>    Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>>    Private Limited, regulated by the Monetary Authority of Singapore, trading
>>    member of The Singapore Exchange Securities Trading Private Limited and
>>    clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>>    member SIPC. *
>>
>>
>>
>> *
>>    =========================================================================================================
>>    *
>>
>>
>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>>
>> =========================================================================================================
>>
>>
>> *<<<< Disclaimer >>>>*
>>
>> This message is intended solely for use by the named addressee(s). If you
>> receive this transmission in error, please immediately notify the sender
>> and destroy this message in its entirety, whether in electronic or hard
>> copy format. Any unauthorized use (and reliance thereon), copying,
>> disclosure, retention, or distribution of this transmission or the material
>> in this transmission is forbidden. We reserve the right to monitor and
>> archive electronic communications. This material does not constitute an
>> offer or solicitation with respect to the purchase or sale of any security.
>> It should not be construed to contain any recommendation regarding any
>> security or strategy. Any views expressed are those of the individual
>> sender, except where the message states otherwise and the sender is
>> authorized to state them to be the views of any such entity. This
>> communication is provided on an “as is” basis. It contains material that is
>> owned by Instinet Incorporated, its subsidiaries or its or their licensors,
>> and may not, in whole or in part, be (i) copied, photocopied or duplicated
>> in any form, by any means, or (ii) redistributed, posted, published,
>> excerpted, or quoted without Instinet Incorporated's prior written consent.
>> Please access the following link for important information and
>> instructions:
>> http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt
>>
>> Securities products and services are provided by locally registered
>> brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
>> Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
>> Securities & Investments Commission; Instinet Canada Limited, member
>> IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
>> Securities and Futures Commission of Hong Kong; Instinet Singapore Services
>> Private Limited, regulated by the Monetary Authority of Singapore, trading
>> member of The Singapore Exchange Securities Trading Private Limited and
>> clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
>> member SIPC.
>>
>>
>>
>> =========================================================================================================
>>
>>
>>
>
>
> --
> Nitin Pawar
>
>
> =========================================================================================================
>
>
> *<<<< Disclaimer >>>>*
>
> This message is intended solely for use by the named addressee(s). If you
> receive this transmission in error, please immediately notify the sender
> and destroy this message in its entirety, whether in electronic or hard
> copy format. Any unauthorized use (and reliance thereon), copying,
> disclosure, retention, or distribution of this transmission or the material
> in this transmission is forbidden. We reserve the right to monitor and
> archive electronic communications. This material does not constitute an
> offer or solicitation with respect to the purchase or sale of any security.
> It should not be construed to contain any recommendation regarding any
> security or strategy. Any views expressed are those of the individual
> sender, except where the message states otherwise and the sender is
> authorized to state them to be the views of any such entity. This
> communication is provided on an “as is” basis. It contains material that is
> owned by Instinet Incorporated, its subsidiaries or its or their licensors,
> and may not, in whole or in part, be (i) copied, photocopied or duplicated
> in any form, by any means, or (ii) redistributed, posted, published,
> excerpted, or quoted without Instinet Incorporated's prior written consent.
> Please access the following link for important information and
> instructions:
> http://instinet.com/includes/index.jsp?thePage=/html/le_index.txt
>
> Securities products and services are provided by locally registered
> brokerage subsidiaries of Instinet Incorporated: Instinet Australia Pty
> Limited (ACN: 131 253 686 AFSL No: 327834), regulated by the Australian
> Securities & Investments Commission; Instinet Canada Limited, member
> IIROC/CIPF; Instinet Pacific Limited, authorized and regulated by the
> Securities and Futures Commission of Hong Kong; Instinet Singapore Services
> Private Limited, regulated by the Monetary Authority of Singapore, trading
> member of The Singapore Exchange Securities Trading Private Limited and
> clearing member of The Central Depository (Pte) Limited; and Instinet, LLC,
> member SIPC.
>
>
>
> =========================================================================================================
>
>
>


-- 
Nitin Pawar

Mime
View raw message