Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4385FCDFF for ; Tue, 10 Sep 2013 18:51:57 +0000 (UTC) Received: (qmail 90493 invoked by uid 500); 10 Sep 2013 18:51:48 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 90340 invoked by uid 500); 10 Sep 2013 18:51:48 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 90332 invoked by uid 99); 10 Sep 2013 18:51:47 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 18:51:47 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of aji1705@gmail.com designates 209.85.217.182 as permitted sender) Received: from [209.85.217.182] (HELO mail-lb0-f182.google.com) (209.85.217.182) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Sep 2013 18:51:41 +0000 Received: by mail-lb0-f182.google.com with SMTP id c11so6677424lbj.13 for ; Tue, 10 Sep 2013 11:51:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=5ZnlPHp+Ev3Sod8+ChPUBklgCA1gL7512S6pIotg4ag=; b=iwT+K75cxBxQexPIX1Io7MMPCzGRPOlaeelB2v3fCevhF+Wkwi9T8XL4Nk+QV9EhlS iVlhlVG0uNJmUPaDjSbYEcKD0jdYssh7e0XBe9+etib7jvG8Ovy2VfHisxmatXs3HZN5 WcSu9EXHvcj18xuf2MALbVoKZOlPq2F71yYFYwKrDudRFZ5ungPoa1uh5D1D+XOZEUgZ 3kfIbHhCbNQHqZY5J3Gs6Y4GnTEx0KlknpxmohaYAZHBbSV0zTxyWQdqjGkoXNhd+0z/ PeU9so08hR28TQ/OmGDyla0yvzbwo48q7Tg2hl+q4okiqS9+FtAgG/a1Epu0j+osiPV6 k9fQ== MIME-Version: 1.0 X-Received: by 10.112.55.173 with SMTP id t13mr2437935lbp.39.1378839080919; Tue, 10 Sep 2013 11:51:20 -0700 (PDT) Received: by 10.112.142.167 with HTTP; Tue, 10 Sep 2013 11:51:20 -0700 (PDT) In-Reply-To: References: Date: Tue, 10 Sep 2013 14:51:20 -0400 Message-ID: Subject: Re: Hadoop on IPv6 From: Aji Janis To: user Content-Type: multipart/alternative; boundary=001a1133e3309ee7d604e60bfda6 X-Virus-Checked: Checked by ClamAV on apache.org --001a1133e3309ee7d604e60bfda6 Content-Type: text/plain; charset=ISO-8859-1 Thank you for the clarification Adam. On Tue, Sep 10, 2013 at 12:34 PM, Adam Muise wrote: > Harsh is giving you a best practice for JVMs using IPv4 in general. As > what I am suggesting is IPv4-only connections to the Hadoop daemons and > clients on the cluster and gateway, you would not have issues heeding his > advice. > > Obviously you would not set "-Djava.net.preferIPv4Stack=true" to any java > client connecting to the proxy server as the proxy would expect IPv6 and > you stated that's what you wanted your network to run on. The assumption is > that any application or client outside of the cluster and gateway node only > uses IPv6. > > I suggest you identify how your clients access the Hadoop cluster (based > on Hive JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a > gateway node or simply log into cluster boxes, and what applications rely > on connections to this cluster. That is your list of hosts and ports that > you will have to find a network solution for or hide behind a gateway node. > Make your decisions around this. > > Remember that my proxy suggestion only worked for RESTful web based > traffic and that's not the entire stack that typically gets used in Hadoop. > ODBC/JDBC does NOT fall under this category for example. If you are > upgrading your network already, ask the network specialists or consultants > about NAT64 and equivalent options. > > > On Tue, Sep 10, 2013 at 12:12 PM, Aji Janis wrote: > >> Harsh, >> >> We have the config params that you mentioned, but to be clear are you >> saying that the suggestions Adam made won't work because of these >> configurations ? >> >> Thanks, >> Aji >> >> >> On Mon, Sep 9, 2013 at 4:55 PM, Harsh J wrote: >> >>> For Hadoop JVMs, generally speaking, need to have the following in >>> your hadoop-env.sh for HADOOP_OPTS: >>> >>> export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true $HADOOP_OPTS" >>> >>> We also ship with this pre-added in the default configs of more recent >>> releases. >>> >>> You may also need these on other JVM based daemons' opts, such as >>> HBASE_OPTS/etc. depending on what all you run. >>> >>> On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis wrote: >>> > Adam, >>> > Thank you for the suggestion. Could you point me to any documentation >>> that >>> > can help with the configuration changes I need to make? Thanks. >>> > >>> > Aji >>> > >>> > >>> > On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise >>> wrote: >>> >> >>> >> Keep the Hadoop cluster and it's network backbone on IPv4. It should >>> be >>> >> isolated from the rest of your network for security and performance >>> reasons >>> >> already anyway. >>> >> >>> >> Enable IPv6 on your gateway nodes and let clients jump in for access >>> via >>> >> the ipv6 network and use the cluster via ipv4 on the gateway node. >>> You could >>> >> even setup a web proxy for some of the required sites using nginx or >>> your >>> >> favorite proxy server. Leave the ipv6 conversion of the Hadoop >>> cluster until >>> >> it is fully supported and well-tested. >>> >> >>> >> >>> >> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis wrote: >>> >>> >>> >>> We are moving to a IPv6 network. I know Hadoop installation says to >>> >>> disable IPv6. Does anyone have any suggestions for work arround? >>> Currently, >>> >>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance. >>> >> >>> >> >>> >> >>> >> >>> >> -- >>> >> >>> >> >>> >> Adam Muise >>> >> Solution Engineer >>> >> Hortonworks >>> >> amuise@hortonworks.com >>> >> 416-417-4037 >>> >> >>> >> Hortonworks - Develops, Distributes and Supports Enterprise Apache >>> Hadoop. >>> >> >>> >> Hortonworks Virtual Sandbox >>> >> >>> >> Hadoop: Disruptive Possibilities by Jeff Needham >>> >> >>> >> CONFIDENTIALITY NOTICE >>> >> NOTICE: This message is intended for the use of the individual or >>> entity >>> >> to which it is addressed and may contain information that is >>> confidential, >>> >> privileged and exempt from disclosure under applicable law. If the >>> reader of >>> >> this message is not the intended recipient, you are hereby notified >>> that any >>> >> printing, copying, dissemination, distribution, disclosure or >>> forwarding of >>> >> this communication is strictly prohibited. If you have received this >>> >> communication in error, please contact the sender immediately and >>> delete it >>> >> from your system. Thank You. >>> > >>> > >>> >>> >>> >>> -- >>> Harsh J >>> >> >> > > > -- > * > * > * > * > *Adam Muise* > Solution Engineer > *Hortonworks* > amuise@hortonworks.com > 416-417-4037 > > Hortonworks - Develops, Distributes and Supports Enterprise Apache Hadoop. > > Hortonworks Virtual Sandbox > > Hadoop: Disruptive Possibilities by Jeff Needham > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > --001a1133e3309ee7d604e60bfda6 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

Thank you for the clarification Adam.


On Tue, S= ep 10, 2013 at 12:34 PM, Adam Muise <amuise@hortonworks.com> wrote:
Harsh is giving you a best = practice for JVMs using IPv4 in general. As what I am suggesting is IPv4-on= ly connections to the Hadoop daemons and clients on the cluster and gateway= , you would not have issues heeding his advice.=A0

Obviously you would not set "-Djava.net.preferIPv4Stack=3Dtrue" to any ja= va client connecting to the proxy server as the proxy would expect IPv6 and= you stated that's what you wanted your network to run on. The assumpti= on is that any application or client outside of the cluster and gateway nod= e only uses IPv6.

I s= uggest you identify how your clients access the Hadoop cluster (based on Hi= ve JDBC/ODBC, JobTracker, Datanodes, NameNode), whether they use a gateway = node or simply log into cluster boxes, and what applications rely on connec= tions to this cluster. That is your list of hosts and ports that you will h= ave to find a network solution for or hide behind a gateway node. Make your= decisions around this.=A0

Rem= ember that my proxy suggestion only worked for RESTful web based traffic an= d that's not the entire stack that typically gets used in Hadoop. ODBC/= JDBC does NOT fall under this category for example.=A0If you are upgrading your net= work already, ask the network specialists or consultants about NAT64 and eq= uivalent options.=A0

On Tue, Sep 10, 2013 at 12:12 PM, Aji Jani= s <aji1705@gmail.com> wrote:
Harsh,

=A0We have the config params that you mentioned, but to be clear are you= saying that the suggestions Adam made won't work because of these conf= igurations ?

Thanks,
Aji


On Mon, Sep 9, 2013 at 4:55 PM, Harsh J <harsh@cloudera.com= > wrote:
For Hadoop JVMs, generally speaking, need to= have the following in
your hadoop-env.sh for HADOOP_OPTS:

export HADOOP_OPTS=3D"-Djava.net.preferIPv4Stack=3Dtrue $HADOOP_OPTS&q= uot;

We also ship with this pre-added in the default configs of more recent rele= ases.

You may also need these on other JVM based daemons' opts, such as
HBASE_OPTS/etc. depending on what all you run.

On Tue, Sep 10, 2013 at 2:05 AM, Aji Janis <aji1705@gmail.com> wrote:
> Adam,
> Thank you for the suggestion. Could you point me to any documentation = that
> can help with the configuration changes I need to make? Thanks.
>
> Aji
>
>
> On Mon, Sep 9, 2013 at 1:30 PM, Adam Muise <amuise@hortonworks.com> wrote:<= br> >>
>> Keep the Hadoop cluster and it's network backbone on IPv4. It = should be
>> isolated from the rest of your network for security and performanc= e reasons
>> already anyway.
>>
>> Enable IPv6 on your gateway nodes and let clients jump in for acce= ss via
>> the ipv6 network and use the cluster via ipv4 on the gateway node.= You could
>> even setup a web proxy for some of the required sites using nginx = or your
>> favorite proxy server. Leave the ipv6 conversion of the Hadoop clu= ster until
>> it is fully supported and well-tested.
>>
>>
>> On Mon, Sep 9, 2013 at 1:19 PM, Aji Janis <aji1705@gmail.com> wrote:
>>>
>>> We are moving to a IPv6 network. I know Hadoop installation sa= ys to
>>> disable IPv6. Does anyone have any suggestions for work arroun= d? Currently,
>>> using Cloudera version 0.20.2-cdh3u5. Thanks in advance.
>>
>>
>>
>>
>> --
>>
>>
>> Adam Muise
>> Solution Engineer
>> Hortonworks
>> amuise= @hortonworks.com
>> 416-417-4037
>>
>> Hortonworks - Develops, Distributes and Supports Enterprise Apache= Hadoop.
>>
>> Hortonworks Virtual Sandbox
>>
>> Hadoop: Disruptive Possibilities by Jeff Needham
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or = entity
>> to which it is addressed and may contain information that is confi= dential,
>> privileged and exempt from disclosure under applicable law. If the= reader of
>> this message is not the intended recipient, you are hereby notifie= d that any
>> printing, copying, dissemination, distribution, disclosure or forw= arding of
>> this communication is strictly prohibited. If you have received th= is
>> communication in error, please contact the sender immediately and = delete it
>> from your system. Thank You.
>
>



--
Harsh J




--
=
CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.

--001a1133e3309ee7d604e60bfda6--