Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of jing@hortonworks.com designates
 209.85.216.54 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1385516012.80467.YahooMailNeo@web125703.mail.ne1.yahoo.com>
References: 
 <CABbyifPzitePWuaBfdm9b03BstOiam95zLo3aW3NxJ=GXCAw=A@mail.gmail.com>
 <CADdVdVH+cr4Wt_6P8aA0xb9NpAxJgd9sadFb+FDV3EsVBiXoTw@mail.gmail.com>
 <CA+01ahh3wRPAwver_kVUMBqLOgPDQW3Oq2ijuN+e=X6sWFtXsw@mail.gmail.com>
 <CABbyifNNDMeN9uKgjoUU6SjM1w93WHKttByajwZPYkTZFt8+Vg@mail.gmail.com>
 <CABbyifNzQh=Vz9UytmzUPD627UqJ_nPUYqgLRnXK+PDOYYOXog@mail.gmail.com>
 <1385516012.80467.YahooMailNeo@web125703.mail.ne1.yahoo.com>
From: Jing Zhao <jing@hortonworks.com>
Date: Tue, 26 Nov 2013 18:01:32 -0800
Message-ID: 
 <CABGkNE9_pdEuXxZ9i1Qbay3TTPq=49y-1JVwSJivaUVUV67EdQ@mail.gmail.com>
Subject: Re: Deprecate hftp / hsftp
To: hdfs-dev@hadoop.apache.org,
	"Tsz Wo (Nicholas), Sze" <s29752-hadoopdev@yahoo.com>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

+1

So I've reviewed most of Haohui's patch for developing swebhdfs,
fixing https setup for hsftp, and supporting HA in webhdfs. For both
webhdfs and swebhdfs we have run system tests on both secured and
insecure cluster, including running distcp to transfer data between
1.x and 2.x. I believe currently webhdfs/swebhdfs is in a very good
shape.

Also, recently the new web UI in HDFS requires enabling WebHDFS by
default (HDFS-5532). Thus it is a good time to deprecate HFTP.

On Tue, Nov 26, 2013 at 5:33 PM, Tsz Wo (Nicholas), Sze
<s29752-hadoopdev@yahoo.com> wrote:
> +1
>
> WebHDFS has been around for two years and has become mature.  It is a rew=
rite of HFTP and is intended to replace HFTP as stated in my blog [1].  Now=
 is indeed a good time to deprecate HFTP.
>
> Haohui, thanks a lot for adding swebhdfs and fixing the secure setups!
>
> Tsz-Wo
> [1] http://hortonworks.com/blog/webhdfs-=96-http-rest-access-to-hdfs/
>
>
>
>
> On Tuesday, November 26, 2013 2:30 PM, Haohui Mai <hmai@hortonworks.com> =
wrote:
>
> I meant HDP 1.3, which is based on Hadoop 1.2.
>>
>>~Haohui
>>
>>
>>On Tue, Nov 26, 2013 at 2:19 PM, Haohui Mai <hmai@hortonworks.com> wrote:
>>
>>> Hi Lohit,
>>>
>>> Some of our customers are running Hadoop 1.3, and they run distcp to
>>> transfer data via webhdfs in production.
>>>
>>> We use distcp to transfer data between Hadoop 1.x and Hadoop 2.x. As fa=
r
>>> as I know webhdfs is the most reliable way to transfer data between sec=
ure
>>> and insecure clusters across different Hadoop versions.
>>>
>>> ~Haohui
>>>
>>>
>>> On Tue, Nov 26, 2013 at 1:51 PM, lohit <lohit.vijayarenu@gmail.com> wro=
te:
>>>
>>>> Curious to know usage of webhdfs for Hadoop 1.0. Are people using it i=
n
>>>> production?
>>>> Reason I ask is because we use lot of hftp to transfer data between Ha=
doop
>>>> 1.0 and Hadoop 2.0 as of today.
>>>> Removing hftp would mean that anyone using Hadoop 1.0 and Hadoop 3.0 w=
ould
>>>> have to move to webhdfs, right?
>>>>
>>>>
>>>>
>>>> 2013/11/26 Suresh Srinivas <suresh@hortonworks.com>
>>>>
>>>> > Thanks Haohui for all your hard work in this area. I am +1 on this
>>>> > proposal.
>>>> >
>>>> >
>>>> > On Tue, Nov 26, 2013 at 12:50 PM, Haohui Mai <hmai@hortonworks.com>
>>>> wrote:
>>>> >
>>>> > > Hi,
>>>> > >
>>>> > > Recently I've been focusing on fixing hftp / hsftp / webhdfs /
>>>> swebhdfs
>>>> > in
>>>> > > various set ups. Now we have reached the state that all the above =
file
>>>> > > systems can work in both secure and insecure clusters, and transfe=
r
>>>> data
>>>> > > through both http and https.
>>>> > >
>>>> > > Taking a step back, these file systems are very similar, and I'm
>>>> > wondering
>>>> > > whether it is a good time to deprecate hftp and hsftp in Hadoop ri=
ght
>>>> > now.
>>>> > >
>>>> > > The main reason is that hftp / hsftp only provides a strict subset=
 of
>>>> > > functionalities that webhdfs / swebhdfs offer. Notably, webhdfs /
>>>> > swebhdfs
>>>> > > support writes and HA which hftp / hsftp do not support. It's more
>>>> > natural
>>>> > > to move forward with webhdfs / swebhdfs, keeping both hftp / hsftp
>>>> around
>>>> > > seems introducing more work.
>>>> > >
>>>> > > Another reason is that webhdfs has been supported since Hadoop 1, =
thus
>>>> > > getting rid of hftp / hsftp does not seem removing any features, e=
ven
>>>> the
>>>> > > users are trying to migrate from Hadoop 1 to Hadoop 2.
>>>> > >
>>>> > >
>>>> > > Your ideas are appreciated.
>>>> > >
>>>> > > Thanks,
>>>> > > Haohui
>>>> > >
>>>> > > --
>>>> > > CONFIDENTIALITY NOTICE
>>>> > > NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>> > to
>>>> > > which it is addressed and may contain information that is
>>>> confidential,
>>>> > > privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> > > of this message is not the intended recipient, you are hereby noti=
fied
>>>> > that
>>>> > > any printing, copying, dissemination, distribution, disclosure or
>>>> > > forwarding of this communication is strictly prohibited. If you ha=
ve
>>>> > > received this communication in error, please contact the sender
>>>> > immediately
>>>> > > and delete it from your system. Thank You.
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > http://hortonworks.com/download/
>>>> >
>>>> > --
>>>> > CONFIDENTIALITY NOTICE
>>>> > NOTICE: This message is intended for the use of the individual or
>>>> entity to
>>>> > which it is addressed and may contain information that is confidenti=
al,
>>>> > privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>> > of this message is not the intended recipient, you are hereby notifi=
ed
>>>> that
>>>> > any printing, copying, dissemination, distribution, disclosure or
>>>> > forwarding of this communication is strictly prohibited. If you have
>>>> > received this communication in error, please contact the sender
>>>> immediately
>>>> > and delete it from your system. Thank You.
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Have a Nice Day!
>>>> Lohit
>>>>
>>>
>>>
>>
>>--
>>CONFIDENTIALITY NOTICE
>>NOTICE: This message is intended for the use of the individual or entity =
to
>>which it is addressed and may contain information that is confidential,
>>privileged and exempt from disclosure under applicable law. If the reader
>>of this message is not the intended recipient, you are hereby notified th=
at
>>any printing, copying, dissemination, distribution, disclosure or
>>forwarding of this communication is strictly prohibited. If you have
>>received this communication in error, please contact the sender immediate=
ly
>>and delete it from your system. Thank You.
>>
>>
>>

--=20
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to=
=20
which it is addressed and may contain information that is confidential,=20
privileged and exempt from disclosure under applicable law. If the reader=
=20
of this message is not the intended recipient, you are hereby notified that=
=20
any printing, copying, dissemination, distribution, disclosure or=20
forwarding of this communication is strictly prohibited. If you have=20
received this communication in error, please contact the sender immediately=
=20
and delete it from your system. Thank You.