Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of
 prvs=80586a3e8=afaris@linkedin.com designates 69.28.149.80 as permitted
 sender)
From: Adam Faris <afaris@linkedin.com>
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Subject: Re: RES: I want to call HDFS REST api to upload a file using
 httplib.
Thread-Topic: RES: I want to call HDFS REST api to upload a file using
 httplib.
Thread-Index: AQHONTd/KjN3maX6lU+BEgecZExuzZjQg3gA
Date: Wed, 10 Apr 2013 22:39:32 +0000
Message-ID: <3DB84EA5F65C804D98EE9F8562FAA53B5137A680@ESV4-MBX02.linkedin.biz>
References: <tencent_2807A0A54837FF7824A6A816@qq.com>
 <F6B59961-E355-4A6C-B80B-11E0B43E58F5@yahoo-inc.com>
In-Reply-To: <F6B59961-E355-4A6C-B80B-11E0B43E58F5@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="GB18030"
Content-ID: <C44AECD7F9086E4AB094B8A2BC066FAF@linkedin.com>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Creating a file on HDFS is a multi-step process. If you allow me to general=
ize and skip over a lot of details, it's essentially a two step process.   =
 1) ask the namenode for a location to write the blocks.   2) connect to th=
e datanode and write your data.   The output from your curl statement is th=
e response from the namenode, which returns a 307 and a location.   Your cl=
ient, (curl) is supposed to say hey I have a new location and connect to th=
e data node to write the data.   If you add -L to your curl request, you'll=
 see this happening.  =20

Just as a FYI, an example of using httplib for webhdfs is a solved problem.=
  You have your pick of languages on github that do this already.  :) =20

https://github.com/search?q=3Dwebhdfs&type=3DRepositories&s=3Dupdated   =20

-- Adam

On Apr 9, 2013, at 8:32 AM, Daryn Sharp <daryn@yahoo-inc.com> wrote:

> Try adding -L to your curl and see if that works.
>=20
> Daryn
>=20
> On Apr 8, 2013, at 11:05 PM, =D0=A1=D1=A7=D4=B0PHP wrote:
>=20
>> Really Thanks.
>> But the returned URL is wrong. And the localhost is the real URL, as i t=
ested successfully with curl using "localhost".
>> Can anybody help me translate the curl to Python httplib?
>> curl -i -X PUT -T <LOCAL_FILE> "http://<DATANODE>:<PORT>/webhdfs/v1/<PAT=
H>?op=3DCREATE"
>> I test it using python httplib, and receive the right response. But the =
file uploaded to HDFS is empty, no data sent!!
>> Is "conn.send(data)"  the problem?
>>=20
>> ------------------ Original ------------------
>> From:  "MARCOS MEDRADO RUBINELLI"<marcosm@buscapecompany.com>;
>> Date:  Mon, Apr 8, 2013 04:22 PM
>> To:  "user@hadoop.apache.org"<user@hadoop.apache.org>;
>> Subject:  RES: I want to call HDFS REST api to upload a file using httpl=
ib.
>>=20
>> On your first call, Hadoop will return a URL pointing to a datanode in t=
he Location header of the 307 response. On your second call, you have to us=
e that URL instead of constructing your own. You can see the specific docum=
entation here:
>> http://hadoop.apache.org/docs/r1.0.4/webhdfs.html#CREATE
>>=20
>> Regards,
>> Marcos
>>=20
>> I want to call HDFS REST api to upload a file using httplib.
>>=20
>> My program created the file, but no content is in it.
>>=20
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D
>>=20
>> Here is my code:
>>=20
>> import
>>  httplib
>>=20
>> conn
>> =3Dhttplib.HTTPConnection("localhost:50070")
>>=20
>> conn
>> .request("PUT","/webhdfs/v1/levi/4?op=3DCREATE")
>>=20
>> res
>> =3Dconn.getresponse()
>> print res.status,res.
>> reason
>> conn
>> .close()
>>=20
>>=20
>> conn
>> =3Dhttplib.HTTPConnection("localhost:50075")
>>=20
>> conn
>> .connect()
>>=20
>> conn
>> .putrequest("PUT","/webhdfs/v1/levi/4?op=3DCREATE&user.name=3Dlevi")
>>=20
>> conn
>> .endheaders()
>>=20
>> a_file
>> =3Dopen("/home/levi/4","rb")
>>=20
>> a_file
>> .seek(0)
>>=20
>> data
>> =3Da_file.read()
>>=20
>> conn
>> .send(data)
>>=20
>> res
>> =3Dconn.getresponse()
>> print res.status,res.
>> reason
>> conn
>> .close()
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D
>>=20
>> Here is the return:
>>=20
>> 307 TEMPORARY_REDIRECT 201 Created
>>=20
>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D
>>=20
>> OK, the file was created, but no content was sent.
>>=20
>> When I comment the #conn.send(data), the result is the same, still no co=
ntent.
>>=20
>> Maybe the file read or the send is wrong, not sure.
>>=20
>> Do you know how this happened?
>>=20
>=20