hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stu24m...@yahoo.com
Subject Re: Best way to write files to hdfs (from a Python app)
Date Tue, 10 Aug 2010 16:02:04 GMT
Hello Bjoern,

 Thrift works with binary data - at least with hbase. I have a C# app that people can use
to put binary (and get) files in hbase via Thrift. I'll send example code later. I also have
java apps that upload files to hdfs directly and are not on the server - but they do need
access to the copies of the config files. But they just use the standard java hdfs api.

------Original Message------
From: Bjoern Schiessle
To: hdfs-user@hadoop.apache.org
ReplyTo: hdfs-user@hadoop.apache.org
Subject: Re: Best way to write files to hdfs (from a Python app)
Sent: Aug 10, 2010 05:06

Hi Philip,

On Mon, 9 Aug 2010 16:35:07 -0700 Philip Zeyliger wrote:
> To give you an example of how this may be done, HUE, under the covers,
> pipes your data to 'bin/hadoop fs -Dhadoop.job.ugi=user,group put -
> path'. (That's from memory, but it's approximately right; the full
> python code is at
> http://github.com/cloudera/hue/blob/master/desktop/libs/hadoop/src/hadoop/fs/hadoopfs.py#L692
> )

Thank you! If I understand it correctly this only works if my python app
runs on the same server as hadoop, right?

I would like to run the python app on a different server. Therefore my
two ideas (1) Thrift or (2) hdfs-fuse.

Thrift seems to be able to store string content only to hdfs but no
binary files. At least I couldn't find an interface for a simple put

So at the moment I'm not sure how to continue.

Any ideas?


View raw message