hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bjoern Schiessle <bjo...@schiessle.org>
Subject Best way to write files to hdfs (from a Python app)
Date Mon, 09 Aug 2010 16:18:27 GMT
Hi all,

I develop a web application with Django(Python) which should access an
hbase database and store large files to hdfs.

I wonder what is the best way to write files to hdfs from my Django app?
Basically I thought about two ways but maybe you know a better option:

1. First store the file on the local file system and than move it with
the thrift interface to hdfs. (downside: needs always enough space on the
web application server)

2. Use hdfs-fuse to mount the hdfs file system and write the file directly
to hdfs. (downside: I don't know how well hdfs-fuse is supported and I'm
not sure if it is a good idea to mount the file system and run large
operation on it).

Since I'm new to hdfs and Hadoop in general I'm not sure what's the best
and less error-prone way.

What would be your recommendation?

Thanks a lot! 

View raw message