hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luca Pireddu <pire...@crs4.it>
Subject Re: Is it possible to access the HDFS via Java OUTSIDE the Cluster?
Date Tue, 06 Sep 2011 12:26:16 GMT
On 09/05/2011 05:42 PM, Sofia Georgiakaki wrote:
> Good evening,
> this topic seems very interesting. To be sure I understood the case -
> do you mean that I can write a simple Java program and access a file
> stored in HDFS from within the java application?
> Assuming that I have e.g. 10 files of size 30GB each stored on HDFS
> on a cluster of 15 nodes, how can I run a java program that accesses
> these files and reads some blocks from them? Is it possible to do it
> without copying the files via -copyToLocal ?
> If yes, could anyone give some general directions on the general form
> of such a java code, and on how to run such a program?
> Thank  you in advance Sofia

You certainly can access a file on HDFS through a simple Java program. 
You can also access your files with an even simpler Python program using 
the Pydoop HDFS module (http://pydoop.sf.net/).  Here's a simple Python 
script to print a file:

import pydoop.hdfs as py_hdfs

fs = py_hdfs.hdfs('default', 0)

for line in fs.open_file("myfile", 'r'):
	print line

Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
Pula 09010 (CA), Italy
Tel: +39 0709250452

View raw message