hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "HadoopDfsReadWriteExample" by LohitVijayarenu
Date Sun, 04 Nov 2007 19:40:21 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by LohitVijayarenu:

New page:
~+Simple Example to Read and Write files from Hadoop DFS+~

Reading from and writing to Hadoop DFS is no different from how it is done with other file
systems. The below example reads a file from DFS and writes it to another file on DFS (copy

Hadoop [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
API describes the methods available to user. Let us walk through the code to understand how
it is done.

Create a [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
instance by passing a new Configuration object. 

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);

Given an input/output file name as string, we construct inFile/outFile Path objects. Most
of the [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
APIs accepts [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/Path.html Path] objects.

Path inFile = new Path(argv[0]);
Path outFile = new Path(argv[1]);

Validate the input/output paths before reading/writing.

if (!fs.exists(inFile))
  printAndExit("Input file not found");
if (!fs.isFile(inFile))
  printAndExit("Input should be a file");
if (fs.exists(outFile))
  printAndExit("Output already exists");

Open inFile for reading.

FSDataInputStream in = fs.open(inFile);

Open outFile for writing.

FSDataOutputStream out = fs.create(outFile);

Read from input stream and write to output stream until EOF.

while ((bytesRead = in.read(buffer)) > 0) {
  out.write(buffer, 0, bytesRead);

Close the streams when done.


View raw message