hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Lucene-hadoop Wiki] Update of "HadoopDfsReadWriteExample" by LohitVijayarenu
Date Sun, 04 Nov 2007 19:40:21 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.

The following page has been changed by LohitVijayarenu:
http://wiki.apache.org/lucene-hadoop/HadoopDfsReadWriteExample

New page:
~+Simple Example to Read and Write files from Hadoop DFS+~

Reading from and writing to Hadoop DFS is no different from how it is done with other file
systems. The below example reads a file from DFS and writes it to another file on DFS (copy
command). 

Hadoop [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
API describes the methods available to user. Let us walk through the code to understand how
it is done.

Create a [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
instance by passing a new Configuration object. 

{{{
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
}}}

Given an input/output file name as string, we construct inFile/outFile Path objects. Most
of the [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html FileSystem]
APIs accepts [http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/Path.html Path] objects.

{{{
Path inFile = new Path(argv[0]);
Path outFile = new Path(argv[1]);
}}}

Validate the input/output paths before reading/writing.

{{{
if (!fs.exists(inFile))
  printAndExit("Input file not found");
if (!fs.isFile(inFile))
  printAndExit("Input should be a file");
if (fs.exists(outFile))
  printAndExit("Output already exists");
}}}


Open inFile for reading.

{{{
FSDataInputStream in = fs.open(inFile);
}}}

Open outFile for writing.

{{{
FSDataOutputStream out = fs.create(outFile);
}}}

Read from input stream and write to output stream until EOF.

{{{
while ((bytesRead = in.read(buffer)) > 0) {
  out.write(buffer, 0, bytesRead);
}
}}}

Close the streams when done.

{{{
in.close();
out.close();
}}}

Mime
View raw message