hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajey Shah <ajey.s...@gmail.com>
Subject Re: How do I copy files from my linux file system to HDFS using a java prog?
Date Thu, 08 May 2008 21:25:00 GMT

Thanks Ted. It solves my problem. :)

Ted Dunning-3 wrote:
> 
> 
> I think that file names of the form "file://directory-path" should work to
> give you local file access using this program.
> 
> 
> On 5/6/08 3:34 PM, "Ajey Shah" <ajey.shah@gmail.com> wrote:
> 
>> 
>> Thanks Suresh. But even this program reads and writes from the HDFS. What
>> i
>> need to do is read from my normal local linux harddrive and write to the
>> HDFS.
>> 
>> I'm sorry if I misunderstood your program.
>> 
>> Thanks for replying. :)
>> 
>> 
>> 
>> Babu, Suresh wrote:
>>> 
>>> 
>>> Try this program. Modify the HDFS configuration, if it is different from
>>> the default.
>>> 
>>> import java.io.File;
>>> import java.io.IOException;
>>> 
>>> import org.apache.hadoop.conf.Configuration;
>>> import org.apache.hadoop.fs.FileStatus;
>>> import org.apache.hadoop.fs.FileSystem;
>>> import org.apache.hadoop.fs.FSDataInputStream;
>>> import org.apache.hadoop.fs.FSDataOutputStream;
>>> import org.apache.hadoop.fs.Path;
>>> import org.apache.hadoop.io.IOUtils;
>>> 
>>> public class HadoopDFSFileReadWrite {
>>> 
>>>   static void usage () {
>>>     System.out.println("Usage : HadoopDFSFileReadWrite <inputfile>
>>> <output file>");
>>>     System.exit(1);
>>>   }
>>> 
>>>   static void printAndExit(String str) {
>>>     System.err.println(str);
>>>     System.exit(1);
>>>   }
>>> 
>>>   public static void main (String[] argv) throws IOException {
>>>     Configuration conf = new Configuration();
>>>     conf.set("fs.default.name", "localhost:9000");
>>>     FileSystem fs = FileSystem.get(conf);
>>> 
>>>     FileStatus[] fileStatus = fs.listStatus(fs.getHomeDirectory());
>>>     for(FileStatus status : fileStatus) {
>>>         System.out.println("File: " + status.getPath());
>>>     }
>>> 
>>>     if (argv.length != 2)
>>>       usage();
>>> 
>>>     // HadoopDFS deals with Path
>>>     Path inFile = new Path(argv[0]);
>>>     Path outFile = new Path(argv[1]);
>>> 
>>>     // Check if input/output are valid
>>>     if (!fs.exists(inFile))
>>>       printAndExit("Input file not found");
>>>     if (!fs.isFile(inFile))
>>>       printAndExit("Input should be a file");
>>>     if (fs.exists(outFile))
>>>       printAndExit("Output already exists");
>>> 
>>>     // Read from and write to new file
>>>     FSDataInputStream in = fs.open(inFile);
>>>     FSDataOutputStream out = fs.create(outFile);
>>>     byte buffer[] = new byte[256];
>>>     try {
>>>       int bytesRead = 0;
>>>       while ((bytesRead = in.read(buffer)) > 0) {
>>>         out.write(buffer, 0, bytesRead);
>>>       }
>>> 
>>>     } catch (IOException e) {
>>>       System.out.println("Error while copying file");
>>>     } finally {
>>>       in.close();
>>>       out.close();
>>>     }
>>>   }
>>> }
>>> 
>>> Suresh
>>> 
>>> 
>>> -----Original Message-----
>>> From: Ajey Shah [mailto:ajey.shah@gmail.com]
>>> Sent: Thursday, May 01, 2008 3:31 AM
>>> To: core-user@hadoop.apache.org
>>> Subject: How do I copy files from my linux file system to HDFS using a
>>> java prog?
>>> 
>>> 
>>> Hello all,
>>> 
>>> I need to copy files from my linux file system to HDFS in a java program
>>> and not manually. This is the piece of code that I have.
>>> 
>>> try {
>>> 
>>> FileSystem hdfs = FileSystem.get(new
>>> Configuration());
>>> 
>>> LocalFileSystem ls = null;
>>> 
>>> ls = hdfs.getLocal(hdfs.getConf());
>>> 
>>> hdfs.copyFromLocalFile(false, new
>>> Path(fileName), new Path(outputFile));
>>> 
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> }
>>> 
>>> The problem is that it searches for the input path on the HDFS and not
>>> my linux file system.
>>> 
>>> Can someone point out where I may be wrong. I feel it's some
>>> configuration issue but have not been able to figure it out.
>>> 
>>> Thanks.
>>> --
>>> View this message in context:
>>> http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-H
>>> DFS-using-a-java-prog--tp16992491p16992491.html
>>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>> 
>>> 
>>> 
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/How-do-I-copy-files-from-my-linux-file-system-to-HDFS-using-a-java-prog--tp16992491p17136713.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message