hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wu, Jiang2 " <jiang2...@citi.com>
Subject read a changing hdfs file
Date Tue, 20 Aug 2013 21:36:40 GMT
Hi,

I did some experiments to read a changing hdfs file. It seems that the reading takes a snapshot
at the file opening moment, and will not read any data appended to the file afterwards. It's
different from what happens when reading a changing local file. My code is as follows

                        Configuration conf = new Configuration();
                        InputStream in = null;
                        try {
                                FileSystem fs = FileSystem.get(URI.create("hdfs://MyCluster/"),
                                                conf);
                                in = fs.open(new Path("/tmp/test.txt"));
                                Scanner scanner=new Scanner(in);
                                while(scanner.hasNextLine()){
                                        System.out.println("+++++++++++++++++++++++++++++++
read "+scanner.nextLine());
                                }
                                System.out.println("+++++++++++++++++++++++++++++++ reader
finished ");
                        } catch (IOException e) {
                                // TODO Auto-generated catch block
                                e.printStackTrace();
                        } finally {
                                IOUtils.closeStream(in);
                        }

I'm wondering if this is the designed hdfs reading behavior, or can be changed by using different
API or configuration? What I expect is the same behavior as a local file reading: when a reader
reads a file while another writer is writing to the file, the reader will receive all data
written by the writer.

Thanks,
Jiang



Mime
View raw message