hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Dolgy <sasha.do...@gmail.com>
Subject proper method for writing files to hdfs
Date Sun, 17 May 2009 14:55:19 GMT
The following graphic outlines the architecture for HDFS:

If one is to write a client that adds data into HDFS, it needs to add it
through the Data Node.  Now, from the graphic I am to understand that the
client doesn't communicate with the NameNode, and only the Data Node.

In the examples I've seen and the playing I am doing, I am connecting to the
hdfs url as a configuration parameter before I create a file.  Is this the
incorrect way to create files in HDFS?

    Configuration config = new Configuration();
    String path = "/tmp/i/am/a/path/to/a/file.name"
    Path hdfsPath = new Path(path);
    FileSystem fileSystem = FileSystem.get(config);
    FSDataOutputStream os = fileSystem.create(hdfsPath, false);

Should the client be connecting to a data node to create the file as
indicated in the graphic above?

If connecting to a data node is possible and suggested, where can I find
more details about this process?

Thanks in advance,

Sasha Dolgy

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message