hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4849) Idempotent create and append operations.
Date Tue, 18 Jun 2013 09:02:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13686514#comment-13686514

Konstantin Shvachko commented on HDFS-4849:

Sorry, should correct myself. The code I posted previously works correctly because FSDataOutputStream
keeps track of the offset in the stream. I'll remove the comment to avoid confusion in reading.
It will still be available in the history.
I am still responding to

> As regards creating blocks simultaneously, it is prevented precisely because multiple
creates are not allowed today and hence the application does not even get to allocating blocks.

The snippet below shows that multiple creates are allowed in current code, and the things
break because there is a competition for block allocations not file creates.
  static volatile int nrDone;
  public static void main(String[] args) throws Exception {
    Configuration conf = new HdfsConfiguration();
    conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024);
    MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf).numDataNodes(3).build();
    final FileSystem fs = cluster.getFileSystem();
    final Path path = new Path("testFile.txt");
    nrDone = 0;
    for(int i = 0; i < 10; i++) {
      new Thread() {
        public void run() {
          try {
            final FSDataOutputStream out = fs.create(path);
            for(int bi = 0; bi < 12; bi++)
              out.write(new byte[1024], 0, 1024);
          } catch (IOException e) {
            LOG.error("Write failed: ", e);
    while(nrDone < 10) ;
    LOG.info("Done. " + fs.getFileStatus(path));
    for(BlockLocation bl : fs.getFileBlockLocations(path, 0, Long.MAX_VALUE))
> Idempotent create and append operations.
> ----------------------------------------
>                 Key: HDFS-4849
>                 URL: https://issues.apache.org/jira/browse/HDFS-4849
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 2.0.4-alpha
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>            Priority: Blocker
>         Attachments: idempotentCreate.patch, idempotentCreate.patch, idempotentCreate.patch
> create, append and delete operations can be made idempotent. This will reduce chances
for a job or other app failures when NN fails over.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message