camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martha Obrinteschi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CAMEL-7318) Concurrency on HDFS Consumer not working efficiently
Date Fri, 21 Mar 2014 09:09:42 GMT
Martha Obrinteschi created CAMEL-7318:
-----------------------------------------

             Summary: Concurrency on HDFS Consumer not working efficiently
                 Key: CAMEL-7318
                 URL: https://issues.apache.org/jira/browse/CAMEL-7318
             Project: Camel
          Issue Type: Bug
          Components: camel-hdfs
    Affects Versions: 2.11.2
            Reporter: Martha Obrinteschi
            Priority: Minor


If we have two HDFS consumers the files are being processed twice (once by each consumer,
waiting one after the other) so the consumers are not working in parallel. If we add this
the consumers will work as a team and the transfer will go faster.

This happens because there is no exception thrown (the rename method just returns true or
false and everything carries on as nothing would have happened :).

In order to fix this we could add in the HdfsInputStream: 49

boolean tf = info.getFileSystem().rename(new Path(ret.actualPath), new Path(ret.suffixedPath));
if (!tf) {
	throw new IOException("This exception is thrown because the rename did not succeded.");
	}

And also in the HdfsConsumer: 150
try {
     this.rwlock.writeLock().lock();
     this.istream = HdfsInputStream.createInputStream(fileStatuses[i].getPath().toString(),
this.config);
} catch (IOException ioe) {
     log.info(ioe.getMessage()	+ " If the rename fails we move on to the next file.");
     continue;
}

  





--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message