hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-3325) DFSClient writes : DataStreamer thread can be removed
Date Tue, 29 Apr 2008 17:53:56 GMT
DFSClient  writes : DataStreamer thread can be removed

                 Key: HADOOP-3325
                 URL: https://issues.apache.org/jira/browse/HADOOP-3325
             Project: Hadoop Core
          Issue Type: Improvement
          Components: dfs
    Affects Versions: 0.16.0
            Reporter: Raghu Angadi

When a client is writing data to DFS, DFSClient keeps two threads for each file open : 
- DataStreamer thread : writes the data to DataNodes (as 64k packets)
- ResponseProcessor : receives acks from the datanodes and detects related errors.

I think job of DataStreamer can be done inside user's write() (i.e. inside the user thread).
So for normal case, there will be one less thread. When there is an error in the write pipeline,
all the un-acked packets need to be resent. In that case, ResponseProcessor can always create
temporary thread to send these packets.
In the future, the acks for multiple pipelines can be handled by a common thread (at least
in the default case where sockets are non-blocking). 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message