hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Clampffer (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
Date Thu, 29 Sep 2016 19:25:20 GMT
James Clampffer created HDFS-10931:

             Summary: libhdfs++: Fix object lifecycle issues in the BlockReader
                 Key: HDFS-10931
                 URL: https://issues.apache.org/jira/browse/HDFS-10931
             Project: Hadoop HDFS
          Issue Type: Sub-task
            Reporter: James Clampffer
            Assignee: James Clampffer
            Priority: Critical

The BlockReader can work itself into a a state during AckRead (possibly other stages as well)
where the pipeline posts a task for asio with a pointer back into itself, then promptly calls
"delete this" without canceling the asio request.  The asio task finishes and tries to acquire
the lock at the address where the DataNodeConnection used to live - but the DN connection
is no longer valid so it's scribbling on some arbitrary bit of memory.  On some platforms
the underlying address used by the mutex state will be handed out to future mutexes so the
scribble breaks that state and all the locks in that process start misbehaving.

This can be reproduced by using the patch from HDFS-8790 and adding more worker threads +
a lot more reader threads.

I'm going to fix this in two parts:
1) Duct tape + superglue patch to make sure that all top level continuations in the block
reader pipeline hold a shared_ptr to the DataNodeConnection.  Nested continuations also get
a copy of the shared_ptr to make sure the connection is alive.  This at least keeps the connection
alive so that it can keep returning asio::operation_aborted.

2) The continuation stuff needs a lot of work to make sure this type of bug doesn't keep popping
up.  We've already fixed these issues in the RPC code.  This will most likely need to be split
into a few jiras.
- Continuation "framework" can be slimmed down quite a bit, perhaps even removed.  Near zero
documentation + many implied contracts = constant bug chasing.
- Add comments to actually describe what's going on in the networking code.  This bug took
significantly longer than it should have to track down because I hadn't worked on the BlockReader
in a while.
- No more "delete this".
- Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead.  It's unclear
why they were implemented like this in the first place and there's no comments to indicate
that this was intentional.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message