hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3617) Writes from map serialization include redundant checks for accounting space
Date Tue, 24 Jun 2008 04:35:46 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Douglas updated HADOOP-3617:
----------------------------------

    Status: Open  (was: Patch Available)

There remains a (practically unrealizable) possibility of deadlock in this patch. Given the
spill thread:
{noformat}
while (true) { 
  synchronized (spillLock) { 
    spillLock.notify();
    while (kvstart == kvend) { 
      spillLock.wait();
    } 
  } 
  try { 
    sortAndSpill();
  } catch (Throwable e) { 
    sortSpillException = e;
  } finally { 
    synchronized (spillLock) { 
      // adjust indices
    } 
  } 
} 
{noformat}

It's possible to acquire the lock in the finally block, release it, block at the front of
the loop as the other thread acquires the lock and signals before the spill thread can reacquire
the lock at the top of the loop. In practice, the only possibility of deadlock would be from
the collection thread acquiring the lock in a write that cannot be satisfied without spilling-
again- immediately. Still, it needs to be fixed.

What this attempts to effect seems ill-suited to block-level synchronization and will either
require some additional state or more fine-grained locking (i.e. java.util.concurrent.lock).

> Writes from map serialization include redundant checks for accounting space
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-3617
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3617
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.17.0
>            Reporter: Chris Douglas
>            Assignee: Chris Douglas
>             Fix For: 0.19.0
>
>         Attachments: 3617-0.patch, 3617-1.patch
>
>
> In BlockingBuffer::write, the collector verifies that there is sufficient space to hold
metadata when it returns to MapOutputBuffer::collect. This check is redundant for serialized
records that make more than one call to write.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message