lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Smiley (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-12338) Replay buffering tlog in parallel
Date Tue, 29 May 2018 20:00:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-12338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494162#comment-16494162
] 

David Smiley commented on SOLR-12338:
-------------------------------------

Please consider this alternative utility class:

{code:java}
/** A set of locks by a key {@code T}, kind of like Google Striped but the keys are sparse/lazy.
*/
private static class SparseStripedLock<T> {
  private final Semaphore sizeSemaphore;
  private ConcurrentHashMap<T, CountDownLatch> map = new ConcurrentHashMap<>();

  SparseStripedLock(int maxSize) {
    this.sizeSemaphore = new Semaphore(maxSize);
  }

  void add(T t) throws InterruptedException {
    if (t != null) {
      CountDownLatch myLock = new CountDownLatch(1);
      while (true) {
        CountDownLatch existingLock = map.putIfAbsent(t, myLock); // returns null if no existing
        if (existingLock == null) {
          break;// myLock was successfully inserted (and is pre-locked) already locked, was
successfully inserted
        }
        existingLock.await();// wait for existing lock/permit to become available (see remove()
below)
        // we will most likely exit in next loop, though if contended then possibly not
      }
    }

    // won the lock
    sizeSemaphore.acquire();  //nocommit do at start of add()?
  }

  void remove(T t) {
    if (t != null) {
      map.remove(t).countDown(); // remove and signal to any "await"-ers
    }
    
    sizeSemaphore.release();
  }
}
{code}

Notice the comments, the loop, the new name, use of CountDownLatch, and the one nocommit/question.

> Replay buffering tlog in parallel
> ---------------------------------
>
>                 Key: SOLR-12338
>                 URL: https://issues.apache.org/jira/browse/SOLR-12338
>             Project: Solr
>          Issue Type: Improvement
>      Security Level: Public(Default Security Level. Issues are Public) 
>            Reporter: Cao Manh Dat
>            Assignee: Cao Manh Dat
>            Priority: Major
>         Attachments: SOLR-12338.patch, SOLR-12338.patch, SOLR-12338.patch, SOLR-12338.patch,
SOLR-12338.patch
>
>
> Since updates with different id are independent, therefore it is safe to replay them
in parallel. This will significantly reduce recovering time of replicas in high load indexing
environment. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message