Mailing-List: contact notifications-help@accumulo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: jira@apache.org
Date: Tue, 12 Feb 2013 22:41:14 +0000 (UTC)
From: "Keith Turner (JIRA)" <jira@apache.org>
To: notifications@accumulo.apache.org
Message-ID: <JIRA.12632105.1360708194721.280821.1360708874886@arcas>
In-Reply-To: <JIRA.12632105.1360708194721@arcas>
References: <JIRA.12632105.1360708194721@arcas>
Subject: [jira] [Commented] (ACCUMULO-1062) large numbers of threads make
 in-memory updates slow
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/ACCUMULO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577122#comment-13577122 ] 

Keith Turner commented on ACCUMULO-1062:
----------------------------------------

Eric and I were poking at this issue using the test program.  We found the following code in InMemoryMap is causing the performance  problem.

{code:java}
      synchronized (this) {
        // Can not update mutationCount while writes that started before
        // are in progress, this would cause partial mutations to be seen.
        // Also, can not continue until mutation count is updated, because
        // a read may not see a successful write. Therefore writes must
        // wait for writes that started before to finish.
        
        while (kvCount.get() != kv - 1) {
          try {
            wait();
          } catch (InterruptedException ex) {
            // ignored
          }
        }
        kvCount.set(kv + numKVs - 1);
        notifyAll();
      }
{code}
                
> large numbers of threads make in-memory updates slow
> ----------------------------------------------------
>
>                 Key: ACCUMULO-1062
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-1062
>             Project: Accumulo
>          Issue Type: Bug
>          Components: tserver
>    Affects Versions: 1.4.2
>         Environment: testing on large clusters
>            Reporter: Eric Newton
>            Assignee: Keith Turner
>            Priority: Critical
>             Fix For: 1.5.0
>
>         Attachments: SlamTest.java
>
>
> Using hundreds of ingest programs, each running several batch writers, ingest would eventually become slow.  Testing (attached) proved that performance fell off as the number of threads increased.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira