accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Created] (ACCUMULO-3259) Randomwalk Bulk.xml create queued threads without bound.
Date Fri, 24 Oct 2014 19:00:35 GMT
Josh Elser created ACCUMULO-3259:
------------------------------------

             Summary: Randomwalk Bulk.xml create queued threads without bound.
                 Key: ACCUMULO-3259
                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3259
             Project: Accumulo
          Issue Type: Improvement
          Components: test
    Affects Versions: 1.6.1, 1.5.2
            Reporter: Josh Elser
            Assignee: Josh Elser
             Fix For: 1.6.2, 1.7.0


While looking at ACCUMULO-3257, I saw that Bulk.xml really goes crazy creating new threads
of bulk-imported related work without respect to the actual available resources.

For example, on a single node after about 5 minutes, I can see a backlog of about 600+ tasks.

This doesn't really hurt things server-side because the threadpool running these tasks is
bounded, but it unnecessarily increases the load on the test runner. We can estimate the "throughput"
of the system based on the number of tservers available, skip the queueing of new tasks when
we have an excessive number already queued (relative to the number of tservers available),
and then wait a bit before we continue to the next node in Bulk.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message