accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Elser (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3259) Randomwalk Bulk.xml create queued threads without bound.
Date Fri, 24 Oct 2014 20:44:33 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14183439#comment-14183439
] 

Josh Elser commented on ACCUMULO-3259:
--------------------------------------

Bounded the size of the queue for tasks to 50 times the number of tservers.

> Randomwalk Bulk.xml create queued threads without bound.
> --------------------------------------------------------
>
>                 Key: ACCUMULO-3259
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3259
>             Project: Accumulo
>          Issue Type: Improvement
>          Components: test
>    Affects Versions: 1.5.2, 1.6.1
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>             Fix For: 1.6.2, 1.7.0
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> While looking at ACCUMULO-3257, I saw that Bulk.xml really goes crazy creating new threads
of bulk-imported related work without respect to the actual available resources.
> For example, on a single node after about 5 minutes, I can see a backlog of about 600+
tasks.
> This doesn't really hurt things server-side because the threadpool running these tasks
is bounded, but it unnecessarily increases the load on the test runner. We can estimate the
"throughput" of the system based on the number of tservers available, skip the queueing of
new tasks when we have an excessive number already queued (relative to the number of tservers
available), and then wait a bit before we continue to the next node in Bulk.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message