hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Lawlor (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-6778) Deprecate Chore; its a thread per task when we should have one thread to do all tasks
Date Sat, 24 Jan 2015 00:20:35 GMT

     [ https://issues.apache.org/jira/browse/HBASE-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jonathan Lawlor updated HBASE-6778:
-----------------------------------
    Attachment: HBASE_6778_WIP_v2.patch
                AFTER_thread_dump.txt
                BEFORE_thread_dump.txt

Work In Progress Update #2:
I have attached the latest version of the solution to this issue above (HBASE_6778_WIP_v2.patch).
Since the last WIP update, the following changes have been made:
- Made changes in response to the feedback that [~stack] provided
- Added Stoppable support to ScheduledChore (Stoppable support was needed as the same behavior
could not be replicated with cancels alone). What this means is that now an instance of a
Stoppable can be passed into the ScheduledChore during construction and the ScheduledChore
will cancel itself if Stoppable#isStopped() becomes true.
- Added shutdown() to ChoreService to allow for convenient all-in-one cleanup of a ChoreService
- Changed usages of Chore (the implementation we would like to get rid of) into usages of
ScheduledChore
- Expanded the test coverage of ScheduledChore and ChoreService

While converting Chores to ScheduledChores the primary goal was to maintain the same behavior.
Most changes can be understood through the following mapping (Chore usage -> ScheduledChore
usage):
- Chore.interrupt() -> ScheduledChore.cancel(mayInterruptWhileRunning = true)
- Threads.setDaemonThreadRunning(Chore) -> ChoreService.scheduleChore(ScheduledChore)
- Chore.isAlive -> ScheduledChore.isScheduled()
- Chore.getSleeper().skipSleepCycle() -> ScheduledChore.triggerNow()

Important Notes on recent changes:
- There are two chores in the codebase that don't seem to do anything at the moment. The first
is DelayedClosing (defined within ConnectionManager.java). The second is MovedRegionsCleaner
(defined within HRegionServer). While converting them into ScheduledChores I noticed that
when they are used, instances of the chores are created but the threads are never started.
I looked into it a bit and the issue with DelayedClosing is called out in HBASE-11354, but
I couldn't find anything that spoke about MovedRegionsCleaner. As it stands, I have not changed
this behavior (i.e. both chores still don't do anything) but I thought I should raise the
issue regardless.

- With the movement to this new implementation of Chores, it should be understood that we
will only see a reduction in the number of threads in cases where we can schedule many chores
with a common ChoreService object (e.g. schedule 5 chores with a single instance of ChoreService).
Scheduling each chore with a different ChoreService instance will not provide a reduction
in the number of threads since each instance will always have at least one thread in its core
pool (in fact, scheduling each Chore with a separate instance of ChoreService would be equivalent
to the current implementation of Chores since it would be a thread per Chore).

- While it would have been nice to have a single ChoreService that all ScheduledChores could
talk to, this did not seem feasible. Instead, an instance of ChoreService has been created
inside HRegionServer, and "getChoreService()" has been added to the interfaces RegionServerServices
and MasterServices. Most of the places where Chores were used had access to an HRegionServer,
so this allowed many of the existing chores to be scheduled under a common ChoreService and
thus reduced the number of threads. However, in instances where there wasn't access to an
HRegionServer, an instance of ChoreService had to be defined. Note that this is simply equivalent
to the old implementation of Chores (1 thread per chore).

Summary:
- If you observe the BEFORE(Chore) and AFTER(ScheduledChore) thread dumps, you will see that
the movement to the usage of ChoreService has reduced the number of threads used for Chores
from 12 to 3 which is promising. 

- While the thread count may increase (due to a larger Chore load), we are guaranteed that
the new implementation's worst case thread count will never exceed the previous implementation's
thread count.

Any feedback with regards to the patch above would be greatly appreciated.


Thanks



> Deprecate Chore; its a thread per task when we should have one thread to do all tasks
> -------------------------------------------------------------------------------------
>
>                 Key: HBASE-6778
>                 URL: https://issues.apache.org/jira/browse/HBASE-6778
>             Project: HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jonathan Lawlor
>         Attachments: AFTER_thread_dump.txt, BEFORE_thread_dump.txt, HBASE_6778_WIP_v1.patch,
HBASE_6778_WIP_v2.patch, thread_dump_HMaster.local.out
>
>
> Should use something like ScheduledThreadPoolExecutor instead (Elliott said this first
I think; J-D said something similar just now).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message