From commits-return-22021-archive-asf-public=cust-asf.ponee.io@accumulo.apache.org Thu Aug 2 19:38:12 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 72953180674 for ; Thu, 2 Aug 2018 19:38:11 +0200 (CEST) Received: (qmail 98246 invoked by uid 500); 2 Aug 2018 17:38:10 -0000 Mailing-List: contact commits-help@accumulo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@accumulo.apache.org Delivered-To: mailing list commits@accumulo.apache.org Received: (qmail 98237 invoked by uid 99); 2 Aug 2018 17:38:10 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Aug 2018 17:38:10 +0000 Received: by gitbox.apache.org (ASF Mail Server at gitbox.apache.org, from userid 33) id 0EC9E825A8; Thu, 2 Aug 2018 17:38:10 +0000 (UTC) Date: Thu, 02 Aug 2018 17:38:10 +0000 To: "commits@accumulo.apache.org" Subject: [accumulo-website] branch master updated: Add docs for scan execution hints apache/accumulo#571 (#100) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Message-ID: <153323149000.30470.15870660782786358358@gitbox.apache.org> From: kturner@apache.org X-Git-Host: gitbox.apache.org X-Git-Repo: accumulo-website X-Git-Refname: refs/heads/master X-Git-Reftype: branch X-Git-Oldrev: 8b2982b069a5630596e49f5ae9ba82b518795fc0 X-Git-Newrev: 7b256022d3111bb455997aa3fdf0181e3a0be5bd X-Git-Rev: 7b256022d3111bb455997aa3fdf0181e3a0be5bd X-Git-NotificationType: ref_changed_plus_diff X-Git-Multimail-Version: 1.5.dev Auto-Submitted: auto-generated This is an automated email from the ASF dual-hosted git repository. kturner pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/accumulo-website.git The following commit(s) were added to refs/heads/master by this push: new 7b25602 Add docs for scan execution hints apache/accumulo#571 (#100) 7b25602 is described below commit 7b256022d3111bb455997aa3fdf0181e3a0be5bd Author: Keith Turner AuthorDate: Thu Aug 2 13:38:07 2018 -0400 Add docs for scan execution hints apache/accumulo#571 (#100) --- _docs-2-0/administration/scan-executors.md | 52 ++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 9 deletions(-) diff --git a/_docs-2-0/administration/scan-executors.md b/_docs-2-0/administration/scan-executors.md index b46ffb3..d7cecbd 100644 --- a/_docs-2-0/administration/scan-executors.md +++ b/_docs-2-0/administration/scan-executors.md @@ -14,7 +14,7 @@ data. The long scans noticeably increase the latency of the short scans. Accumulo offers two mechanisms to help improve situations like this: multiple scan executors and per executor prioritizers. Additional scan executors can give tables dedicated resources. For each scan executor, an optional -prioritizer can reorder queued work. +prioritizer can reorder queued work. ### Configuring and using Scan Executors @@ -22,7 +22,7 @@ By default, Accumulo sets `tserver.scan.executors.default.threads=16` which creates the default scan executor. To configure additional scan executors, chose a unique name and configure {% plink tserver.scan.executors.* %}. Setting the following causes each tablet server to create a scan executor with the -specified threads. +specified threads. ``` tserver.scan.executors..threads= @@ -30,7 +30,7 @@ tserver.scan.executors..threads= Optionally, some of the following can be set. The `priority` setting determines thread priority. The `prioritizer` settings specifies a class that -orders pending work. +orders pending work. ``` tserver.scan.executors..priority= @@ -40,7 +40,7 @@ tserver.scan.executors..prioritizer.opts.= After creating an executor, configure {% plink table.scan.dispatcher %} to use it. A dispatcher is Java subclass of {%jlink org.apache.accumulo.core.spi.scan.ScanDispatcher %} -that decides which scan executor should service a table. Set the following table +that decides which scan executor should service a table. Set the following table property to configure a dispatcher. ``` @@ -56,9 +56,9 @@ table.scan.dispatcher.opts.= The default value for `table.scan.dispatcher` is {% jlink org.apache.accumulo.core.spi.scan.SimpleScanDispatcher %}. SimpleScanDispatcher supports an `executor` option for choosing a scan executor. If this option is not set, then SimpleScanDispatcher will dispatch -to the scan executor named `default`. +to the scan executor named `default`. -To to tie everything together, consider the following use case. +To to tie everything together, consider the following use case. * Create tables named LOW1 and LOW2 using a scan executor with a single thread. * Create a table named HIGH with a dedicated scan executor with 8 threads. @@ -81,12 +81,15 @@ config -s tserver.scan.executors.high.threads=8 Tablet servers should be restarted after configuring scan executors, then tables can be configured. ``` +config -t LOW1 -s table.scan.dispatcher=org.apache.accumulo.core.spi.scan.SimpleScanDispatcher config -t LOW1 -s table.scan.dispatcher.opts.executor=low +config -t LOW2 -s table.scan.dispatcher=org.apache.accumulo.core.spi.scan.SimpleScanDispatcher config -t LOW2 -s table.scan.dispatcher.opts.executor=low +config -t HIGH -s table.scan.dispatcher=org.apache.accumulo.core.spi.scan.SimpleScanDispatcher config -t HIGH -s table.scan.dispatcher.opts.executor=high ``` -While not necessary because its the default, it would be safer to also set +While not necessary because its the default, it is safer to also set `table.scan.dispatcher=org.apache.accumulo.core.spi.scan.SimpleScanDispatcher` for each table. This ensures things work as expected in the case where `table.scan.dispatcher` was set at the system or namespace level. @@ -104,11 +107,42 @@ executor. ``` tserver.scan.executors.default.prioritizer=org.apache.accumulo.core.spi.scan.IdleRatioScanPrioritizer -``` +``` Using the IdleRatioScanPrioritizer in a test with 50 long running scans and 5 threads repeatedly doing small random lookups made a significant difference. -In this test the average lookup time for the 5 threads went from 250ms to 5 ms. +In this test the average lookup time for the 5 threads went from 250ms to 5 ms. + +### Providing hints from the client side. + +Scanners can provide hints to ScanDispatchers and ScanPriotizers by calling +[setExecutionHints] on the Scanner. What, if anything, is done with these +hints depends on what is configured for the table and system. Accumulo's +default configuration ignores hints. The following shell commands make it +possible to choose an executor and set priorities from a scanner for the +table `tex`. + +``` +config -s tserver.scan.executors.special.threads=8 +config -s tserver.scan.executors.special.prioritizer=org.apache.accumulo.core.spi.scan.HintScanPrioritizer +createtable tex +config -t tex -s table.scan.dispatcher=org.apache.accumulo.core.spi.scan.SimpleScanDispatcher +config -t tex -s table.scan.dispatcher.opts.heed_hints=true +``` + +The {% jlink org.apache.accumulo.core.spi.scan.HintScanPrioritizer %} honors +hints of the form `priority=` to prioritize scans, with lower integers +resulting in a higher priority. The `SimpleScanDispatcher`, which is the +default dispatcher, supports the `heed_hints` option. By default the +`SimpleScanDispatcher` ignores hints, but when `heed_hints` is set to `true` it +will honor hints of the form `executor=` when choosing an +executor. After restarting tservers, the following command will start a scan +that uses the executor `special` with a priority of 3. + +``` +scan -t tex --execution-hints priority=3,executor=special +``` [tserver]: {{ page.docs_baseurl }}/getting-started/design#tablet-server-1 +[setExecutionHints]: {% jurl org.apache.accumulo.core.client.ScannerBase#setExecutionHints-java.util.Map- %}