hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10993) Deprioritize long-running scanners
Date Wed, 30 Apr 2014 16:22:18 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985685#comment-13985685
] 

Jonathan Hsieh commented on HBASE-10993:
----------------------------------------

Instead of boolean could we make this a string and have some validation it?  'fifo' and 'deadline'
seems like reasonable names for the two versions.  This would allow for newer queues to be
added without adding a new wonky configs and invalid config patterns.
{code}
+  /** If set to true, uses a priority queue and deprioritize long-running scans */
+  public static final String DEADLINE_CALL_QUEUE_CONF_KEY = "ipc.server.use.deadline.callqueue";
+
{code}

This should have one more consistent tiebreaker -- deadlineA and deadlineB could be equal
(see http://docs.oracle.com/javase/7/docs/api/java/util/Comparator.html).. hm.. revisiting
after reading BoundedPriorityBlockingQueue -- need to put warning on this saying that ties
are explicitly allowed and handled by special queue.
{code}
+    @Override
+    public int compare(CallRunner a, CallRunner b) {
+      RpcServer.Call callA = a.getCall();
+      RpcServer.Call callB = b.getCall();
+      long deadlineA = priority.getDeadline(callA.getHeader(), callA.param);
+      long deadlineB = priority.getDeadline(callB.getHeader(), callB.param);
+      deadlineA = callA.timestamp + Math.min(deadlineA, maxDelay);
+      deadlineB = callB.timestamp + Math.min(deadlineB, maxDelay);
+      return (int)(deadlineA - deadlineB);
+    }
{code}

It isn't how long the scan request lives but how many times it is used (which is captured
by vtime),  right?
{code}
+      // get the 'virtual time' of the scanner, and applies sqrt() to get a
+      // nice curve for the delay. longer a scan request lives the less priority it gets.
+      // The weight is used to have more control on the delay.
{code}

nice, I like how the queue logic is separate from locking behavior of the queue. (would have
been nice to have some javadoc about this pattern on the private static class PriorityQueue<E>
(I thought this was racy until I got to the next section).

nit: two ';'s 
+  // Lock used for all operations
+  private final ReentrantLock lock = new ReentrantLock();;

nit: why this (multiple times). is this an artifact of previous iteration?
+
+    final ReentrantLock lock = this.lock;
+    lock.lock();

this is basically the right idea, but I think the equation is a little off -- shouldn't it
the sum of time each call was waiting as opposed to the more costly 2x penalization of subsequent
calls?
{code}
+      int cumsum = 0;
+      for (int i = 0; i < work.size(); ++i) {
+        LOG.debug("Request i=" + i + " value=" + work.get(i));
+        cumsum += cumsum + work.get(i);
+      }
+      LOG.debug("Time cumulative sum=" + cumsum);
+
+      // -> [small small small huge small large small small]
+      // -> NO REORDER   [10 10 10 100 10 50 10 10] -> 4150 (FIFO Queue)
                                           10 20 30 130 140 190 200 210 -> 930   // this
instead of this sequence
                                           10 30 70 240 490 1030 2070 4150
+      // -> WITH REORDER [10 10 10 10 10 10 50 100] -> 2720 (Deadline Queue)
                                            10 20 30 40 50 60 110 210 -> 530    // this
instead of this sequence.
                                            10 30 70 150 310 630 1310 2720

{code}



> Deprioritize long-running scanners
> ----------------------------------
>
>                 Key: HBASE-10993
>                 URL: https://issues.apache.org/jira/browse/HBASE-10993
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Matteo Bertozzi
>            Assignee: Matteo Bertozzi
>            Priority: Minor
>             Fix For: 0.99.0
>
>         Attachments: HBASE-10993-v0.patch, HBASE-10993-v1.patch, HBASE-10993-v2.patch
>
>
> Currently we have a single call queue that serves all the "normal user"  requests, and
the requests are executed in FIFO.
> When running map-reduce jobs and user-queries on the same machine, we want to prioritize
the user-queries.
> Without changing too much code, and not having the user giving hints, we can add a “vtime”
field to the scanner, to keep track from how long is running. And we can replace the callQueue
with a priorityQueue. In this way we can deprioritize long-running scans, the longer a scan
request lives the less priority it gets.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message