quickstep-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hbdeshmukh <...@git.apache.org>
Subject [GitHub] incubator-quickstep pull request #14: Long lived Foreman thread and introduc...
Date Thu, 09 Jun 2016 01:19:25 GMT
GitHub user hbdeshmukh opened a pull request:

    https://github.com/apache/incubator-quickstep/pull/14

    Long lived Foreman thread and introduction of the PolicyEnforcer

    This PR introduces basic skeleton of PolicyEnforcer and makes the Foreman thread understand
concurrent queries, as well as allows the Foreman thread to live throughout the lifetime of
the Quickstep process. Some key highlights are described below: 
    
    - Foreman (C++ class) is now primarily responsible for dispatching work orders to workers
and collect messages coming from worker threads during query execution. 
    
    - Foreman thread now doesn't die after receipt of a new query, unlike earlier. Instead
it stays alive throughout the lifetime of Quickstep process. 
    
    - The main thread is now capable of sending and receiving messages through TMB. 
    
    - A new query sent via command line is sent from the main thread to the Foreman thread
via a TMB message. 
    
    - Creation of a sekeleton PolicyEnforcer class.
    
    - PolicyEnforcer can talk to multiple QueryManager instances to collect WorkOrders to
be dispatched. 
    
    - PolicyEnforcer receives messages from Foreman, figures out the recipient of the message
(using query ID in the message) and relays it to the appropriate QueryManager instance. 
    
    - PolicyEnforcer can also perform admission control. 
    
    - Moving some commonly used code in utility class. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apache/incubator-quickstep query-manager-used-in-foreman

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-quickstep/pull/14.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14
    
----
commit 3afd59163769f059f9ff5586bb7f524b1c387998
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-09T20:08:40Z

    Created QueryManager class and tests.
    
    - Responsible for managing execution state, generation of work orders of
      the query and providing work orders for execution.
    - Tests for QueryManager, which are simplified versions of the Foreman
      tests.

commit 72ab1219cd5ba45612bb19d21d157800b69f24bd
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-09T20:12:40Z

    Merge branch 'master' into query-manager

commit d63e53c6da9c4b3af652487aed59a9b98aebd50a
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-11T12:40:22Z

    WorkerMessage enum type change in Foreman.

commit 5ec8fdd138553191bfc41e1bc391567de2b1a520
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-11T12:41:03Z

    Merge branch 'master' into query-manager

commit 9fb7b1e2fa7ee75c9849ce9aa5b1aed359243a30
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-11T18:39:02Z

    Replaced DCHECK_NOTNULL with DCHECK.

commit ff28cd5f7550eda126420bf00b1aa093091697d8
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-11T18:39:44Z

    Merge branch 'master' into query-manager

commit 4a4273b31412414fe415cf9a11d9a04f1d33c826
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T15:44:32Z

    Query ID can be set for every WorkOrder.

commit 6e680d7ffe2e5856cefabbe2a88a0849465a57de
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T16:19:10Z

    Query ID can now be added to WorkOrders in WorkOrdersContainer.

commit 5e8efab4879151222c0379e2dd89040c411dd6e8
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T16:26:11Z

    Added query ID field in Work order completion message.
    
    - Set before the worker sends the completion message.

commit 0874004f6dca917201aee4bef269ac25d2010037
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T16:27:30Z

    Merge branch 'master' into query-manager

commit 15291f61126c9873f322ae17a9533bc29a80f5ce
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T16:28:15Z

    Merge branch 'query-manager' into query-manager-used-in-foreman

commit 43a82d9210dec0ca4d5a7427f6855ae1e77dbf85
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-12T16:36:14Z

    QueryManager unittest fix.

commit 90e3c105ae9b25904c77fd4f39a72a492377215b
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-13T02:32:57Z

    Added query ID field to messages, query context and InsertDestination.
    
    - WorkOrderCompletion and DataPipelineMessage have query ID field.
    - InsertDestination proto and query context have query ID field.
    - ExecutionGenerator sets the query ID in InsertDestination.

commit f47ce7667be57fc6c60f6c8de5730ec25d5b6aa6
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-13T21:58:32Z

    Added query ID to WorkOrdersAvailable message.

commit 8833964178bf39fb91f9d10d1cc901cd6fe90a1e
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-13T22:11:32Z

    Added query ID to CatalogRelationNewBlockMessage.

commit 509c72a8e22e90740a4d4dbbf21b1d8686a79834
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-15T20:30:50Z

    Restored building of Foreman unittest.
    
    - Minor fixes.

commit c2bedc6b490f5bfc0586b4011c764877b45ae858
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-15T20:39:18Z

    Minor style fix in CMakeLists.

commit 0964a5569a7e7d4fdf452bd4f5899cb5d1b2d412
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-21T00:14:39Z

    Merge branch 'query-manager' into query-manager-used-in-foreman
    
    Conflicts:
    	query_execution/WorkOrdersContainer.hpp

commit 65e225ffa8c190b5e1efa8d6727a02711ea37849
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-21T01:23:45Z

    Merge branch 'master' into query-manager-used-in-foreman
    
    Conflicts:
    	query_execution/QueryManager.cpp
    	query_execution/WorkOrdersContainer.hpp
    	query_execution/tests/QueryManager_unittest.cpp

commit 6ed0d58a6803f4982807fe96675a3ed4facc40e6
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-21T13:11:27Z

    Created PolicyEnforcer class.
    
    - First point of entry for queries in the scheduler.
    - Can perform admission control.
    - Can talk to the QueryManagers of the active queries to provide them
      messages to process and collect work orders for execution from them.

commit b6ea8b130e9e1410174ec904296b7ebc8e1e3cb1
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-21T15:20:03Z

    Merge branch 'master' into query-manager-used-in-foreman
    
    Conflicts:
    	query_optimizer/ExecutionGenerator.hpp

commit d13598defd115ed46e80c14d4f960756383645e9
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-21T15:35:16Z

    Support for admitting multiple queries to the PolicyEnforcer.
    
    - Function to remove a query made public.

commit 39e758b936488f9ef13128e306dc0a781601bb0c
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-22T01:39:44Z

    Added a new class for sending queries to the Foreman.

commit 0c8f0b3d672384f700d65a572f9aaacdc6bdbfd9
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-24T00:37:07Z

    Merge branch 'master' into query-manager-used-in-foreman

commit 38ed241f7a524417477252feaac39482c17cc146
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-24T01:06:08Z

    Minor error fix in PolicyEnforcer

commit 0da3b76df1d4907fb96fdd8f5e9a73d135f21c25
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-25T04:30:13Z

    Long lived Foreman thread
    
    - Foreman thread lives through the lifetime of the Quickstep process.
    - Foreman and main thread communicate through TMB messages.
    - Foreman admits queries and routes them to PolicyEnforcer.
    - Foreman relays messages to policy enforcer which in turn processes it,
      based on the query ID of the message.
    - All the tests modified accordingly.

commit dd62b80efab2f97c96cc118c87754c2a640cba73
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-25T20:33:53Z

    IWYU and lint fixes.

commit 24c93ca5a11e453d45821bbe64e7946ca02948e5
Author: Harshad Deshmukh <harshad@cs.wisc.edu>
Date:   2016-04-29T21:05:16Z

    Code review changes.

commit 5ea898b228df3a2166ed82d242f0a50a78f1c290
Author: Zuyu ZHANG <zuyu@users.noreply.github.com>
Date:   2016-04-30T04:11:23Z

    Refactored HashJoinWorkOrder protos. (#203)

commit 89acf230e85f5b3d1bdf01bb5decdf76f6dec128
Author: Adalbert Gerald Soosai Raj <adalbert.gerald@gmail.com>
Date:   2016-05-01T20:38:38Z

    Partition aware selection using NUMA-awareness (#175)
    
    Link: https://github.com/pivotalsoftware/quickstep/pull/175

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message