drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Westin" <chriswesti...@gmail.com>
Subject Review Request 31023: DRILL-2245: improve drillbit resiliency (plus other cleanup)
Date Fri, 13 Feb 2015 22:28:24 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31023/
-----------------------------------------------------------

Review request for drill and Jacques Nadeau.


Bugs: DRILL-2245
    https://issues.apache.org/jira/browse/DRILL-2245


Repository: drill-git


Description
-------

(*) Cleaned up handling of Interrupted Exception, or added
 TODOs in questionable cases.

AutoCloseables
- created org.apache.drill.common.AutoCloseables to handle closing these
  quietly

BaseTestQuery, and derivatives
- factored out pieces into QueryTestUtil so they can be reused

BasicClient
- removed unused variable

BroadcastSenderRootExec
- removed unused imports

ControlHandlerImpl
- member access changes
- formatting

DataResponseHandlerImpl
- formatting

Drillbit
- uses AutoCloseables for the WorkManager and for the storeProvider
- allow start() to take a RemoteServiceSet
- private, final, formatting

DrillbitContext
- minor formatting

DrillClient
- minor cleanup

DrillSystemTestBase
- correct logger instantiation

FilePStore
- vastly improved error message
- a few private renames

Foreman
- does not need to implement Comparable
- does not need to implement Closeable
- thread blocking fixes
- add resultSent flag
- add code to log plan fragments with endpoint assignments
- added finals, cleaned up formatting
- do queue management in acquireQuerySemaphore; local tests pass
- rename getContext() to getQueryContext()
- retain DrillbitContext
- a couple of exception injections for testing
- minor formatting
- TODOs

FragmentContext
- removed unused daemonThreads and related methods
- local cleanup

FragmentData
- add finals and formatting

FragmentExecutor
- eliminated CancelableQuery
- common subexpression elimination
- cleaned up

FragmentStatusListener
- comment out unused logger

JdbcAssert
- removed unused variable
- reused expression within function

LocalClusterCoordinator
- use a map that supports concurrent iteration
- formatting

MaterializedField
- remove unused import

NamedThreadFactory
- make threads daemons
- commented
- general cleanup

Optimizer, BasicOptimizer, IdentityOptimizer
- cleanup

QueryContext
- removed unnecessary functions (with some outside classes tweaked for this)
- finals, formatting

QueryManager
- merge in QueryStatus
  - affects Foreman, ../batch/ControlHandlerImpl,
    and ../../server/rest/ProfileResources
- made some methods private
- removed unused imports
- add finals and formatting
- variable renaming to improve readability
- formatting
- comments
- TODOs

QueryStatus
- getAsInfo() private
- member renaming
- member access changes
- formatting
- TODOs

QueryTestUtil, BaseTestQuery, TestDrillbitResilience
- make maxWidth a parameter to server startup

QueryWorkUnit
- finals
- formatting

RootExec and derived classes
- minor cleanup

RpcBus
- remove unused import

SelfCleaningRunnable
- created org.apache.drill.common.SelfCleaningRunnable

SingleRowListener
- created org.apache.drill.SingleRowListener results listener
- use in TestDrillbitResilience

StatusProvider
- eliminate unused class

TestDrillbitResilience
- created org.apache.drill.exec.server.TestDrillbitResilience to test drillbit
  resilience in the face of exceptions and failures during queries

TestWithZookeeper
- factor out work into ZookeeperHelper so that it can be reused by
  TestDrillbitResilience

UserBitShared
- get rid of unused UNKNOWN_QUERY

UserServer
- remove unused import

WorkEventBus
- rename methods, affects Foreman and ControlHandlerImpl
- remove unused WorkerBee reference
- most members final
- formatting

WorkManager
- Closeable to AutoCloseable
- removed unused incomingFragments Set
- eliminated unnecessary eventThread and pendingTasks by posting Runnables
  directly to executor
- use SelfCleaningRunnable for Foreman management
- FragmentExecutor management uses SelfCleaningRunnable
- runningFragments to be a ConcurrentHashMap; TestTpchDistributed passes
- other improvements due to bee no longer needed in various places
- most members final
- minor formatting
- comments
- TODOs

(*) Created exception injection classes to simulate exceptions for testing
- ExceptionInjection
- ExceptionInjector
- ExceptionInjectionUtil
- TestExceptionInjection


Diffs
-----

  common/src/main/java/org/apache/drill/common/AutoCloseables.java PRE-CREATION 
  common/src/main/java/org/apache/drill/common/SelfCleaningRunnable.java PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/ExecConstants.java 5efcce8 
  exec/java-exec/src/main/java/org/apache/drill/exec/client/DrillClient.java 04b955b 
  exec/java-exec/src/main/java/org/apache/drill/exec/coord/local/LocalClusterCoordinator.java
035c1aa 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/QueryContext.java c881432 
  exec/java-exec/src/main/java/org/apache/drill/exec/opt/BasicOptimizer.java 5288f5d 
  exec/java-exec/src/main/java/org/apache/drill/exec/opt/IdentityOptimizer.java 979c5e2 
  exec/java-exec/src/main/java/org/apache/drill/exec/opt/Optimizer.java 34d0622 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/BaseRootExec.java 412da85

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/RootExec.java a644c34 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SendingAccountor.java 7af7b65

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/broadcastsender/BroadcastSenderRootExec.java
22fa047 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/orderedpartitioner/OrderedPartitionRecordBatch.java
a062074 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/producer/ProducerConsumerBatch.java
4c9b33b 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Fragment.java ac63bde

  exec/java-exec/src/main/java/org/apache/drill/exec/record/MaterializedField.java bcc226f

  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BasicClient.java f358097 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/BasicServer.java c00df4e 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/NamedThreadFactory.java 2b49579 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/ReconnectingConnection.java f214c4d

  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RemoteConnection.java 3a139f8 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/RpcBus.java 96c9911 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/Controller.java 7f84a2b 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/ControllerImpl.java f8f6fd7

  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/control/WorkEventBus.java d6b8637

  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataConnectionCreator.java 197996d

  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataResponseHandlerImpl.java
1fcb3e9 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/data/DataTunnel.java 5aa4aa6 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserClient.java 4e7fc92 
  exec/java-exec/src/main/java/org/apache/drill/exec/rpc/user/UserServer.java dffb9a1 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/Drillbit.java 67342c4 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitContext.java 83a89df 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
aa0a5ad 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/profile/ProfileResources.java
ae04bad 
  exec/java-exec/src/main/java/org/apache/drill/exec/service/ServiceEngine.java ff6e13c 
  exec/java-exec/src/main/java/org/apache/drill/exec/store/sys/local/FilePStore.java baa998d

  exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExceptionInjection.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/testing/ExceptionInjector.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/testing/InjectionSite.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/testing/SimulatedExceptions.java PRE-CREATION

  exec/java-exec/src/main/java/org/apache/drill/exec/work/CancelableQuery.java 5b11943 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/QueryWorkUnit.java 9743d6e 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/StatusProvider.java 6086f74 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/ControlHandlerImpl.java 3228da9

  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/SpoolingRawBatchBuffer.java
6ee93ab 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/batch/UnlimitedRawBatchBuffer.java
895918c 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/DrillbitStatusListener.java
ca52f0c 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/Foreman.java 378e81a 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentData.java 52fd0a9

  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/FragmentStatusListener.java
6a719d2 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryManager.java 2de3592

  exec/java-exec/src/main/java/org/apache/drill/exec/work/foreman/QueryStatus.java 4e18da6

  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/FragmentExecutor.java a8f07b5

  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/NonRootFragmentManager.java
3671804 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/fragment/RootFragmentManager.java
54fc8c4 
  exec/java-exec/src/test/java/org/apache/drill/BaseTestQuery.java 17bcb79 
  exec/java-exec/src/test/java/org/apache/drill/PlanTestBase.java c52545d 
  exec/java-exec/src/test/java/org/apache/drill/QueryTestUtil.java PRE-CREATION 
  exec/java-exec/src/test/java/org/apache/drill/SingleRowListener.java PRE-CREATION 
  exec/java-exec/src/test/java/org/apache/drill/TestBugFixes.java 51f3121 
  exec/java-exec/src/test/java/org/apache/drill/TestBuilder.java 978e565 
  exec/java-exec/src/test/java/org/apache/drill/exec/DrillSystemTestBase.java 75ba3a9 
  exec/java-exec/src/test/java/org/apache/drill/exec/TestWithZookeeper.java bb69c9a 
  exec/java-exec/src/test/java/org/apache/drill/exec/ZookeeperHelper.java PRE-CREATION 
  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/SimpleRootExec.java 0277876

  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java 4aaaa78

  exec/java-exec/src/test/java/org/apache/drill/exec/server/TestDrillbitResilience.java PRE-CREATION

  exec/java-exec/src/test/java/org/apache/drill/exec/testing/ExceptionInjectionUtil.java PRE-CREATION

  exec/java-exec/src/test/java/org/apache/drill/exec/testing/TestExceptionInjection.java PRE-CREATION

  exec/jdbc/src/test/java/org/apache/drill/jdbc/test/JdbcAssert.java b88d880 
  protocol/src/main/protobuf/UserBitShared.proto 8f05c45 

Diff: https://reviews.apache.org/r/31023/diff/


Testing
-------

mvn install
Functional - Passing - New
Advanced - TPCH SF100 - Parquet
Ericsson - functional


Thanks,

Chris Westin


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message