drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yuliya Feldman" <yufeldma...@gmail.com>
Subject Re: Review Request 31107: Ability to make PartitionSender multithreaded - useful in case of LocalExchange being enabled, as it allows to deal with high volume of incoming data
Date Fri, 27 Feb 2015 03:19:42 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31107/
-----------------------------------------------------------

(Updated Feb. 26, 2015, 7:19 p.m.)


Review request for drill, Chris Westin, Jacques Nadeau, Steven Phillips, and Venki Korukanti.


Changes
-------

Added calculation of number of threads based on the cost (number of rows).

Formula is:  cost/slicetarget/#senders/threadfactor

threadfactor is set to 4 by default

Additional config param is max number of threads - by default set to 32

Will need to play around with those params to figure out good combination


Bugs: DRILL-2210
    https://issues.apache.org/jira/browse/DRILL-2210


Repository: drill-git


Description
-------

In addition to description

Fixed few classes that did not handle multithreading well
Added/Changed some Stats behavior to allow stats merge from multiple threads, since again
this class is not suitable to be used in multithreaded environment
Introduced new decorator class to handle multi thrteading (or not)  to minimize changes to
ParitionSenderRootExec class


Diffs (updated)
-----

  exec/java-exec/src/main/java/org/apache/drill/exec/compile/CodeCompiler.java 7cc350e 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/FragmentContext.java e413921 
  exec/java-exec/src/main/java/org/apache/drill/exec/ops/OperatorStats.java 0e9da0e 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/config/IteratorValidator.java
64cf7c5 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/SendingAccountor.java 7af7b65

  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionSenderRootExec.java
f09acaa 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/Partitioner.java
5ed9c39 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerDecorator.java
PRE-CREATION 
  exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/partitionsender/PartitionerTemplate.java
4292c09 
  exec/java-exec/src/main/java/org/apache/drill/exec/planner/fragment/Materializer.java 961b603

  exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/PlannerSettings.java
faa8546 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/DrillbitContext.java 83a89df 
  exec/java-exec/src/main/java/org/apache/drill/exec/server/options/SystemOptionManager.java
aa0a5ad 
  exec/java-exec/src/main/java/org/apache/drill/exec/work/WorkManager.java 99c6ab8 
  exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/TestOptiqPlans.java 4aaaa78


Diff: https://reviews.apache.org/r/31107/diff/


Testing
-------

Still need to provide Unit Tests.

Functional tests are passing

Performance tests were run and look promising for some queries


Thanks,

Yuliya Feldman


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message