tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-184) Refactor GlobalPlanner and global plan data structure
Date Mon, 16 Sep 2013 11:39:52 GMT

    [ https://issues.apache.org/jira/browse/TAJO-184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768246#comment-13768246
] 

Hudson commented on TAJO-184:
-----------------------------

SUCCESS: Integrated in Tajo-trunk-postcommit #434 (See [https://builds.apache.org/job/Tajo-trunk-postcommit/434/])
TAJO-184: Refactor GlobalPlanner and global plan data structure. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=1b1d1e8c1a6b82ccc5c3ce4daeb9e3daa309cde4)
* tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Column.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/EvalExprNode.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/global/TestGlobalQueryOptimizer.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVisitor.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalOptimizer.java
* tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/ProjectionPushDownRule.java
* tajo-core/tajo-core-backend/src/test/queries/complex_union_2.sql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/IndexScanNode.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/IntersectNode.java
* tajo-core/tajo-core-backend/src/main/resources/webapps/worker/querydetail.jsp
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java
* tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/graph/DirectedGraph.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestNLJoinExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ProjectionNode.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/LocalTajoTestingUtility.java
* CHANGES.txt
* tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/index/TestSingleCSVFileBSTIndex.java
* tajo-core/tajo-core-backend/src/test/queries/complex_union_1.sql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/UnaryNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ExceptNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/BaseAlgebraVisitor.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestGenericDirectedGraph.java
* tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/LogicalRootNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/RelationNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/ipc/protocolrecords/QueryUnitRequest.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/EvalTreeUtil.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalPlanner.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestQueryIdFactory.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanUtil.java
* tajo-core/tajo-core-backend/benchmark/tpch/q4.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/graph/DirectedGraphVisitor.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/plan/global/TestGlobalQueryPlanner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/QueryBlockGraph.java
* tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/TestTupleComparator.java
* tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/statistics/StatisticsUtil.java
* tajo-core/tajo-core-backend/src/test/resources/catalog-default.xml
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TajoAsyncDispatcher.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
* tajo-core/tajo-core-backend/benchmark/tpch/q2.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/CreateTableNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/JoinNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
* tajo-core/tajo-core-backend/benchmark/tpch/q7.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Query.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/LogicalNode.java
* tajo-core/tajo-core-backend/benchmark/tpch/q13.tql
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/dataserver/TestHttpDataServer.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalExecutorVisitor.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/parser/TestSQLAnalyzer.java
* tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestColumn.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PhysicalPlanningException.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/LimitNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/SelectionNode.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestRepartitioner.java
* tajo-core/tajo-core-backend/benchmark/tpch/q8.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/eval/FieldEval.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/UnionNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ReceiveNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryInProgress.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/SortNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/ExecutionBlockCursor.java
* tajo-core/tajo-core-backend/src/test/queries/select_13.hiveql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/graph/SimpleDirectedGraph.java
* tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestSchema.java
* tajo-core/tajo-core-backend/benchmark/tpch/q12.tql
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/TestTajoIds.java
* tajo-core/tajo-core-backend/benchmark/tpch/q9.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/GlobalEngine.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Target.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/query/ResultSetMetaDataImpl.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/InsertNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ScanNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/TaskAttemptContext.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/GroupbyNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/join/JoinTree.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/NodeType.java
* tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestResultSetImpl.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/QueryContext.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/rewrite/FilterPushDownRule.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/global/TestMasterPlan.java
* tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestTableSubQuery.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java
* tajo-core/tajo-core-backend/benchmark/tpch/q14.tql
* tajo-common/src/main/java/org/apache/tajo/QueryIdFactory.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/query/QueryUnitRequestImpl.java
* tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/Schema.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/ExplainLogicalPlanVisitor.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/DropTableNode.java
* tajo-core/tajo-core-backend/benchmark/tpch/q17.tql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlannerUtil.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/TableSubQueryNode.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/BasicPhysicalExecutorVisitor.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/BinaryNode.java
* tajo-core/tajo-core-backend/src/test/queries/table_subquery2.sql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/BasicLogicalPlanVisitor.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/Projector.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/ExecutionBlock.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/AlgebraVisitor.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/SendNode.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/TaskSchedulerImpl.java
* tajo-common/src/main/java/org/apache/tajo/ExecutionBlockId.java
* tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
* tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java
* tajo-core/tajo-core-backend/src/test/queries/set_4.sql
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/DataChannel.java
* tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PlanString.java

                
> Refactor GlobalPlanner and global plan data structure
> -----------------------------------------------------
>
>                 Key: TAJO-184
>                 URL: https://issues.apache.org/jira/browse/TAJO-184
>             Project: Tajo
>          Issue Type: Improvement
>          Components: master, physical operator, planner/optimizer
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
>             Fix For: 0.2-incubating
>
>         Attachments: TAJO-184_2.patch, TAJO-184.patch
>
>
> Above all, I'm sorry for submitting a big patch. This patch modifies and refactors broadly
global planning, logical planning, and physical planning parts. It was hard to separate this
issue into smaller issues.
> Especially, this patch primarily rewrites GlobalPlanner and MasterPlan (global plan data
structure) as follows:
>  * Removed GlobalPlanOptimizer
>  * Added DirectedGraph interface, SimpleDirectedGraph concret class, and a visitor class
to visit a graph in post-order traverse way.
>  * Improved MasterPlan by using new graph API
>   ** query block graphs and an execution block graph are represented by SimpleDirectedGraph.
>   ** Now, we can traverse above graphs easily by using graph APIs.
>   ** Added DataChannel class to represent a data flow between execution blocks.
>  * MasterPlan.toString() prints a text graph to represent relationships among execution
blocks and a distributed plan.
>  * Add more sophisticated explain feature for a distributed plan and logical plan. It
is very useful for plan debugging.
>  * Now, the limit operator is pushed down to child execution block.
>   ** So, the intermediate data volume of a sort query with limit is reduced significantly.
>  * TableSubQuery (inline view) is supported. It follows SQL standards. So, you can do
a query as follows:
> {code}
> SELECT *
> FROM
> (
>     SELECT
>         l_orderkey,
>         l_partkey,
>         url
>     FROM
>         (
>           SELECT
>             l_orderkey,
>             l_partkey,
>             CASE
>               WHEN
>                 l_partkey IS NOT NULL THEN ''
>               WHEN l_orderkey = 1 THEN '1'
>             ELSE
>               '2'
>             END AS url
>           FROM
>             lineitem
>         ) res1
>         JOIN
>         (
>           SELECT
>             *
>           FROM
>             part
>         ) res2
>         ON l_partkey = p_partkey
> ) result
> {code}
> In addition, I've refactored as follows:
>  * Column has a qualifier name.
>  * Improved Schema to deal with qualified column names
>  * When a TableDesc instance is retrieved, it is forced to have qualifier columns.
>  * Fixed TAJO-162 bug.
>  * Lots of trivial improvement and refactors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message