impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Tauber-Marshall (Code Review)" <>
Subject [Impala-ASF-CR] IMPALA-3742: partitions DMLs for Kudu tables
Date Sat, 18 Mar 2017 00:21:33 GMT
Thomas Tauber-Marshall has uploaded a new patch set (#9).

Change subject: IMPALA-3742: partitions DMLs for Kudu tables

IMPALA-3742: partitions DMLs for Kudu tables

are currently painful because we just send rows randomly,
which creates a lot of work for Kudu since it partitions
and sorts data before writing, causing writes to be slow.

We can alleviate this by sending the rows to Kudu already
partitioned and sorted. This patch partitions the rows
according to Kudu's partitioning scheme. A followup patch
will deal with sorting.

It accomplishes this by inserting an exchange node into the
plan before the DML operation. The DataStreamSender then uses
a new abstraction, DataStreamPartitioner, that calls into the
Kudu client to determine the partition for each row.

- Updated planner tests.
- Manually verified the partitioning works as expected.

Change-Id: Ic10b3295159354888efcde3df76b0edb24161515
M be/src/exec/
M be/src/exec/
M be/src/exec/kudu-util.h
M be/src/runtime/CMakeLists.txt
M be/src/runtime/
A be/src/runtime/
A be/src/runtime/data-stream-partitioner.h
M be/src/runtime/
M be/src/runtime/data-stream-sender.h
M be/src/scheduling/
M bin/
M common/thrift/Partitions.thrift
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/analysis/
M fe/src/main/java/org/apache/impala/catalog/
M fe/src/main/java/org/apache/impala/planner/
M fe/src/main/java/org/apache/impala/planner/
M fe/src/main/java/org/apache/impala/planner/
M fe/src/main/java/org/apache/impala/planner/
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-delete.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-update.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-upsert.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
23 files changed, 588 insertions(+), 104 deletions(-)

  git pull ssh:// refs/changes/37/6037/9
To view, visit
To unsubscribe, visit

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic10b3295159354888efcde3df76b0edb24161515
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <>
Gerrit-Reviewer: Henry Robinson <>
Gerrit-Reviewer: Marcel Kornacker <>
Gerrit-Reviewer: Matthew Jacobs <>
Gerrit-Reviewer: Thomas Tauber-Marshall <>

View raw message