impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Impala Public Jenkins (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4943: Speed up block md loading for add/recover partition calls.
Date Fri, 21 Apr 2017 20:53:27 GMT
Impala Public Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4943: Speed up block md loading for add/recover partition calls.
......................................................................


IMPALA-4943: Speed up block md loading for add/recover partition calls.

This change makes alter table add/recover partitions calls use the
per directory block metadata loading routines instead of doing it
per file. This is done since these calls always load the entire
partition directory from scratch and there is no advantage in
loading them incrementally on a per-file basis.

Tests: Ran core tests and the metadata benchmark tests.

(I) Improvement: METADATA-BENCHMARKS()
100K-PARTITIONS-1M-FILES-03-RECOVER [text / none / none] (718.62s ->
549.91s [-23.48%])

(I) Improvement: METADATA-BENCHMARKS()
100K-PARTITIONS-1M-FILES-08-ADD-PARTITION [text / none / none] (46.92s
-> 26.20s [-44.15%])

Change-Id: I331f1f090518f317bcd7df069e480edbd8f039f1
Reviewed-on: http://gerrit.cloudera.org:8080/6651
Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
1 file changed, 20 insertions(+), 5 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Bharath Vissapragada: Looks good to me, approved



-- 
To view, visit http://gerrit.cloudera.org:8080/6651
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I331f1f090518f317bcd7df069e480edbd8f039f1
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bharathv@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins

Mime
View raw message