impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiaomeng Zhang (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-8632: Add support for self-event detection for insert events
Date Wed, 08 Apr 2020 04:48:22 GMT
Xiaomeng Zhang has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/15648 )

Change subject: IMPALA-8632: Add support for self-event detection for insert events
......................................................................

IMPALA-8632: Add support for self-event detection for insert events

In case of INSERT_EVENTS if Impala inserts into a table it causes a
refresh to the underlying table/partition. This could be unnecessary
when there is only one Impala cluster in the system.
We can detect a self-event in such cases when the HMS API to fire a
listener event returns the event id. This is used by EventProcessor
to ignore the event when it is fetched later in the next polling cycle.

Testing:
Add testInsertFromImpala() in MetastoreEventsProcessorTest.java to test
insert event self-event detection when insert into table and partition.

Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf
---
M be/src/common/global-flags.cc
M be/src/util/backend-gflag-util.cc
M bin/impala-config.sh
M common/thrift/BackendGflags.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Db.java
M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/events/InFlightEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M fe/src/main/java/org/apache/impala/catalog/events/SelfEventContext.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
M tests/custom_cluster/test_event_processing.py
18 files changed, 487 insertions(+), 171 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/15648/5
-- 
To view, visit http://gerrit.cloudera.org:8080/15648
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7873fbb2c159343690f93b9d120f6b425b983dcf
Gerrit-Change-Number: 15648
Gerrit-PatchSet: 5
Gerrit-Owner: Xiaomeng Zhang <xiaomeng@cloudera.com>
Gerrit-Reviewer: Anurag Mantripragada <anurag@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
Gerrit-Reviewer: Vihang Karajgaonkar <vihang@cloudera.com>
Gerrit-Reviewer: Xiaomeng Zhang <xiaomeng@cloudera.com>

Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message