impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huaisi Xu (Code Review)" <>
Subject [Impala-CR](cdh5-trunk) IMPALA-3499: Split catalog update
Date Thu, 09 Jun 2016 04:30:07 GMT
Huaisi Xu has posted comments on this change.

Change subject: IMPALA-3499: Split catalog update

Patch Set 9:


for local testing i just started the cluster and saw everything is "fine".
File be/src/service/frontend.h:

PS9, Line 38: The TUpdateCatalogCacheRequest contains
            :   /// a list of objects that should be added/removed from the Catalog. Returns
a response
            :   /// that contains details such as the new max catalog version.
> Update the comment

PS9, Line 178: 2D_
> I don't think we need to change the name of this jmethod.
File be/src/service/

Line 123: 
> will remove

> With this new patch, this no longer represents the maximum size of a catalo

PS9, Line 1243: Update is split in pieces with size limit to
              :   // be max_catalog_cache_update_size for multiple updates
> nit: Maybe we should just say "An update is split into batches of size MAX.

PS9, Line 1278: LOG(WARNING)
> This should be VLOG(1) or 2.
I have a feeling that VLOG(1) or 2 means no log.. I never saw anyone(including customers)
enable this.. will just remove these to make it cleaner..

PS9, Line 1284: push_back(update_req);
> Isn't this calling the copy constructor of TUpdateCatalogCacheRequest? If s

Line 1291:     update_reqs.push_back(update_req);
> I believe we do, otherwise the size of the current update_req may be close 
Yes. just looks bad..
File be/src/util/jni-util.h:

PS9, Line 275: _2D
> I just made up something randomly.. I know I am poor at naming things.. :-)

PS9, Line 286: jbytearray_2D
> I don't think SetObjectArrayElement() would copy the bytes - the byte buffe
I tested locally by manually corrupting the memory which request_bytes points to after line
285 and frontend sees corrupted update. perhaps these two are all java objects so they are
copied by reference. :)
File fe/src/main/java/com/cloudera/impala/service/

PS9, Line 160: cur_update
> You're in Java world :) maybe catalogUpdate?

PS9, Line 161: incremental_req
> incrementalRequest

PS9, Line 168: req.getUpdated_objects() == null
> if (!req.isSetUpdated_objects())
I tried but I got bad operand type java.util.List<com.cloudera.impala.thrift.TCatalogObject>
for unary operator '!'.

I think that is a null object? no "!" for this object?

PS9, Line 173: if (req.getRemoved_objects() == null) {
> if (!req.isSetRemoved_objects())
see above

To view, visit
To unsubscribe, visit

Gerrit-MessageType: comment
Gerrit-Change-Id: I176db25124a32944f2396ce8aafbed49cac95928
Gerrit-PatchSet: 9
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <>
Gerrit-Reviewer: Dan Hecht <>
Gerrit-Reviewer: Dimitris Tsirogiannis <>
Gerrit-Reviewer: Henry Robinson <>
Gerrit-Reviewer: Huaisi Xu <>
Gerrit-HasComments: Yes

View raw message