impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dimitris Tsirogiannis (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-5500: Reduce catalog update topic size
Date Tue, 27 Jun 2017 04:17:03 GMT
Dimitris Tsirogiannis has posted comments on this change.

Change subject: IMPALA-5500: Reduce catalog update topic size
......................................................................


Patch Set 2:

(7 comments)

http://gerrit.cloudera.org:8080/#/c/7268/2/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

Line 330:       if (!status.ok()) {
> take out of if/else since it's common to both cases
Done


Line 367:   uint8_t* compressed_buffer_ptr = &compressed_buffer[0];
> comptessed_buffer.data()
Done


Line 370:   // Base64 encode the compressed catalog object and store it in the topic item.
> Is base64-encoding really necessary? Why don't the raw bytes work?
Hm, not really. It works without it, I only used it because the topic items use strings for
keys and values. Removed it.


http://gerrit.cloudera.org:8080/#/c/7268/2/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:

Line 200:   /// Serializes 'catalog_object' and compresses it using Snappy compression.
> remove second "compression"
Done


http://gerrit.cloudera.org:8080/#/c/7268/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:

Line 1417: Status ImpalaServer::DecompressAndDeserializeCatalogObject(const TTopicItem&
item,
> Why not separate decompression from deserialization? Seem like separate con
Done


http://gerrit.cloudera.org:8080/#/c/7268/2/be/src/util/compress.h
File be/src/util/compress.h:

Line 103:   SnappyCompressor(MemPool* mem_pool = NULL, bool reuse_buffer = false);
> Why these changes? Shouldn't these always be created using CreateCompressor
oops forgot about these. I was playing around with directly creating the compressor. reverted.


http://gerrit.cloudera.org:8080/#/c/7268/2/tests/custom_cluster/test_compact_catalog_updates.py
File tests/custom_cluster/test_compact_catalog_updates.py:

Line 45:           "select count(*) from functional.alltypes")
> Check result? Maybe also use another query on a parquet table.
Added the result check. Out of curiosity, what's so special about the parquet table in this
case?


-- 
To view, visit http://gerrit.cloudera.org:8080/7268
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2f725cd8596205e6101d5b56abf08125faa30b0a
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message