impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Armstrong (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-5500: Reduce catalog update topic size
Date Wed, 28 Jun 2017 00:23:23 GMT
Tim Armstrong has posted comments on this change.

Change subject: IMPALA-5500: Reduce catalog update topic size
......................................................................


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/7268/5/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

Line 358:   RETURN_IF_ERROR(Codec::CreateCompressor(nullptr, false, THdfsCompression::SNAPPY,
> I am hesitant to do that. We need essentially to populate the TopicItem wit
I see your point.

https://github.com/lz4/lz4 reports that it's maybe 50% faster. According to https://www.percona.com/blog/2016/04/13/evaluating-database-compression-methods-update/
maybe 30-40%

We could alternatively just write the size at the start or end of the buffer on both ends
with ReadWriteUtil::PutVLong() or something like that. That's what snappy is doing internally
I think.

This may be overengineering it so I'm fine if you just want to explain the motivation for
snappy.


-- 
To view, visit http://gerrit.cloudera.org:8080/7268
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2f725cd8596205e6101d5b56abf08125faa30b0a
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message