impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tianyi Wang (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-5250: Unify decompressor output length semantics
Date Mon, 11 Sep 2017 17:24:51 GMT
Tianyi Wang has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/8030

Change subject: IMPALA-5250: Unify decompressor output_length semantics
......................................................................

IMPALA-5250: Unify decompressor output_length semantics

This patch makes the semantics output_length parameter in
Codec::ProcessBlock to be the same. In existing code different
decompressor treats output_length differently:
1. SnappyDecompressor needs output_length to be greater than or equal
   to actual decompressed length, but it does not set it to actual
   decompressed length after decompression.
2. SnappyBlockDecompressor and Lz4Decompressor require output_length to
   be exactly the same as actual decompressed length, otherwise
   decompression fails.
3. Other decompressors need output_length to be greater than or equal to
   actual decompressed length and will set it to actual decompressed
   length if oversized.
This inconsistency leads to a bug where the error message is
undeterministic when the compressed block is corrupted. This patch makes
all decompressor behave as 3 and adds a testcase checking that
decompressors can handle oversized output buffer correctly.
Lz4Decompressor will use the "safe" version of decompression function
instead of the "fast" version, for the latter is insecure with corrupted
data and requires decompressed length to be known.

Change-Id: Ifd42942b169921a7eb53940c3762bc45bb82a993
---
M be/src/util/decompress-test.cc
M be/src/util/decompress.cc
2 files changed, 31 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/8030/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8030
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifd42942b169921a7eb53940c3762bc45bb82a993
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Tianyi Wang <twang@cloudera.com>

Mime
View raw message