Mailing-List: contact dev-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@impala.incubator.apache.org
Message-Id: <201607202020.u6KKK7jT005876@ip-10-146-233-104.ec2.internal>
Date: Wed, 20 Jul 2016 20:20:06 +0000
From: "Tim Armstrong (Code Review)" <gerrit@cloudera.org>
To: impala-cr@cloudera.com, dev@impala.incubator.apache.org
CC: David Knupp <dknupp@cloudera.com>,
        Taras Bobrovytsky <tbobrovytsky@cloudera.com>,
        Alex Behm <alex.behm@cloudera.com>
Reply-To: tarmstrong@cloudera.com
Subject: =?UTF-8?Q?=5BImpala-CR=5D=28cdh5-trunk=29_IMPALA-3764=3A_fuzz_test_HDFS_scanners=0A?=
In-Reply-To: <gerrit.1466620074000.I50cf43195a7c582caa02c85ae400ea2256fa3a3b@gerrit.cloudera.org>
References: <gerrit.1466620074000.I50cf43195a7c582caa02c85ae400ea2256fa3a3b@gerrit.cloudera.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Content-Disposition: inline
User-Agent: Gerrit/2.12.2
archived-at: Wed, 20 Jul 2016 20:20:15 -0000

Tim Armstrong has uploaded a new patch set (#6).

Change subject: IMPALA-3764: fuzz test HDFS scanners
......................................................................

IMPALA-3764: fuzz test HDFS scanners

This adds a test that performs some simple fuzz testing of HDFS
scanners. It creates a copy of a given HDFS table, with each
file in the table corrupted in a random way: either a single
byte is set to a random value, or the file is truncated to a
random length. It then runs a query that scans the whole table
with several different batch_size settings. I made some effort
to make the failures reproducible by explicitly seeding the
random number generator, and providing a mechanism to override
the seed.

The fuzzer has found crashes resulting from corrupted or truncated
input files for RCFile, SequenceFile, Parquet, and Text LZO so far.
Avro only had a small buffer read overrun detected by ASAN.

Includes fixes for Parquet crashes found by the fuzzer and a
small buffer overrun in Avro.

Initially it is only enabled for Avro, Parquet, and uncompressed
text. As follow-up work we should fix the bugs in the other scanners
and enable the test for them.

We also don't implement abort_on_error=0 correctly in Parquet:
for some file formats, corrupt headers result in the query being
aborted, so an exception will xfail the test.

Testing:
Ran the test with exploration_strategy=exhaustive in a loop locally
with both DEBUG and ASAN builds. It's been running successfully for
a few hours now without hitting any crashes or test failures after
I made the fixes included in the patch. Also ran exhaustive private
build and core ASAN build.

Change-Id: I50cf43195a7c582caa02c85ae400ea2256fa3a3b
---
M be/src/exec/base-sequence-scanner.cc
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/parquet-column-readers.cc
M be/src/exec/parquet-column-readers.h
M be/src/exec/parquet-metadata-utils.cc
M be/src/exec/parquet-metadata-utils.h
M be/src/runtime/disk-io-mgr.cc
A be/src/runtime/scoped-buffer.h
M be/src/util/bit-stream-utils.h
M be/src/util/bit-stream-utils.inline.h
M be/src/util/compress.cc
M be/src/util/dict-encoding.h
M be/src/util/dict-test.cc
M be/src/util/rle-encoding.h
M be/src/util/rle-test.cc
M testdata/workloads/functional-query/queries/QueryTest/parquet.test
M tests/common/impala_test_suite.py
M tests/query_test/test_scanners.py
A tests/query_test/test_scanners_fuzz.py
19 files changed, 385 insertions(+), 47 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/48/3448/6
-- 
To view, visit http://gerrit.cloudera.org:8080/3448
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I50cf43195a7c582caa02c85ae400ea2256fa3a3b
Gerrit-PatchSet: 6
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-Reviewer: Alex Behm <alex.behm@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>