impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Skye Wanderman-Milne (Code Review)" <ger...@cloudera.org>
Subject [Impala-CR](cdh5-trunk) IMPALA-3092: Avro table un-queriable after add columns without a default value
Date Tue, 01 Mar 2016 03:12:02 GMT
Skye Wanderman-Milne has posted comments on this change.

Change subject: IMPALA-3092: Avro table un-queriable after add columns without a default value
......................................................................


Patch Set 1:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/2361/1//COMMIT_MSG
Commit Message:

Line 8: without a default value
Instead of copying the JIRA title, describe what this patch actually does. e.g.

IMPALA-3092: return NULL for Avro fields missing defaults


Line 11: ****(can someone help fill this?) slots. With this patch
We used to require that any missing fields missing in an Avro file have a corresponding default
value in the table schema.


http://gerrit.cloudera.org:8080/#/c/2361/1/be/src/exec/hdfs-avro-scanner.cc
File be/src/exec/hdfs-avro-scanner.cc:

Line 217:             "WriteDefaultValue() doesn't support default records yet, should have
failed";
You'll hit this DCHECK if there's a missing record with a default value, since WriteDefaultValue()
will return without an error.

I would add the following above the WriteDefaultValue() call:

if (is_avro_complex_type(table_record->schema)) {
  return Status(TErrorCode::AVRO_UNSUPPORTED_DEFAULT_VALUE,
    field_name, avro_type_name(table_record->schema->type));
}

We should add a test case for this if we don't already.


Line 252:   if (!default_value) {
if (default_value == NULL)


Line 253:     RawValue::Write(NULL, avro_header_->template_tuple, slot_desc, NULL);
We usually do template_tuple->SetNull(slot_desc->null_indicator_offset()) directly


http://gerrit.cloudera.org:8080/#/c/2361/1/tests/query_test/test_avro.py
File tests/query_test/test_avro.py:

Line 12: class TestAvroAddColumn(ImpalaTestSuite):
Instead of creating a new test class, let's add to query_test/test_avro_schema_resolution.py.
Check out $IMPALA_HOME/testdata/avro_schema_resolution. In create_table.sql, the schema_resolution_test
table defines default values for every column, and then contains a file with extra columns
and file with missing columns. You should add a new similar table that's the same except it
doesn't contain any default values, and then add queries to avro-schema-resolution.test that
select from this table. Please add a record field to this table as well.


-- 
To view, visit http://gerrit.cloudera.org:8080/2361
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ie86421fc3da51086e566998af317faa62ba9789b
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Huaisi Xu <hxu@cloudera.com>
Gerrit-Reviewer: Skye Wanderman-Milne <skye@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message