impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Volker (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4223: Handle truncated file read from HDFS cache
Date Tue, 25 Oct 2016 19:20:00 GMT
Lars Volker has posted comments on this change.

Change subject: IMPALA-4223: Handle truncated file read from HDFS cache
......................................................................


Patch Set 1:

(1 comment)

> (1 comment)
 > 
 > Can we test this by having the test load metadata and then truncate
 > a cached file?

We can try to do this in a custom cluster test. It needs to follow the steps outlined here:
https://issues.cloudera.org/browse/IMPALA-4223?focusedCommentId=212776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-212776

They require changes to the system limits to allow for larger cached files, and to the data
nodes to increase the file cache as well, so they might be rather disruptive. Once these are
changed we can write a custom cluster test to download, truncate, and upload files and run
queries over them, checking that the correct log messages appear.

Should we break this out into several Jiras / changes? The limits should be change in impala-setup,
too. The datanode settings change will be required to integrate this into fuzz testing, too.

I will also have a look at the scanner fuzz test and see if it is easy to inject these error
there.

http://gerrit.cloudera.org:8080/#/c/4828/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:

PS1, Line 448: disk
> how about "uncached" instead?
I followed the error from L433, assuming that since we already report a similar case it made
sense to be consistent. Should I change both?


-- 
To view, visit http://gerrit.cloudera.org:8080/4828
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id1e1fdb0211819c5938956abb13b512350a46f1a
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message