impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dan Hecht (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4223: Handle truncated file read from HDFS cache
Date Tue, 25 Oct 2016 20:29:33 GMT
Dan Hecht has posted comments on this change.

Change subject: IMPALA-4223: Handle truncated file read from HDFS cache
......................................................................


Patch Set 1:

(1 comment)

> (1 comment)
 > 
 > > (1 comment)
 > >
 > > Can we test this by having the test load metadata and then
 > truncate
 > > a cached file?
 > 
 > We can try to do this in a custom cluster test. It needs to follow
 > the steps outlined here: https://issues.cloudera.org/browse/IMPALA-4223?focusedCommentId=212776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-212776
 > 
 > They require changes to the system limits to allow for larger
 > cached files, and to the data nodes to increase the file cache as
 > well, so they might be rather disruptive. Once these are changed we
 > can write a custom cluster test to download, truncate, and upload
 > files and run queries over them, checking that the correct log
 > messages appear.
 > 
 > Should we break this out into several Jiras / changes? The limits
 > should be change in impala-setup, too. The datanode settings change
 > will be required to integrate this into fuzz testing, too.
 > 
 > I will also have a look at the scanner fuzz test and see if it is
 > easy to inject these error there.

Yes, if it's a good amount of work to set up that kind of test, let's save it for another
step.  It's probably better to spend time with getting the fuzzer working rather than this
one particular test case.

http://gerrit.cloudera.org:8080/#/c/4828/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:

PS1, Line 448: disk
> I followed the error from L433, assuming that since we already report a sim
Okay. I'm fine either leaving this one as "disk" or changing both.


-- 
To view, visit http://gerrit.cloudera.org:8080/4828
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id1e1fdb0211819c5938956abb13b512350a46f1a
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message