impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Volker (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4223: Handle truncated file read from HDFS cache
Date Fri, 28 Oct 2016 22:52:23 GMT
Lars Volker has posted comments on this change.

Change subject: IMPALA-4223: Handle truncated file read from HDFS cache
......................................................................


Patch Set 1:

(1 comment)

> (1 comment)
 > 
 > > (1 comment)
 > >
 > > > (1 comment)
 > > >
 > > > Can we test this by having the test load metadata and then
 > > truncate
 > > > a cached file?
 > >
 > > We can try to do this in a custom cluster test. It needs to
 > follow
 > > the steps outlined here: https://issues.cloudera.org/browse/IMPALA-4223?focusedCommentId=212776&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-212776
 > >
 > > They require changes to the system limits to allow for larger
 > > cached files, and to the data nodes to increase the file cache as
 > > well, so they might be rather disruptive. Once these are changed
 > we
 > > can write a custom cluster test to download, truncate, and upload
 > > files and run queries over them, checking that the correct log
 > > messages appear.
 > >
 > > Should we break this out into several Jiras / changes? The limits
 > > should be change in impala-setup, too. The datanode settings
 > change
 > > will be required to integrate this into fuzz testing, too.
 > >
 > > I will also have a look at the scanner fuzz test and see if it is
 > > easy to inject these error there.
 > 
 > Yes, if it's a good amount of work to set up that kind of test,
 > let's save it for another step.  It's probably better to spend time
 > with getting the fuzzer working rather than this one particular
 > test case.

I Opened IMPALA-4394 and IMPALA-4395 to track both adding tests for this change and adding
Hdfs caching to the fuzz tests.

http://gerrit.cloudera.org:8080/#/c/4828/1/be/src/runtime/disk-io-mgr-scan-range.cc
File be/src/runtime/disk-io-mgr-scan-range.cc:

PS1, Line 448: disk
> Okay. I'm fine either leaving this one as "disk" or changing both.
Thanks. I'll leave it to keep any tooling that searches for this working.


-- 
To view, visit http://gerrit.cloudera.org:8080/4828
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Id1e1fdb0211819c5938956abb13b512350a46f1a
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dhecht@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <tarmstrong@cloudera.com>
Gerrit-HasComments: Yes

Mime
View raw message