hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9263) tests are using /test/build/data; breaking Jenkins
Date Wed, 21 Oct 2015 21:20:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967948#comment-14967948

Chris Nauroth commented on HDFS-9263:

Hi [~stevel@apache.org].

I'm a little unclear on what this patch is trying to solve.  The various test working dir
properties are all built up from a base of {{project.build.directory}} in hadoop-project/pom.xml.
 I'd expect that to avoid dirtying the workspace by writing files outside of {{target}}. 
It's true that there are too many redundant properties like Andrew pointed out, but I don't
think consolidating those was the original intent of this issue.  Is there a specific test
that you see writing outside of {{target}}?

Or is the issue specifically that this causes a problem only when running outside the Maven

bq. many buildups of test dirs now use something random, rather than a hard-coded path like
"dfs". This includes minidfs cluster...which should improve parallelism on test runs.

I don't think we'll be able to do this.  As you've seen, there are various tests that need
to do a NameNode restart without reformatting so that we can verify things like correct metadata
reloaded from fsimage on startup.  If this is something needed to support runs outside of
Maven, then would it be possible to statically intialize a random directory once at startup
and then reuse it for the whole suite?  Maybe that would help.

When running through Maven, the build will generate N different unique directories, where
N = number of concurrent test processes.  Then, each concurrent test process really operates
in its own unique testing directory.  That has been sufficient for isolation in the Maven
runs.  (Again, maybe you're trying to improve support for running outside of Maven?)

> tests are using /test/build/data; breaking Jenkins
> --------------------------------------------------
>                 Key: HDFS-9263
>                 URL: https://issues.apache.org/jira/browse/HDFS-9263
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 3.0.0
>         Environment: Jenkins
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Blocker
>         Attachments: HDFS-9263-001.patch
> Some of the HDFS tests are using the path {{test/build/data}} to store files, so leaking
files which fail the new post-build RAT test checks on Jenkins (and dirtying all development
systems with paths which {{mvn clean}} will miss.
> fix

This message was sent by Atlassian JIRA

View raw message