hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4491) Parallel testing HDFS
Date Tue, 19 Feb 2013 17:15:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581449#comment-13581449

Chris Nauroth commented on HDFS-4491:

I noticed branch "branch-trunk-win" which seems to be dedicated to Windows support - is it

You are correct that branch-trunk-win is a branch off of trunk dedicated to making the current
trunk codebase compatible with Windows.

Unfortunately I couldn't proceed too far with a Windows VM and using this branch, as one of
the requirements is MS Visual Studio which I don't have.

I'll work on providing more documentation about dev environment requirements for Windows work.
 Meanwhile, if you don't have access to Visual Studio, then you can download the Windows SDK
for free:


This is sufficient to enable a command line build of the Windows native components.  Run mvn
with the -Pnative-win profile enabled.

1. Why are these fixes applied to trunk while in general Windows support is kept in "branch-trunk-win"?

Whenever we can make a fix that doesn't have a direct dependency on Windows-specific code,
then we target trunk for that patch.  Several examples of this kind of fix include file path
corrections, switching from hard-coded \n to {{System.getProperty("line.separator")}}, and
fixes for race conditions in the code that by mere coincidence were never visible on Linux.
 Essentially, these are not really Windows fixes but rather general bug fixes that just happen
to cause visible problems on Windows.  We merge trunk to branch-trunk-win frequently, so there
is little delay between committing these fixes to trunk and seeing them in branch-trunk-win.

Whenever the fix requires a dependency on Windows-specific code, then we target branch-trunk-win.
 This includes larger changes, such as variants of the {{Shell}} commands that work on Windows,
native code support for Windows, and startup scripts for Windows.

By targeting trunk when possible, we achieve a simpler merge back to trunk later.  When we
eventually merge branch-trunk-win back to trunk, that code review can focus on the truly platform-specific
Windows code.

2. Why are just a few tests fixed? There are many others which use "test.build.data"

The only tests that have been changed to use /tmp/<test class name> are specifically
tests that would attempt to use a path in HDFS containing a ':' due to the presence of drive

3. Is it a proper way to fix it? There are reasons of using "test.build.data" to place all
test data under. E.g. this data can be placed on ramfs to speedup testing.

Once again, this change does not alter local file system interactions in any way.  Tests against
local file system continue to use {{test.build.data}}. (i.e. C:\hadoop-common\... on Windows).
 Additionally, {{MiniDFSCluster}} still uses {{test.build.data}} for storage of namenode metadata
and datanode block files.

The only reason for this change is that HDFS has validation rules that reject any path containing
a ':'.  This just changes the HDFS path used for test files during the test run, which does
not directly correlate to storage on the local file system.  This does not alter the ability
to mount a ramfs for a faster {{test.build.data}}.

> Parallel testing HDFS
> ---------------------
>                 Key: HDFS-4491
>                 URL: https://issues.apache.org/jira/browse/HDFS-4491
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: test
>    Affects Versions: 3.0.0
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Andrey Klochkov
>         Attachments: HDFS-4491.patch
> Parallel execution of HDFS tests in multiple forks. See HADOOP-9287 for details.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message