commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Eckenfels (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (VFS-530) Update HDFS to latest release in prep for VFS 2.1 release
Date Wed, 18 Jun 2014 02:15:09 GMT

    [ https://issues.apache.org/jira/browse/VFS-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14034696#comment-14034696
] 

Bernd Eckenfels edited comment on VFS-530 at 6/18/14 2:14 AM:
--------------------------------------------------------------

If this is a unrelated/unneeded change I would do it in a seperate commit. In all cases the
comment needs to be fixed (1M):

{code}-            conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 100); // 100K blocksize
+            conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 1024); // 100K blocksize{code}

I tested it on windows, and it seems to have the problem to solve the cygwin dependency with
a own binary and native code, but the POM changes alone do not work, it uses the wunutils.exe
(bit recovers from it:)

{code}Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
2014-06-18 03:13:44,939 [main] ERROR: Failed to locate the winutils binary in the hadoop binary
path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
        at org.apache.hadoop.conf.Configuration.getTrimmedStringCollection(Configuration.java:1707)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:1238)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceDirs(FSNamesystem.java:1219)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:816)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
        at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95){code}

and it also needs the hadoop.dll, especially the accesscheck method. Looking at the code this
really can be stubbed out, but for now at least this one place is demanding in a way that
the DLL would have to be installed to run the unittests. :-/



was (Author: b.eckenfels):
If this is a unrelated/unneeded change I would do it in a seperate commit. In all cases the
comment needs to be fixed (1M):

{code}-            conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 100); // 100K blocksize
+            conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024 * 1024); // 100K blocksize{code}

I tested it on windows, and it seems to have the problem to solve the cygwin dependency with
a own binary and native code, but the POM changes alone do not work, it isses the wunutils.exe
and a dependency.

{code}Running org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
2014-06-18 03:13:44,939 [main] ERROR: Failed to locate the winutils binary in the hadoop binary
path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
        at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
        at org.apache.hadoop.conf.Configuration.getTrimmedStringCollection(Configuration.java:1707)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:1238)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceDirs(FSNamesystem.java:1219)
        at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:816)
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
        at org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95){code}


> Update HDFS to latest release in prep for VFS 2.1 release
> ---------------------------------------------------------
>
>                 Key: VFS-530
>                 URL: https://issues.apache.org/jira/browse/VFS-530
>             Project: Commons VFS
>          Issue Type: Task
>    Affects Versions: 2.1
>            Reporter: Dave Marion
>             Fix For: 2.1
>
>         Attachments: VFS-530-1.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message