hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernd Eckenfels <e...@zusammenkunft.net>
Subject Re: MiniDFSCluster on Windows unfortunatelly requires native toolchain for automated tests
Date Mon, 23 Jun 2014 22:35:10 GMT
Hello Chris,

thanks for answering. I am not happy about the fact, that it might be
not possible to have generic fallbacks in all cases.

For the actual
method in question I think File#isWriteable is a perfectly fine
fallback especially as it is used in Posix systems anyway (and is never
really reliable). I think the method is also used without a real need
in this place. I hade the hope that other places look similiar.

Of course I can understand that you dont want to allow generic
fallbacks in production environments. Would you suggest to have a
system property?

I have seen you comments on the VFS bugs. I am not so sure about the
binaries policy for testing in apache commons, I will need to check
this. However I would simply disable the tests on Windows again.

There are some comments talking about the need for fallbacks in the
source. If this is not the actual desire it should be removed.

Gruss
Bernd

Am Fri, 20 Jun 2014 10:02:49 -0700
schrieb Chris Nauroth <cnauroth@hortonworks.com>:

> Hello Bernd,
> 
> This is my first time looking at Commons VFS.  It looks like a very
> useful project, and I'm sorry to hear that we've caused some grief
> for your tests.
> 
> I can provide some background on these changes.  Traditionally, 1.2
> and earlier versions didn't officially support Windows beyond a best
> effort to keep some approximation of the functionality running on
> developer desktops. As we worked on enabling real production
> deployments on Windows, we discovered multiple bugs caused by those
> approximations.
> 
> In some cases, the Cygwin and GnuWin32 tools had incomplete
> implementations or bugs.  Examples of the buggy areas include finding
> the groups for a user and process management.  Basically, anything in
> org.apache.hadoop.util.Shell was problematic.  For this category of
> bugs, the solution was winutils.exe, which delegates straight to the
> necessary Windows API calls.
> 
> Other problems arose from bugs or missing functionality in the
> Windows JDK implementation.  For example, the canRead/canWrite
> methods in java.io.File don't work correctly on Windows.  (This is
> the code path shown in your stack trace.)  For this category of bugs,
> the solution was hadoop.dll, which is a JNI layer that delegates to
> the Windows API.
> 
> Without these workarounds, production deployment on Windows wouldn't
> have been feasible.  There would have been too many dangerous bugs.
> 
> Unfortunately, we have a weakness right now in that the Windows native
> build artifacts aren't bundled into the Apache releases.  We also
> don't have a fallback mode that works without the native code.  (Even
> if there was such a fallback mode, it would be very dangerous to run
> in this mode in production, so I'd want to treat it as a test-only
> setting.)
> 
> I expect you'll run into a lot of challenges while fully implementing
> a Java-only fallback.  I'd like to help by checking out the Commons
> VFS code and experimenting a bit.  I'll try to put some time into
> this next week and then comment on the VFS jira issues with my
> findings.  Does this sound good?
> 
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
> 
> 
> 
> On Tue, Jun 17, 2014 at 7:46 PM, Bernd Eckenfels
> <ecki@zusammenkunft.net> wrote:
> 
> > Hello,
> >
> > in the Commons VFS project we have a Provider for HDFS. This
> > Provider is unit-tested with the help of the excelent MiniDFSCluster
> > component.
> >
> > A while back I enabled automated testing on Windows (cause it worked
> > for me), but I did not realize it only works with cygwin (namely
> > ls.exe) in the Path. This was some 1.2 version and it used the ls
> > command to read the owner.
> >
> > We currently switch the dependency to 2.4.0 and had the hope that it
> > better (i.e.
> > without binary dependencies) works in this scenario - as we want to
> > run the maven tests without pre-installing a tool chain on Windows.
> >
> > However it got actually worse, it warns about a missing winutils.exe
> > and it requires the hadoop.dll. The first location where it fails is
> > testing writeability for the name node. And unlike all the comments
> > in FileUtil/NativeIO it does not cleanly fall back to java portable
> > methods.
> >
> > I think the access/access0 method can easily be fixed, but before I
> > work on this patch, does it make sense to go in this direction or
> > will I have to fix some other, more complicated platform
> > dependencies. My goal would be to start MiniDFSCluster inside
> > surefire from maven without the need for complicated external
> > toolchain setup.
> >
> > Is actually anybody caring about making this possible?
> >
> > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest
> > Time elapsed: 2.291 sec  <<< ERROR!
> > java.lang.UnsatisfiedLinkError:
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
> >         at
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native
> > Method) at
> > org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
> >         at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
> >         at
> > org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:292)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:879)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> >         at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> >         at
> > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:975)
> >         at
> > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:856)
> >         at
> > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
> >         at
> > org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:640)
> >         at
> > org.apache.commons.vfs2.provider.hdfs.test.HdfsFileProviderTest.<clinit>(HdfsFileProviderTest.java:95)
> >
> > Greetings
> > Bernd
> >
> > PS - in case you want to follow the VFS progress:
> > * https://issues.apache.org/jira/browse/VFS-530 - tracks 2.4
> > migration
> > * https://issues.apache.org/jira/browse/VFS-529 - the broken
> > windows tests
> > * https://issues.apache.org/jira/browse/VFS-526 - failed enable of
> > win tests
> >
> >
> 


Mime
View raw message