accumulo-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rich Fecher (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ACCUMULO-3293) bin\winutils.exe not found when running MiniAccumuloCluster under Windows
Date Mon, 10 Nov 2014 14:56:35 GMT

    [ https://issues.apache.org/jira/browse/ACCUMULO-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14204851#comment-14204851
] 

Rich Fecher commented on ACCUMULO-3293:
---------------------------------------

We ran into the same problems. We wanted to get MiniAccumuloCluster working on versions of
Hadoop >= 2.2 even though it does this check on windows for {{HADOOP_HOME}}/bin/winutils.exe.
 Furthermore, winutils.exe depends on hadoop.dll to be in the path, so I think for now we
are willing to accept that MiniAccumuloCluster cannot be self-encapsulated in this instance
until Hadoop changes this dependency or the windows native resources are published as an artifact.

But rather than modifying the implementation of MiniAccumuloCluster, we assume {{HADOOP_HOME}}
must be set to a valid installation if it is windows and the version of Hadoop requires native
dependencies.  We solve our problems by copying winutils.exe to the correct location relative
to our MiniAccumuloCluster directory - after instantiating MiniAccumuloCluster because it
ensures the directory is empty, but before calling start().

In the long run, being able to supply an existing {{HADOOP_HOME}} to MiniAccumuloCluster would
be nice (similar to Johannes' suggestion).  Then we wouldn't need to copy any files, but for
now, this interim solution works well for us.

Here's the code example of what I describe: https://github.com/ngageoint/geowave/blob/master/geowave-test/src/test/java/mil/nga/giat/geowave/test/GeoWaveITSuite.java#L76

> bin\winutils.exe not found when running MiniAccumuloCluster under Windows
> -------------------------------------------------------------------------
>
>                 Key: ACCUMULO-3293
>                 URL: https://issues.apache.org/jira/browse/ACCUMULO-3293
>             Project: Accumulo
>          Issue Type: Bug
>          Components: mini
>         Environment: MS Windows
>            Reporter: Johannes Bauer
>
> When running JUnit-Tests on Windows with the MiniAccumuloCluster, the file {{bin\winutils.exe}}
is needed, which is searched in the path, given by the system variable {{HADOOP_HOME}}. This
worked fine with the MiniAccumuloCluster 1.4.4.
> When I try the same with Accumulo 1.6, I get the error 
> {code}
> 14/10/24 16:57:40 ERROR util.Shell: Failed to locate the winutils binary in the hadoop
binary path
> java.io.IOException: Could not locate executable C:\Users\jsbaue\AppData\Local\Temp\1414162594270-0\bin\winutils.exe
in the Hadoop binaries.
> 	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
> 	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
> 	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
> 	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> {code}
> which basically says, that {{bin\winutils.exe}} is now searched in the temporary path,
given to the MiniAccumuloCluster. When I try to put the file {{bin\winutils.exe}} in that
path, I get an error, saying that the path has to be empty. 
> I fixed this problem by changing {{accumulo\minicluster\src\main\java\org\apache\accumulo\minicluster\impl\MiniAccumuloClusterImpl.java}}:
> I exchanged Line 309-310:
> {code}
>     // hadoop-2.2 puts error messages in the logs if this is not set
>     builder.environment().put("HADOOP_HOME", config.getDir().getAbsolutePath());
> {code}
> with the following lines: 
> {code}
>     env = System.getProperty("hadoop.home.dir");
>     if(env == null) 
>         env = System.getenv("HADOOP_HOME");
>     // hadoop-2.2 puts error messages in the logs if this is not set
>     if(env == null)
>         env = config.getDir().getAbsolutePath();
>     builder.environment().put("HADOOP_HOME", env);
> {code}
> This fixes the bug on Windows. I did no checks on Linux. 
> Could you check, if you chould change this in the next version of the MiniAccumuloCluster
or help me with an other solution? Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message