systemml-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niketan Pansare (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SYSTEMML-2514) Hadoop gives java.lang.UnsatisfiedLinkError
Date Wed, 20 Mar 2019 23:49:00 GMT

    [ https://issues.apache.org/jira/browse/SYSTEMML-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797676#comment-16797676
] 

Niketan Pansare commented on SYSTEMML-2514:
-------------------------------------------

I usually use SystemML with Spark. To verify whether Spark is correctly installed, please
follow the below steps:

 

1. Java 
=======
The Java version should be > 1.8.

> java -version

Set JAVA_HOME environment variable and include %JAVA_HOME%\bin in the environment variable
PATH

> ls "%JAVA_HOME%"

2. Spark
========
Download and extract Spark from https://spark.apache.org/downloads.html,

> tar -xzf spark-2.1.0-bin-hadoop2.7.tgz

and set environment variable SPARK_HOME to point to the extracted directory.
 
Next step, install winutils:

- Download winutils.exe from http://github.com/steveloughran/winutils/raw/master/hadoop-2.6.0/bin/winutils.exe

- Place it in c:\winutils\bin
- Set environment variable HADOOP_HOME to point to c:\winutils
- Add c:\winutils\bin to the environment variable PATH.
- Finally, modify permission of hive directory that will be used by spark

> winutils.exe chmod 777 /tmp/hive

Finally, check if Spark is correctly installed:

> %SPARK_HOME%\bin\spark-shell
 > %SPARK_HOME%\bin\pyspark

 

> Hadoop gives java.lang.UnsatisfiedLinkError
> -------------------------------------------
>
>                 Key: SYSTEMML-2514
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2514
>             Project: SystemML
>          Issue Type: Sub-task
>          Components: Build
>            Reporter: Janardhan
>            Priority: Minor
>         Attachments: hadoop_on_win.pdf
>
>
>  
> {code:java}
> java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
> at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
> at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
> at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:295)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
> at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
> at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:910)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:891)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:788)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1458)
> at org.apache.sysml.test.utils.TestUtils.writeTestMatrix(TestUtils.java:1514)
> at org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:530)
> at org.apache.sysml.test.integration.AutomatedTestBase.writeInputMatrix(AutomatedTestBase.java:587)
> at org.apache.sysml.test.integration.functions.blocks.VariableTest.testVariablePassing1(VariableTest.java:80)
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message