hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Re: HDFS input/output error - fuse mount
Date Sat, 19 Jul 2014 10:46:16 GMT
Apache Bigtop detects JVM based on the most recent version first, if
you use packages based on their framework:
https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/bigtop-utils/bigtop-detect-javahome
(switch branch/tag appropriately to a specific release you may be
using/based off)

Within Apache Hadoop, we rely on the definition of JAVA_HOME
explicitly. In a packaged environment, the above script typically
supplies it.

On Sat, Jul 19, 2014 at 6:35 AM, Chris Mawata <chris.mawata@gmail.com> wrote:
> Great that you got it sorted out. I'm afraid I don't know if there is a
> configuration that would automatically check the versions -- maybe someone
> who knows might chime in.
> Cheers
> Chris
>
> On Jul 18, 2014 3:06 PM, "andrew touchet" <adt027@latech.edu> wrote:
>>
>> Thanks Chris!
>>
>> The issue was that even though I set jdk-7u21 as my default, it checked
>> for /usr/java/jdk-1.6* first.  Even though it was compiled with 1.7.
>>
>> Is there anyway to generate a proper hadoop-config.sh to reflect the minor
>> version hadoop was built with? So that in my case, it would check for
>> /usr/java/jdk-1.7* instead?  I appreciate the help!
>>
>>
>> On Thu, Jul 17, 2014 at 11:11 PM, Chris Mawata <chris.mawata@gmail.com>
>> wrote:
>>>
>>> Yet another place to check -- in the hadoop-env.sh file there is also a
>>> JAVA_HOME setting.
>>> Chris
>>>
>>> On Jul 17, 2014 9:46 PM, "andrew touchet" <adt027@latech.edu> wrote:
>>>>
>>>> Hi Fireflyhoo,
>>>>
>>>> Below I follow the symbolic links for the jdk-7u21. These links are
>>>> changed accordingly as I change between versions. Also, I have 8 datanodes
>>>> and 2 other various servers that are capable of mounting /hdfs.  So it is
>>>> just this server is an issue.
>>>>
>>>> $ java -version
>>>> java version "1.7.0_21"
>>>> Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
>>>>
>>>> java
>>>> $ ls -l `which java`
>>>> lrwxrwxrwx 1 root root 26 Jul 17 19:50 /usr/bin/java ->
>>>> /usr/java/default/bin/java
>>>> $ ls -l /usr/java/default
>>>> lrwxrwxrwx 1 root root 16 Jul 17 19:50 /usr/java/default ->
>>>> /usr/java/latest
>>>> $ ls -l /usr/java/latest
>>>> lrwxrwxrwx 1 root root 21 Jul 17 20:29 /usr/java/latest ->
>>>> /usr/java/jdk1.7.0_21
>>>>
>>>> jar
>>>> $ ls -l `which jar`
>>>> lrwxrwxrwx 1 root root 21 Jul 17 20:18 /usr/bin/jar ->
>>>> /etc/alternatives/jar
>>>> $ ls -l /etc/alternatives/jar
>>>> lrwxrwxrwx 1 root root 29 Jul 17 20:26 /etc/alternatives/jar ->
>>>> /usr/java/jdk1.7.0_21/bin/jar
>>>>
>>>> javac
>>>> $ ls -l `which javac`
>>>> lrwxrwxrwx 1 root root 23 Jul 17 20:18 /usr/bin/javac ->
>>>> /etc/alternatives/javac
>>>> $ ls -l /etc/alternatives/javac
>>>> lrwxrwxrwx 1 root root 31 Jul 17 20:26 /etc/alternatives/javac ->
>>>> /usr/java/jdk1.7.0_21/bin/javac
>>>>
>>>> Now that I've tried version from  6 & 7, I'm really not sure what is
>>>> causing this issue.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Jul 17, 2014 at 8:21 PM, fireflyhoo@gmail.com
>>>> <fireflyhoo@gmail.com> wrote:
>>>>>
>>>>> I think  you first confirm you local java version ,
>>>>> Some  liux will  pre-installed java ,that version is very low
>>>>>
>>>>> ________________________________
>>>>> fireflyhoo@gmail.com
>>>>>
>>>>>
>>>>> From: andrew touchet
>>>>> Date: 2014-07-18 09:06
>>>>> To: user
>>>>> Subject: Re: HDFS input/output error - fuse mount
>>>>> Hi Chris,
>>>>>
>>>>> I tried to mount /hdfs with java versions below but there was no change
>>>>> in output.
>>>>> jre-7u21
>>>>> jdk-7u21
>>>>> jdk-7u55
>>>>> jdk1.6.0_31
>>>>> jdk1.6.0_45
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 17, 2014 at 6:56 PM, Chris Mawata <chris.mawata@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Version 51 ia Java 7
>>>>>> Chris
>>>>>>
>>>>>> On Jul 17, 2014 7:50 PM, "andrew touchet" <adt027@latech.edu>
wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Hadoop package installed:
>>>>>>> hadoop-0.20-0.20.2+737-33.osg.el5.noarch
>>>>>>>
>>>>>>> Operating System:
>>>>>>> CentOS release 5.8 (Final)
>>>>>>>
>>>>>>> I am mounting HDFS from my namenode to another node with fuse.
 After
>>>>>>> mounting to /hdfs, any attempts to 'ls', 'cd', or use 'hadoop
fs' leads to
>>>>>>> the below output.
>>>>>>>
>>>>>>>
>>>>>>> $ls /hdfs
>>>>>>> ls: /hdfs: Input/output error
>>>>>>> $hadoop fs -ls
>>>>>>> Exception in thread "main" java.lang.UnsupportedClassVersionError:
>>>>>>> org/apache/hadoop/fs/FsShell : Unsupported major.minor version
51.0
>>>>>>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>>>>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>>>>>     at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>>>>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>>>>>     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>>>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>> Could not find the main class: org.apache.hadoop.fs.FsShell.
 Program
>>>>>>> will exit.
>>>>>>>
>>>>>>>
>>>>>>> I have attempted to mount /hdfs manually in debug mode and then
>>>>>>> attempted to access /hdfs from a different terminal. This is
the output. The
>>>>>>> namenode is glados. The server where /hdfs is being mounted is
glados2.
>>>>>>>
>>>>>>>
>>>>>>> $hdfs -oserver=glados,port=9000,rdbuffer=131072,allow_other /hdfs
-d
>>>>>>> fuse-dfs ignoring option allow_other
>>>>>>> ERROR fuse_options.c:162 fuse-dfs didn't recognize /hdfs,-2
>>>>>>>
>>>>>>> fuse-dfs ignoring option -d
>>>>>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>>>>>>> INIT: 7.10
>>>>>>> flags=0x0000000b
>>>>>>> max_readahead=0x00020000
>>>>>>> INFO fuse_init.c:115 Mounting glados:9000
>>>>>>> Exception in thread "main" java.lang.UnsupportedClassVersionError:
>>>>>>> org/apache/hadoop/conf/Configuration : Unsupported major.minor
version 51.0
>>>>>>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>>>>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>>>>>     at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>>>>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>>>>>     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>>>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>> Can't construct instance of class
>>>>>>> org.apache.hadoop.conf.Configuration
>>>>>>> ERROR fuse_init.c:127 Unable to establish test connection to
server
>>>>>>>    INIT: 7.8
>>>>>>>    flags=0x00000001
>>>>>>>    max_readahead=0x00020000
>>>>>>>    max_write=0x00020000
>>>>>>>    unique: 1, error: 0 (Success), outsize: 40
>>>>>>> unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>>>> Exception in thread "Thread-0"
>>>>>>> java.lang.UnsupportedClassVersionError: org/apache/hadoop/conf/Configuration
>>>>>>> : Unsupported major.minor version 51.0
>>>>>>>     at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>     at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
>>>>>>>     at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
>>>>>>>     at
>>>>>>> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
>>>>>>>     at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
>>>>>>>     at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
>>>>>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
>>>>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>>>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>>>> Can't construct instance of class
>>>>>>> org.apache.hadoop.conf.Configuration
>>>>>>> ERROR fuse_connect.c:83 Unable to instantiate a filesystem for
>>>>>>> user027
>>>>>>> ERROR fuse_impls_getattr.c:40 Could not connect to glados:9000
>>>>>>>    unique: 2, error: -5 (Input/output error), outsize: 16
>>>>>>> unique: 3, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>>>>
>>>>>>> I adopted this system after this was already setup, so I do not
know
>>>>>>> which java version was used during install. Currently I'm using:
>>>>>>>
>>>>>>> $java -version
>>>>>>> java version "1.6.0_45"
>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
>>>>>>>
>>>>>>> $java -version
>>>>>>> java version "1.6.0_45"
>>>>>>> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
>>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
>>>>>>>
>>>>>>>
>>>>>>> Is my java version really the cause of this issue?  What is the
>>>>>>> correct java version to be used for this version of hadoop. 
I have also
>>>>>>> tried 1.6.0_31 but no changes were seen.
>>>>>>>
>>>>>>> If java isn't my issue, then what is?
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>
>



-- 
Harsh J

Mime
View raw message