hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parag Dhanuka <parag.dhan...@gmail.com>
Subject Re: Problem copying data to hadoop
Date Tue, 24 Feb 2009 14:36:06 GMT
If it helps I also have this in my namenode logs
2009-02-24 05:31:14,161 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 9000, call getListing(/data/hadoop/mapred/system) from
192.168.1.20
1:40598: error: org.apache.hadoop.security.AccessControlException:
Permission denied: user=webuser, access=READ_EXECUTE,
inode="system":hadoop:supergr
oup:rwx-wx-wx
org.apache.hadoop.security.AccessControlException: Permission denied:
user=webuser, access=READ_EXECUTE,
inode="system":hadoop:supergroup:rwx-wx-wx
        at
org.apache.hadoop.hdfs.server.namenode.PermissionChecker.check(PermissionChecker.java:176)
        at
org.apache.hadoop.hdfs.server.namenode.PermissionChecker.checkPermission(PermissionChecker.java:111)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:4386)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPathAccess(FSNamesystem.java:4346)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListing(FSNamesystem.java:1969)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.getListing(NameNode.java:485)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:452)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:892)



On Tue, Feb 24, 2009 at 7:51 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:

> Sorry for spamming but I also found some interesting things in the
> secondary data node and here is the log again.
> My name node crashed at
> 2009-02-24 06:02:38,771 FATAL
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
> storage directories are inaccessible.
>
> whilst secondary data node had this
> 2009-02-24 05:02:38,741 WARN
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint done.
> New Image Size: 28519
> 2009-02-24 06:02:38,757 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions:
> 0 Total time for transactions(ms): 0 Number
> of syncs: 0 SyncTimes(ms): 0
> 2009-02-24 06:02:39,089 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in
> doCheckpoint:
> 2009-02-24 06:02:39,089 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Call to hadoop1/192.168.1.201:9000 failed
> :
>
>
> Can checkpointing be creating some problem..
>
> Parag
>
> On Tue, Feb 24, 2009 at 7:46 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:
>
>> This is how my data log looks like..... Please help.... I am lost now :(
>> 2009-02-24 05:52:17,613 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>> blk_-7039044529189438628_1012 terminating
>> 2009-02-24 05:52:17,619 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
>> 192.168.1.200:55700, dest: /192.168.1.201:50010, byte
>> : 22260762, op: HDFS_WRITE, cliID: DFSClient_1664136935, srvID:
>> DS-1255721471-127.0.1.1-50010-1235480246191, blockid:
>> blk_7602190212483678862_1017
>> 2009-02-24 05:52:17,619 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
>> blk_7602190212483678862_1017 terminating
>> 2009-02-24 05:55:54,536 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
>> succeeded for blk_-477808440580664947_1021
>> 2009-02-24 05:56:54,539 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
>> succeeded for blk_1534358347758148859_1021
>> 2009-02-24 06:01:38,546 INFO
>> org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
>> succeeded for blk_-3560215495928841638_1004
>> 2009-02-24 06:02:41,560 WARN
>> org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call
>> to hadoop1/192.168.1.201:9000 failed on local
>> exception: null
>>         at org.apache.hadoop.ipc.Client.call(Client.java:699)
>>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>         at $Proxy4.sendHeartbeat(Unknown Source)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:669)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1100)
>>         at java.lang.Thread.run(Thread.java:619)
>> Caused by: java.io.EOFException
>>         at java.io.DataInputStream.readInt(DataInputStream.java:375)
>>         at
>> org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:493)
>>         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:438)
>>
>> Parag
>>
>> On Tue, Feb 24, 2009 at 7:43 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:
>>
>>> It doesnot seem to be helping.... my name node got down again and on
>>> trying to restart it I got this error....
>>> 2009-02-24 06:06:59,063 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem
>>> initialization failed.
>>> java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock
>>> (Input/output error)
>>>         at java.io.RandomAccessFile.open(Native Method)
>>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
>>> 2009-02-24 06:06:59,065 INFO org.apache.hadoop.ipc.Server: Stopping
>>> server on 9000
>>> 2009-02-24 06:06:59,065 ERROR
>>> org.apache.hadoop.hdfs.server.namenode.NameNode:
>>> java.io.FileNotFoundException: /data/hadoop/dfs/name/in_use.lock (Inpu
>>> /output error)
>>>         at java.io.RandomAccessFile.open(Native Method)
>>>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.tryLock(Storage.java:524)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.lock(Storage.java:505)
>>>         at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:363)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:273)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:87)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:311)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:290)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:163)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:208)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:194)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:859)
>>>         at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:868)
>>>
>>>
>>>
>>> I am lost please helpppppppppppp
>>>
>>> Parag
>>>
>>> On Tue, Feb 24, 2009 at 7:01 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:
>>>
>>>> I think I know what the problem is... It was related to maximum number
>>>> of file descriptors I can had opened. The limit was 1024 and I was exceeding
>>>> that limit. In order to fix it I increased the number to 65535 using help
>>>> from here
>>>> http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html
>>>>
>>>> The process takes 2 hrs to finish but I am hoping it does finish this
>>>> time without causing any pains.
>>>>
>>>> Parag
>>>>
>>>>
>>>> On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka <parag.dhanuka@gmail.com
>>>> > wrote:
>>>>
>>>>> I re ran after formatting and got this error.. Can it be because I have
>>>>> too many files opened?
>>>>> 2009-02-24 02:46:19,284 ERROR
>>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
>>>>> 192.168.1.201:50010, storageID=DS-1261214527-127.0
>>>>> .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
>>>>> java.io.IOException: Input/output error
>>>>>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>>>>>         at java.io.File.createNewFile(File.java:883)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
>>>>>         at
>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka <
>>>>> parag.dhanuka@gmail.com> wrote:
>>>>>
>>>>>> Hi Mithila,
>>>>>> Thanks for a quick reply...
>>>>>>
>>>>>> I did check the name node and data node logs... They have died and
I
>>>>>> ll restart them but I just want to know what happened before doing
anything.
>>>>>>
>>>>>> On the data node the interesting error which I saw is this:
>>>>>>
>>>>>>
>>>>>> java.io.EOFException: while trying to read 65557 bytes
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
>>>>>>         at
>>>>>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>>>>>>
>>>>>> At the same time I saw the following error on name node
>>>>>>  2009-02-23 10:02:57,181
>>>>>> FATAL  org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal
Error :
>>>>>> All storage directories are inaccessible.
>>>>>>
>>>>>> Can you please elaborate as to what this might be.
>>>>>>
>>>>>> Parag
>>>>>>
>>>>>> On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra <mnagendr@asu.edu>wrote:
>>>>>>
>>>>>>> Hey Parag
>>>>>>> Check if the namenode and the datanode are up and running. Use
the
>>>>>>> 'jps'
>>>>>>> command to do so. If they are not running U ll have to do a stop-all
>>>>>>> and
>>>>>>> reformat the namenode using hadoop namenode -format (make sure
u have
>>>>>>> no
>>>>>>> data on the HDFS). Then restart hadoop using start-all.sh.
>>>>>>>
>>>>>>> If U have the datanode and the namenode running, then check the
log
>>>>>>> files
>>>>>>> for errors.
>>>>>>>
>>>>>>> Mithila
>>>>>>>
>>>>>>> On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <
>>>>>>> parag.dhanuka@gmail.com>wrote:
>>>>>>>
>>>>>>> > I have setup hadoop in pseudo distributed mode with namenode,
>>>>>>> datanode,
>>>>>>> > jobtracker and tasktracker all on the same machine...
>>>>>>> > I also have a code which I use to write my data into hadoop.
The
>>>>>>> code of my
>>>>>>> > mine reads data from the local disk does some preprocessing
and
>>>>>>> after that
>>>>>>> > uses (multiple) FSDataOutputStream to write data to hadoop.
I have
>>>>>>> multiple
>>>>>>> > FSDataOutputStreams open at one time cause I want to write
data
>>>>>>> into
>>>>>>> > different files based on some logic I have.
>>>>>>> >
>>>>>>> > Now the problem... While the process was writing data to
hadoop I
>>>>>>> got this
>>>>>>> > error Problem renewing lease for DFSClient_1637324984. On
going to
>>>>>>> name
>>>>>>> > node
>>>>>>> > logs I found this
>>>>>>> > 2009-02-23 10:02:57,181 FATAL
>>>>>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal
Error :
>>>>>>> All
>>>>>>> > storage directories are inaccessible.
>>>>>>> >
>>>>>>> > I have absolutely no idea as to what might have caused this.
Can
>>>>>>> some one
>>>>>>> > please help.
>>>>>>> >
>>>>>>> > --
>>>>>>> > Parag Dhanuka
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Parag Dhanuka
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Parag Dhanuka
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Parag Dhanuka
>>>>
>>>
>>>
>>>
>>> --
>>> Parag Dhanuka
>>>
>>
>>
>>
>> --
>> Parag Dhanuka
>>
>
>
>
> --
> Parag Dhanuka
>



-- 
Parag Dhanuka

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message