hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parag Dhanuka <parag.dhan...@gmail.com>
Subject Re: Problem copying data to hadoop
Date Tue, 24 Feb 2009 13:31:29 GMT
I think I know what the problem is... It was related to maximum number of
file descriptors I can had opened. The limit was 1024 and I was exceeding
that limit. In order to fix it I increased the number to 65535 using help
from here
http://www.cs.uwaterloo.ca/~brecht/servers/openfiles.html

The process takes 2 hrs to finish but I am hoping it does finish this time
without causing any pains.

Parag

On Tue, Feb 24, 2009 at 5:29 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:

> I re ran after formatting and got this error.. Can it be because I have too
> many files opened?
> 2009-02-24 02:46:19,284 ERROR
> org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
> 192.168.1.201:50010, storageID=DS-1261214527-127.0
> .1.1-50010-1235462901573, infoPort=50075, ipcPort=50020):DataXceiver
> java.io.IOException: Input/output error
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.createNewFile(File.java:883)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:398)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.createTmpFile(FSDataset.java:371)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.createTmpFile(FSDataset.java:1108)
>         at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:997)
>         at
> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:98)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:258)
>         at
> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>         at java.lang.Thread.run(Thread.java:619)
>
> On Tue, Feb 24, 2009 at 12:58 PM, Parag Dhanuka <parag.dhanuka@gmail.com>wrote:
>
>> Hi Mithila,
>> Thanks for a quick reply...
>>
>> I did check the name node and data node logs... They have died and I ll
>> restart them but I just want to know what happened before doing anything.
>>
>> On the data node the interesting error which I saw is this:
>>
>>
>> java.io.EOFException: while trying to read 65557 bytes
>>         at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:254)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:298)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:362)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:514)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:356)
>>         at
>> org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:102)
>>
>> At the same time I saw the following error on name node
>>  2009-02-23 10:02:57,181
>> FATAL  org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error :
>> All storage directories are inaccessible.
>>
>> Can you please elaborate as to what this might be.
>>
>> Parag
>>
>> On Tue, Feb 24, 2009 at 12:49 PM, Mithila Nagendra <mnagendr@asu.edu>wrote:
>>
>>> Hey Parag
>>> Check if the namenode and the datanode are up and running. Use the 'jps'
>>> command to do so. If they are not running U ll have to do a stop-all and
>>> reformat the namenode using hadoop namenode -format (make sure u have no
>>> data on the HDFS). Then restart hadoop using start-all.sh.
>>>
>>> If U have the datanode and the namenode running, then check the log files
>>> for errors.
>>>
>>> Mithila
>>>
>>> On Tue, Feb 24, 2009 at 10:02 AM, Parag Dhanuka <parag.dhanuka@gmail.com
>>> >wrote:
>>>
>>> > I have setup hadoop in pseudo distributed mode with namenode, datanode,
>>> > jobtracker and tasktracker all on the same machine...
>>> > I also have a code which I use to write my data into hadoop. The code
>>> of my
>>> > mine reads data from the local disk does some preprocessing and after
>>> that
>>> > uses (multiple) FSDataOutputStream to write data to hadoop. I have
>>> multiple
>>> > FSDataOutputStreams open at one time cause I want to write data into
>>> > different files based on some logic I have.
>>> >
>>> > Now the problem... While the process was writing data to hadoop I got
>>> this
>>> > error Problem renewing lease for DFSClient_1637324984. On going to name
>>> > node
>>> > logs I found this
>>> > 2009-02-23 10:02:57,181 FATAL
>>> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
>>> > storage directories are inaccessible.
>>> >
>>> > I have absolutely no idea as to what might have caused this. Can some
>>> one
>>> > please help.
>>> >
>>> > --
>>> > Parag Dhanuka
>>> >
>>>
>>
>>
>>
>> --
>> Parag Dhanuka
>>
>
>
>
> --
> Parag Dhanuka
>



-- 
Parag Dhanuka

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message