hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: NameNode low on available disk space
Date Thu, 28 Feb 2013 08:29:26 GMT
Hi,

The exact error is displayed on your log and should be somewhat self
explanatory:

org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
Directory /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent
state: storage directory does not exist or is not accessible.

Please check this one's availability, permissions (the NN user should
be able to access it).

On Thu, Feb 28, 2013 at 1:46 PM, Mohit Vadhera
<project.linux.proj@gmail.com> wrote:
> Please find below logs for shutting down the namenode service. Can anybody
> check this
>
> 2013-02-28 02:07:51,752 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN org.apache.hadoop.hdfs.server.common.Util: Path
> /mnt/san1/hdfs/cache/hdfs/dfs/name should be specified as a URI in
> configuration files. Please update hdfs configuration.
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
> of redundant storage directories!
> 2013-02-28 02:07:51,754 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
> 2013-02-28 02:07:51,884 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
> 2013-02-28 02:07:51,890 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2013-02-28 02:07:51,909 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication         = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
> = 512
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
> = 1
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams      = 2
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks  = false
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2013-02-28 02:07:51,910 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer        = false
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> hdfs (auth:SIMPLE)
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> hadmin
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2013-02-28 02:07:51,920 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2013-02-28 02:07:51,925 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2013-02-28 02:07:52,462 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names occuring
> more than 10 times
> 2013-02-28 02:07:52,466 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2013-02-28 02:07:52,467 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 0
> 2013-02-28 02:07:52,469 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /mnt/san1/hdfs/cache/hdfs/dfs/name does not exist.
> 2013-02-28 02:07:52,471 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 2013-02-28 02:07:52,472 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2013-02-28 02:07:52,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2013-02-28 02:07:52,473 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory
> /mnt/san1/hdfs/cache/hdfs/dfs/name is in an inconsistent state: storage
> directory does not exist or is not accessible.
>        at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:295)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:201)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
> 2013-02-28 02:08:48,908 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2013-02-28 02:08:48,913 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/192.168.1.3
>
>
> On Thu, Feb 28, 2013 at 1:27 PM, Mohit Vadhera
> <project.linux.proj@gmail.com> wrote:
>>
>> Hi Guys,
>>
>> I have space on other partition. Can I change the path for cache files on
>> other partition ? I have below properties . Can it resolve the issue ? If i
>> change the path to other directories and restart services I get the below
>> error while starting the service namenode. I didn't find anything in logs so
>> far.  Can you please suggest something ?
>>
>>   <property>
>>      <name>hadoop.tmp.dir</name>
>>      <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
>>   </property>
>>   <property>
>>      <name>dfs.namenode.name.dir</name>
>>      <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/name</value>
>>   </property>
>>   <property>
>>      <name>dfs.namenode.checkpoint.dir</name>
>>
>> <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/namesecondary</value>
>>   </property>
>>   <property>
>>
>>
>> Service namenode is failing
>>
>> # for service in /etc/init.d/hadoop-hdfs-* ; do sudo $service status; done
>> Hadoop datanode is running                                 [  OK  ]
>> Hadoop namenode is dead and pid file exists                [FAILED]
>> Hadoop secondarynamenode is running                        [  OK  ]
>>
>> Thanks,
>>
>>
>>
>> On Wed, Jan 23, 2013 at 11:15 PM, Mohit Vadhera
>> <project.linux.proj@gmail.com> wrote:
>>>
>>>
>>> On Wed, Jan 23, 2013 at 10:41 PM, Harsh J <harsh@cloudera.com> wrote:
>>>>
>>>> http://NNHOST:50070/conf
>>>
>>>
>>>
>>> Harsh, I changed the value as said & restarted service NN. For verifying
>>> i checked the http link that you gave and i saw the property their but on
>>> http://NNHOST:50070  i noticed warning( WARNING : There are 4 missing
>>> blocks. Please check the logs or run fsck in order to identify the missing
>>> blocks.)  when i clicked on this  link i can see file names . Do I need to
>>> reboot the machine to run fsck on root fs/ or is there hadoop command fsck
>>> that i can run on the running hadoop ?
>>>
>>> Thanks,
>>>
>>
>



--
Harsh J

Mime
View raw message