hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravindra Kumar Naik <ravin.i...@gmail.com>
Subject Re: Name node starting intermittently
Date Tue, 28 Apr 2015 06:45:14 GMT
Hi,

Please add these lines in hdfs-site.xml. Please create the directories if
doesn't exist. You may choose other directory name if you want.









*<property><name>dfs.name.dir</name><value>/opt/hadoop/hdfs/name</value></property><property><name>dfs.data.dir</name><value>/opt/hadoop/hdfs/data</value></property>*

and then format namenode
*hadoop namenode -format*
and then start hadoop daemons using the script.

Regards,
Ravindra



On Tue, Apr 28, 2015 at 12:03 PM, Anand Murali <anand_vihar@yahoo.com>
wrote:

>
> Anand Murali
> 11/7, 'Anand Vihar', Kandasamy St, Mylapore
> Chennai - 600 004, India
> Ph: (044)- 28474593/ 43526162 (voicemail)
>
>
>
>   On Tuesday, April 28, 2015 12:00 PM, Ravindra Kumar Naik <
> ravin.iitb@gmail.com> wrote:
>
>
> Hi,
>
> Could you please post your hdfs-site.xml
>
> Regards,
> Ravindra
>
> On Tue, Apr 28, 2015 at 11:53 AM, Anand Murali <anand_vihar@yahoo.com>
> wrote:
>
> Ravindra:
>
> I am trying to use Hadoop out of the box. Please provide a remedy to fix
> this. I shall be thankful. I am a beginner with Hadoop.
>
> Thanks
>
> Anand Murali
> 11/7, 'Anand Vihar', Kandasamy St, Mylapore
> Chennai - 600 004, India
> Ph: (044)- 28474593/ 43526162 (voicemail)
>
>
>
>   On Tuesday, April 28, 2015 11:51 AM, Ravindra Kumar Naik <
> ravin.iitb@gmail.com> wrote:
>
>
> Hi,
>
> Using /tmp/ folder for hdfs storage directory is not a good idea.
> The /tmp/ directory is wiped out after reboot.
>
> Regards,
> Ravindra
>
>
> On Tue, Apr 28, 2015 at 11:31 AM, Anand Murali <anand_vihar@yahoo.com>
> wrote:
>
> Dear All:
>
> I am running Hadoop-2.6 on Ubuntu 15.04 desktop pseudo mode. Yesterday
> there was normal startup and shutdown couple of times. This morning it is
> not so. Find below section of log file. Shall be thankful if somebody can
> advise.
>
>
> STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git
> -r e3496499ecb8d220fba99dc5ed4c99c8f9e33bb1; compiled by 'jenkins' on
> 2014-11-13T21:10Z
> STARTUP_MSG:   java = 1.7.0_75
> ************************************************************/
> 2015-04-28 11:21:48,167 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: registered UNIX signal
> handlers for [TERM, HUP, INT]
> 2015-04-28 11:21:48,176 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []
> 2015-04-28 11:21:48,574 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2015-04-28 11:21:48,805 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2015-04-28 11:21:48,805 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
> 2015-04-28 11:21:48,806 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is
> hdfs://localhost:9000
> 2015-04-28 11:21:48,806 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use
> localhost:9000 to access this namenode/service.
> 2015-04-28 11:21:49,193 INFO org.apache.hadoop.hdfs.DFSUtil: Starting
> Web-server for hdfs at: http://0.0.0.0:50070
> 2015-04-28 11:21:49,287 INFO org.mortbay.log: Logging to
> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> org.mortbay.log.Slf4jLog
> 2015-04-28 11:21:49,291 INFO org.apache.hadoop.http.HttpRequestLog: Http
> request log for http.requests.namenode is not defined
> 2015-04-28 11:21:49,303 INFO org.apache.hadoop.http.HttpServer2: Added
> global filter 'safety'
> (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
> 2015-04-28 11:21:49,305 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context hdfs
> 2015-04-28 11:21:49,306 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context static
> 2015-04-28 11:21:49,306 INFO org.apache.hadoop.http.HttpServer2: Added
> filter static_user_filter
> (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
> context logs
> 2015-04-28 11:21:49,394 INFO org.apache.hadoop.http.HttpServer2: Added
> filter 'org.apache.hadoop.hdfs.web.AuthFilter'
> (class=org.apache.hadoop.hdfs.web.AuthFilter)
> 2015-04-28 11:21:49,397 INFO org.apache.hadoop.http.HttpServer2:
> addJerseyResourcePackage:
> packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
> pathSpec=/webhdfs/v1/*
> 2015-04-28 11:21:49,448 INFO org.apache.hadoop.http.HttpServer2: Jetty
> bound to port 50070
> 2015-04-28 11:21:49,448 INFO org.mortbay.log: jetty-6.1.26
> 2015-04-28 11:21:49,906 INFO org.mortbay.log: Started HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:50070
> 2015-04-28 11:21:50,021 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of data loss due to
> lack of redundant storage directories!
> 2015-04-28 11:21:50,021 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of data
> loss due to lack of redundant storage directories!
> 2015-04-28 11:21:50,070 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: No KeyProvider found.
> 2015-04-28 11:21:50,138 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsLock is fair:true
> 2015-04-28 11:21:50,215 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
> 2015-04-28 11:21:50,216 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.namenode.datanode.registration.ip-hostname-check=true
> 2015-04-28 11:21:50,243 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
> 2015-04-28 11:21:50,244 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: The block
> deletion will start around 2015 Apr 28 11:21:50
> 2015-04-28 11:21:50,246 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map BlocksMap
> 2015-04-28 11:21:50,246 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2015-04-28 11:21:50,252 INFO org.apache.hadoop.util.GSet: 2.0% max memory
> 889 MB = 17.8 MB
> 2015-04-28 11:21:50,252 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^21 = 2097152 entries
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication         = 1
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication             = 512
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication             = 1
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams      = 2
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks  = false
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer        = false
> 2015-04-28 11:21:50,274 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxNumBlocksToLog          = 1000
> 2015-04-28 11:21:50,278 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> anand_vihar (auth:SIMPLE)
> 2015-04-28 11:21:50,278 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> supergroup
> 2015-04-28 11:21:50,278 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
> 2015-04-28 11:21:50,278 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
> 2015-04-28 11:21:50,290 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
> 2015-04-28 11:21:50,776 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
> 2015-04-28 11:21:50,776 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2015-04-28 11:21:50,776 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> 889 MB = 8.9 MB
> 2015-04-28 11:21:50,776 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^20 = 1048576 entries
> 2015-04-28 11:21:50,777 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
> 2015-04-28 11:21:50,827 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map cachedBlocks
> 2015-04-28 11:21:50,827 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2015-04-28 11:21:50,827 INFO org.apache.hadoop.util.GSet: 0.25% max memory
> 889 MB = 2.2 MB
> 2015-04-28 11:21:50,827 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^18 = 262144 entries
> 2015-04-28 11:21:50,828 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
> 2015-04-28 11:21:50,828 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
> 2015-04-28 11:21:50,828 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 30000
> 2015-04-28 11:21:50,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
> 2015-04-28 11:21:50,829 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
> 2015-04-28 11:21:50,831 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map NameNodeRetryCache
> 2015-04-28 11:21:50,831 INFO org.apache.hadoop.util.GSet: VM type       =
> 64-bit
> 2015-04-28 11:21:50,831 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory 889 MB = 273.1 KB
> 2015-04-28 11:21:50,831 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^15 = 32768 entries
> 2015-04-28 11:21:50,833 INFO
> org.apache.hadoop.hdfs.server.namenode.NNConf: ACLs enabled? false
> 2015-04-28 11:21:50,833 INFO
> org.apache.hadoop.hdfs.server.namenode.NNConf: XAttrs enabled? true
> 2015-04-28 11:21:50,833 INFO
> org.apache.hadoop.hdfs.server.namenode.NNConf: Maximum size of an xattr:
> 16384
> 2015-04-28 11:21:50,834 WARN org.apache.hadoop.hdfs.server.common.Storage:
> Storage directory /tmp/hadoop-anand_vihar/dfs/name does not exist
> 2015-04-28 11:21:50,835 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Encountered exception
> loading fsimage
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /tmp/hadoop-anand_vihar/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
>     at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:313)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1020)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:739)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:536)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:595)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:762)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:746)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1438)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1504)
> 2015-04-28 11:21:50,869 INFO org.mortbay.log: Stopped HttpServer2$
> SelectChannelConnectorWithSafeStartup@0.0.0.0:50070
> 2015-04-28 11:21:50,969 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
> 2015-04-28 11:21:50,970 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 2015-04-28 11:21:50,970 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 2015-04-28 11:21:50,970 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
> org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:
> Directory /tmp/hadoop-anand_vihar/dfs/name is in an inconsistent state:
> storage directory does not exist or is not accessible.
>     at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:313)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1020)
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:739)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:536)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:595)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:762)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:746)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1438)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1504)
> 2015-04-28 11:21:50,972 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
> 2015-04-28 11:21:50,973 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at Latitude-E5540/127.0.1.1
> ************************************************************/
>
> Regards,
>
> Anand Murali
> 11/7, 'Anand Vihar', Kandasamy St, Mylapore
> Chennai - 600 004, India
> Ph: (044)- 28474593/ 43526162 (voicemail)
>
>
>
>
>
>
>
>

Mime
View raw message