hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kartashov, Andy" <Andy.Kartas...@mpac.ca>
Subject datanode daemon
Date Thu, 25 Oct 2012 19:40:14 GMT
Guys,

I finally solved ALL the Errors: in  ...datanode*.log  after trying to start the node with
"service datanode start".
The errors were:
- conflicting NN DD ids - solved through reformatting NN.
- could not connect to 127.0.0.1:8020 - Connection refused - solved through correcting a typo
inside hdfs-site.xml under dfs.namenode.http-address; somehow had the default value i/o localhost.
(Running pseudo-mode)
- conf was pointing to the wrong sLink - solved by running alternatives -set hadoop-conf <conf.myconf>

However, when I run "service -status-all", still see that datanode [FAILED] message. All others,
NN, SNN, JT, TT are running [OK].


1.       Starting daemons, all seems OK:
Starting Hadoop datanode:                                  [  OK  ]
starting datanode, logging to /home/hadoop/logs/hadoop-root-datanode-ip-10-204-47-138.out
Starting Hadoop namenode:                                  [  OK  ]
starting namenode, logging to /home/hadoop/logs/hadoop-hdfs-namenode-ip-10-204-47-138.out
Starting Hadoop secondarynamenode:                         [  OK  ]
starting secondarynamenode, logging to /home/hadoop/logs/hadoop-hdfs-secondarynamenode-ip-10-204-47-138.out

2.
running service -status-all command and get:
Hadoop datanode is not running                             [FAILED]
Hadoop namenode is running                                 [  OK  ]
Hadoop secondarynamenode is running                        [  OK  ]

3.
Here is log file on DN:
2012-10-25 15:33:37,554 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = ip-10-204-47-138.ec2.internal/10.204.47.138
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.0.0-cdh4.1.1
STARTUP_MSG:   classpath = /etc/ha..........
...............................
..............................
2012-10-25 15:33:38,098 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data
should be specified as a URI in configuration files. Please update hdfs configuration.
2012-10-25 15:33:41,589 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties
from hadoop-metrics2.properties
2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2012-10-25 15:33:42,125 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics
system started
2012-10-25 15:33:42,204 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname
is ip-10-204-47-138.ec2.internal
2012-10-25 15:33:42,319 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming
server at /0.0.0.0:50010
2012-10-25 15:33:42,323 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwith
is 1048576 bytes/s
2012-10-25 15:33:42,412 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
2012-10-25 15:33:42,603 INFO org.apache.hadoop.http.HttpServer: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2012-10-25 15:33:42,607 INFO org.apache.hadoop.http.HttpServer: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2012-10-25 15:33:42,682 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened info
server at 0.0.0.0:50075
2012-10-25 15:33:42,690 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dfs.webhdfs.enabled
= false
2012-10-25 15:33:42,690 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
2012-10-25 15:33:42,690 INFO org.mortbay.log: jetty-6.1.26.cloudera.2
2012-10-25 15:33:43,601 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50075
2012-10-25 15:33:43,787 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
50020
2012-10-25 15:33:43,905 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server
at /0.0.0.0:50020
2012-10-25 15:33:43,917 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request
received for nameservices: null
2012-10-25 15:33:43,943 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices
for nameservices: <default>
2012-10-25 15:33:43,950 WARN org.apache.hadoop.hdfs.server.common.Util: Path /home/hadoop/dfs/data
should be specified as a URI in configuration files. Please update hdfs configuration.
2012-10-25 15:33:43,958 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering>
(storage id unknown) service to localhost/127.0.0.1:8020 starting to offer service
2012-10-25 15:33:44,297 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2012-10-25 15:33:44,304 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
2012-10-25 15:33:45,551 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:46,605 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:47,865 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:48,945 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:49,946 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:50,947 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:51,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:52,997 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:54,047 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: localhost/127.0.0.1:8020.
Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
2012-10-25 15:33:57,579 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/hadoop/dfs/data/in_use.lock
acquired by nodename 21886@ip-10-204-47-138.ec2.internal
2012-10-25 15:33:59,279 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled
2012-10-25 15:33:59,297 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage:
nsid=587246899;bpid=BP-443540453-10.204.47.138-1351175525331;lv=-40;nsInfo=lv=-40;cid=CID-55808532-08a8-447c-b5e7-e609f6c546ee;nsid=587246899;c=0;bpid=BP-443540453-10.204.47.138-1351175525331
2012-10-25 15:33:59,457 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Added volume - /home/hadoop/dfs/data/current
2012-10-25 15:33:59,583 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Registered FSDatasetState MBean
2012-10-25 15:33:59,680 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic
Directory Tree Verification scan starting at 1351195722679 with interval 21600000
2012-10-25 15:33:59,688 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl:
Adding block pool BP-443540453-10.204.47.138-1351175525331
2012-10-25 15:34:00,339 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-443540453-10.204.47.138-1351175525331
(storage id DS-1489896713-10.204.47.138-50010-1351175686521) service to localhost/127.0.0.1:8020
beginning handshake with NN
2012-10-25 15:34:00,485 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block
pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521)
service to localhost/127.0.0.1:8020 successfully registered with NN
2012-10-25 15:34:00,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode
localhost/127.0.0.1:8020 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL
of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block
pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521)
service to localhost/127.0.0.1:8020 trying to claim ACTIVE state with txid=2697
2012-10-25 15:34:01,106 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging
ACTIVE Namenode Block pool BP-443540453-10.204.47.138-1351175525331 (storage id DS-1489896713-10.204.47.138-50010-1351175686521)
service to localhost/127.0.0.1:8020
2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport
of 30 blocks took 8 msec to generate and 369 msecs for RPC and NN processing
2012-10-25 15:34:01,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report,
processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@d2f41a5
2012-10-25 15:34:01,496 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-443540453-10.204.47.138-1351175525331.
2012-10-25 15:34:01,516 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added
bpid=BP-443540453-10.204.47.138-1351175525331 to blockPoolScannerMap, new size=1
2012-10-25 15:34:04,939 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-2780693186250942847_1214
2012-10-25 15:34:04,962 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3001053775353836209_1049
2012-10-25 15:34:05,081 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-5830536650401518708_1776
2012-10-25 15:34:05,100 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7796621539450888414_1161
2012-10-25 15:34:05,110 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_3113923085061591346_1318
2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-3360339171714568163_1477
2012-10-25 15:34:05,111 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-8153671645708824194_1779
2012-10-25 15:34:05,112 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-1321321546165212517_1002
2012-10-25 15:34:05,116 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner:
Verification succeeded for BP-443540453-10.204.47.138-1351175525331:blk_-7769865739291371890_1479

Rgds,
AK47
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and
may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not
the intended recipient, please delete and contact the sender immediately. Please consider
the environment before printing this e-mail. AVIS : le pr?sent courriel et toute pi?ce jointe
qui l'accompagne sont confidentiels, prot?g?s par le droit d'auteur et peuvent ?tre couverts
par le secret professionnel. Toute utilisation, copie ou divulgation non autoris?e est interdite.
Si vous n'?tes pas le destinataire pr?vu de ce courriel, supprimez-le et contactez imm?diatement
l'exp?diteur. Veuillez penser ? l'environnement avant d'imprimer le pr?sent courriel

Mime
View raw message