hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-4202) Check filesystem permissions on startup
Date Tue, 16 Aug 2011 20:26:30 GMT

     [ https://issues.apache.org/jira/browse/HBASE-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-4202:
-------------------------

    Labels: noob  (was: )

Marking as noob.  Would be sweet if someone tried this on 0.90.x hbase and if still an issue,
make a change so we exit rather than do the pathological dance Matthias describes above.

> Check filesystem permissions on startup
> ---------------------------------------
>
>                 Key: HBASE-4202
>                 URL: https://issues.apache.org/jira/browse/HBASE-4202
>             Project: HBase
>          Issue Type: Bug
>          Components: regionserver
>    Affects Versions: 0.20.4
>         Environment: debian squeeze
>            Reporter: Matthias Hofschen
>              Labels: noob
>
> We added a new node to a 44 node cluster starting the datanode, mapred and regionserver
processes on it. The Unix filesystem was configured incorrectly, i.e. /tmp was not writable
to processes. All three processes had issues with this. Datanode and mapred shutdown on exception.
> Regionserver did not stop, in fact reported to master that its up without regions. So
master assigned regions to it. Regionserver would not accept them, resulting in a constant
assign, reject, reassign cycle, that put many regions into a state of not being available.
There are no logs about this, but we could observer the regioncount fluctuate by hundredths
of regions and the application throwing many NotServingRegion exceptions.  
> In fact to the master process the regionserver looked fine, so it was trying to send
regions its way. Regionserver rejected them. So the master/balancer was going into a assign/reassign
cycle destabilizing the cluster. Many puts and gets simply failed with NotServingRegionExceptions
and took a long time to complete.
> Exception from regionserver:
> 2011-08-06 23:57:13,953 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Got
ZooKeeper event, state: SyncConnected, type: NodeCreated, path: /hbase/master
> 2011-08-06 23:57:13,957 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Telling
master at 17.1.0.1:60000 that we are up
> 2011-08-06 23:57:13,957 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Telling
master at 17.1.0.1:60000 that we are up
> 2011-08-07 00:07:39.648::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
> 2011-08-07 00:07:39.712::INFO:  jetty-6.1.14
> 2011-08-07 00:07:39.742::WARN:  tmpdir
> java.io.IOException: Permission denied
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.checkAndCreate(File.java:1704)
>         at java.io.File.createTempFile(File.java:1792)
>         at java.io.File.createTempFile(File.java:1828)
>         at org.mortbay.jetty.webapp.WebAppContext.getTempDirectory(WebAppContext.java:745)
>         at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:458)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>         at org.mortbay.jetty.Server.doStart(Server.java:222)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:461)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.startServiceThreads(HRegionServer.java:1168)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.init(HRegionServer.java:792)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:430)
>         at java.lang.Thread.run(Thread.java:619)
> Exception from datanode:
> 2011-08-06 23:37:20,444 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50075
> 2011-08-06 23:37:20,444 INFO org.mortbay.log: jetty-6.1.14
> 2011-08-06 23:37:20,469 WARN org.mortbay.log: tmpdir
> java.io.IOException: Permission denied
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.checkAndCreate(File.java:1704)
>         at java.io.File.createTempFile(File.java:1792)
>         at java.io.File.createTempFile(File.java:1828)
>         at org.mortbay.jetty.webapp.WebAppContext.getTempDirectory(WebAppContext.java:745)
>         at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:458)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>         at org.mortbay.jetty.Server.doStart(Server.java:222)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:463)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:384)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:225)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1309)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1264)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1272)
>         at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394)
> 2011-08-06 23:37:20,471 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

> /************************************************************
> SHUTDOWN_MSG: Shutting down DataNode at hdp1122/17.1.0.22
> ************************************************************/
> Exception from tasktracker:
> 2011-08-06 23:33:50,380 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50060
> 2011-08-06 23:33:50,380 INFO org.mortbay.log: jetty-6.1.14
> 2011-08-06 23:33:50,415 WARN org.mortbay.log: tmpdir
> java.io.IOException: Permission denied
>         at java.io.UnixFileSystem.createFileExclusively(Native Method)
>         at java.io.File.checkAndCreate(File.java:1704)
>         at java.io.File.createTempFile(File.java:1792)
>         at java.io.File.createTempFile(File.java:1828)
>         at org.mortbay.jetty.webapp.WebAppContext.getTempDirectory(WebAppContext.java:745)
>         at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:458)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>         at org.mortbay.jetty.Server.doStart(Server.java:222)
>         at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>         at org.apache.hadoop.http.HttpServer.start(HttpServer.java:463)
>         at org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:935)
>         at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2837)
> 2011-08-06 23:33:50,416 INFO org.apache.hadoop.mapred.TaskTracker: SHUTDOWN_MSG: 
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at hdp1122/17.1.0.22
> ************************************************************/

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message