hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From F21 <f21.gro...@gmail.com>
Subject Re: HBase process not starting and uses a lot of CPU
Date Thu, 17 Dec 2015 04:31:28 GMT
I didn't create ` hbase` user, but because I installed the hadoop-client 
from HortonWorks's apt-repo, it created the `hdfs` user, so I thought I 
might just use that.

What is really interesting with the issue I am seeing is that there is 
no info in the logs besides the following in the .out file:

Thu Dec 17 02:37:26 UTC 2015 Starting master on hmaster2
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23668
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1048576
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Is there anyway I can make it show more information to debug the system?




On 17/12/2015 3:15 PM, Ted Yu wrote:
> Interesting, looks like you launched hbase daemon as user hdfs.
> Don't you have user 'hbase' ?
>
> Below is an example of region server process (non-docker). You can see
> that HRegionServer was run by bin/java
>
> hbase    13878  0.2  4.3 3963016 703116 ?      Sl   Dec15   5:26
> /usr/jdk64/jdk1.8.0_60/bin/java -Dproc_regionserver
> -XX:OnOutOfMemoryError=kill -9 %p -Dhdp.version=2.3.4.0-3471
> -XX:+UseConcMarkSweepGC -XX:ErrorFile=/grid/0/log/hbase/hs_err_pid%p.log
> -Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_client_jaas.conf
> -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -Xloggc:/grid/0/log/hbase/gc.log-201512152150
> -Xmn408m -XX:CMSInitiatingOccupancyFraction=70 -Xms2048m -Xmx2048m
> -Djava.security.auth.login.config=/usr/hdp/current/hbase-regionserver/conf/hbase_regionserver_jaas.conf
> -Dhbase.log.dir=/grid/0/log/hbase
> -Dhbase.log.file=hbase-hbase-regionserver-ted-hbase-devenv-4.log
> -Dhbase.home.dir=/usr/hdp/current/hbase-regionserver/bin/..
> -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA
> -Djava.library.path=:/usr/hdp/2.3.4.0-3471/hadoop/lib/native/Linux-amd64-64:/grid/0/hdp/2.3.4.0-3471/hadoop/lib/native
> -Dhbase.security.logger=INFO,RFAS
> org.apache.hadoop.hbase.regionserver.HRegionServer start
>
> FYI
>
> On Wed, Dec 16, 2015 at 8:05 PM, F21 <f21.groups@gmail.com> wrote:
>
>> Thanks.
>>
>> Here are the results of jstack:
>>
>> $ ps aux | grep master
>> root          1  0.0  0.0  18140  3040 ?        Ss   02:37   0:01 bash
>> /opt/hbase/bin/run-master.sh
>> hdfs        130  0.0  0.0   9644  2716 ?        S    02:40   0:00 bash
>> /opt/hbase/bin/hbase-daemon.sh --config /opt/hbase/conf foreground_start
>> master
>> hdfs        144  0.0  0.0   9640  2720 ?        S    02:40   0:00 bash
>> /opt/hbase/bin/../bin/hbase --config /opt/hbase/conf master start
>> hdfs        155  0.0  0.0   9640  1900 ?        S    02:40   0:00 bash
>> /opt/hbase/bin/../bin/hbase --config /opt/hbase/conf master start
>> root       5195  0.0  0.0   8868   776 ?        S+   04:02   0:00 grep
>> --color=auto master
>>
>> $ jps
>> 5321 Jps
>> $ jstack 130
>> 130: Unable to open socket file: target process not responding or HotSpot
>> VM not loaded
>> The -F option can be used when the target process is not responding
>>
>> $ jstack 144
>> 144: Unable to open socket file: target process not responding or HotSpot
>> VM not loaded
>> The -F option can be used when the target process is not responding
>>
>> $ jstack 155
>> 155: Unable to open socket file: target process not responding or HotSpot
>> VM not loaded
>> The -F option can be used when the target process is not responding
>>
>> $ jstack -F 130
>> Attaching to process ID 130, please wait...
>> Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
>> cannot open binary file
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
>>      at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
>>      at
>> sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
>>      at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
>>      at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
>>      at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
>>      at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>>      at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>>      at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>      at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>      at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>      at java.lang.reflect.Method.invoke(Method.java:497)
>>      at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
>>      at sun.tools.jstack.JStack.main(JStack.java:106)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
>> file
>>      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
>> Method)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
>>
>> $ jstack -F 144
>> Attaching to process ID 144, please wait...
>> Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
>> cannot open binary file
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
>>      at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
>>      at
>> sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
>>      at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
>>      at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
>>      at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
>>      at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>>      at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>>      at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>      at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>      at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>      at java.lang.reflect.Method.invoke(Method.java:497)
>>      at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
>>      at sun.tools.jstack.JStack.main(JStack.java:106)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
>> file
>>      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
>> Method)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
>>
>> $ jstack -F 155
>> Attaching to process ID 155, please wait...
>> Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException:
>> cannot open binary file
>> sun.jvm.hotspot.debugger.DebuggerException:
>> sun.jvm.hotspot.debugger.DebuggerException: cannot open binary file
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:163)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:278)
>>      at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:671)
>>      at
>> sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:611)
>>      at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:337)
>>      at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304)
>>      at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:140)
>>      at sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
>>      at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
>>      at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
>>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>      at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>      at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>      at java.lang.reflect.Method.invoke(Method.java:497)
>>      at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
>>      at sun.tools.jstack.JStack.main(JStack.java:106)
>> Caused by: sun.jvm.hotspot.debugger.DebuggerException: cannot open binary
>> file
>>      at sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native
>> Method)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.access$100(LinuxDebuggerLocal.java:62)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:269)
>>      at
>> sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:138)
>>
>>
>>
>> On 17/12/2015 3:01 PM, Ted Yu wrote:
>>
>>> ps aux | grep aster
>>>
>>


Mime
View raw message