cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Finko <denis.fi...@ecommerce.com>
Subject CloudStack 4.2 web interface is not responding during handling big logs
Date Thu, 14 Nov 2013 08:52:47 GMT
Hello Cloudstack community,

During last week on CloudStack 4.2 I have seen following issue:

CloudStack web interface is not responding in browser. So, I looked to
server side for determining what's going on.

# cat /etc/redhat-release
CentOS release 6.4 (Final)

top - 03:09:43 up 8 days, 14:59,  4 users,  load average: 0.99, 0.97, 0.99
Tasks: 234 total,   1 running, 233 sleeping,   0 stopped,   0 zombie
Cpu0  :*100.0%*us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi, 
0.0%si,  0.0%st
Cpu1  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu2  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu3  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu4  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu5  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu6  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
Cpu7  :  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si, 
0.0%st
*Mem:  16333844k total,  9968012k used,  6365832k free,   169624k buffers*
Swap:  2097144k total,      976k used,  2096168k free,  5900544k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ 
COMMAND                                                                                  
                                      

*30869* cloud     20   0 7859m *2.2g*  23m S *100.4* 14.0 430:30.61
java                                                                                     
                                     

   36 root      20   0     0    0    0 S  1.0  0.0   0:48.13
events/1                                                                                 
                                      

 3833 root      20   0 15160 1436 1000 R  1.0  0.0   0:00.53
top                                                                                      
                                      

    1 root      20   0 19228 1464 1256 S  0.0  0.0   0:02.48 init

*java* process use 100.4% CPU and 2.2G of memory.

lsof -p 30869  | sort -nk7

COMMAND   PID  USER   FD   TYPE   DEVICE   *SIZE/OFF*    NODE NAME
java    30869 cloud  284r    REG                8,3    7707959 1841612
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/aws-java-sdk-1.3.22.jar
java    30869 cloud  mem    REG                8,3    7707959 1841612
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/aws-java-sdk-1.3.22.jar
java    30869 cloud  mem    REG                8,3   10252136 1706534
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server/libjvm.so
java    30869 cloud    3r      REG                8,3   60484682 1706608
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
java    30869 cloud  mem    REG                8,3   60484682 1706608
/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar
java    30869 cloud  mem    REG                8,3   99158576 1183591
/usr/lib/locale/locale-archive
java    30869 cloud  271w   REG                8,3  *395727946*  265565
/var/log/cloudstack/management/access_log.2013-11-13.txt
java    30869 cloud  462w   REG                8,3  637460040  265558
/var/log/cloudstack/management/apilog.log.2013-11-13
java    30869 cloud  467w   REG                8,3 *1144568381*  265563
/var/log/cloudstack/management/management-server.log.2013-11-13

As you can see above *java* process use a lot of memory for log files.

# ls -lha /var/log/cloudstack/management/management-server.log*
-rw-rw-r--. 1 cloud cloud 179K Nov 14 02:59
/var/log/cloudstack/management/management-server.log
-rw-rw-r--. 1 cloud cloud 1.1M Oct 25 00:00
/var/log/cloudstack/management/management-server.log.2013-10-24.gz
-rw-rw-r--. 1 cloud cloud 1.4M Oct 26 00:00
/var/log/cloudstack/management/management-server.log.2013-10-25.gz
-rw-rw-r--. 1 cloud cloud 1.2M Oct 27 00:00
/var/log/cloudstack/management/management-server.log.2013-10-26.gz
-rw-rw-r--. 1 cloud cloud 1.2M Oct 28 00:00
/var/log/cloudstack/management/management-server.log.2013-10-27.gz
-rw-rw-r--. 1 cloud cloud 1.6M Oct 29 00:00
/var/log/cloudstack/management/management-server.log.2013-10-28.gz
-rw-rw-r--. 1 cloud cloud 2.0M Oct 30 00:00
/var/log/cloudstack/management/management-server.log.2013-10-29.gz
-rw-rw-r--. 1 cloud cloud 2.2M Oct 31 00:00
/var/log/cloudstack/management/management-server.log.2013-10-30.gz
-rw-rw-r--. 1 cloud cloud 2.0M Nov  1 00:00
/var/log/cloudstack/management/management-server.log.2013-10-31.gz
-rw-rw-r--. 1 cloud cloud 2.7M Nov  2 00:00
/var/log/cloudstack/management/management-server.log.2013-11-01.gz
-rw-rw-r--. 1 cloud cloud 3.0M Nov  3 00:00
/var/log/cloudstack/management/management-server.log.2013-11-02.gz
-rw-rw-r--. 1 cloud cloud 1.2M Nov  4 00:00
/var/log/cloudstack/management/management-server.log.2013-11-03.gz
-rw-rw-r--. 1 cloud cloud  34M Nov  5 00:01
/var/log/cloudstack/management/management-server.log.2013-11-04.gz
-rw-rw-r--. 1 cloud cloud  35M Nov  6 00:00
/var/log/cloudstack/management/management-server.log.2013-11-05.gz
-rw-rw-r--. 1 cloud cloud  15M Nov  7 00:00
/var/log/cloudstack/management/management-server.log.2013-11-06.gz
-rw-rw-r--. 1 cloud cloud  30M Nov  8 00:00
/var/log/cloudstack/management/management-server.log.2013-11-07.gz
-rw-rw-r--. 1 cloud cloud 1.6M Nov  9 00:00
/var/log/cloudstack/management/management-server.log.2013-11-08.gz
-rw-rw-r--. 1 cloud cloud 1.3M Nov 10 00:00
/var/log/cloudstack/management/management-server.log.2013-11-09.gz
-rw-rw-r--. 1 cloud cloud  18M Nov 11 00:00
/var/log/cloudstack/management/management-server.log.2013-11-10.gz
-rw-rw-r--. 1 cloud cloud  12M Nov 12 00:00
/var/log/cloudstack/management/management-server.log.2013-11-11.gz
-rw-rw-r--. 1 cloud cloud 6.4M Nov 13 00:00
/var/log/cloudstack/management/management-server.log.2013-11-12.gz
*-rw-rw-r--. 1 cloud cloud 1.1G Nov 13 23:45
/var/log/cloudstack/management/management-server.log.2013-11-13*


# ls -lah /var/log/cloudstack/management/apilog*
-rw-rw-r--. 1 cloud cloud  336 Nov 14 00:27
/var/log/cloudstack/management/apilog.log
-rw-rw-r--. 1 cloud cloud 4.6K Oct 24 04:32
/var/log/cloudstack/management/apilog.log.2013-10-18.gz
-rw-rw-r--. 1 cloud cloud 340K Oct 25 03:03
/var/log/cloudstack/management/apilog.log.2013-10-24.gz
-rw-rw-r--. 1 cloud cloud  82K Oct 28 03:04
/var/log/cloudstack/management/apilog.log.2013-10-25.gz
-rw-rw-r--. 1 cloud cloud 708K Oct 29 00:09
/var/log/cloudstack/management/apilog.log.2013-10-28.gz
-rw-rw-r--. 1 cloud cloud 1.1M Oct 30 00:09
/var/log/cloudstack/management/apilog.log.2013-10-29.gz
-rw-rw-r--. 1 cloud cloud 1.5M Oct 31 00:09
/var/log/cloudstack/management/apilog.log.2013-10-30.gz
-rw-rw-r--. 1 cloud cloud 1.7M Nov  1 00:04
/var/log/cloudstack/management/apilog.log.2013-10-31.gz
-rw-rw-r--. 1 cloud cloud 1.7M Nov  2 00:00
/var/log/cloudstack/management/apilog.log.2013-11-01.gz
-rw-rw-r--. 1 cloud cloud 2.6M Nov  3 00:11
/var/log/cloudstack/management/apilog.log.2013-11-02.gz
-rw-rw-r--. 1 cloud cloud  60K Nov  4 02:24
/var/log/cloudstack/management/apilog.log.2013-11-03.gz
-rw-rw-r--. 1 cloud cloud  21M Nov  5 00:00
/var/log/cloudstack/management/apilog.log.2013-11-04.gz
-rw-rw-r--. 1 cloud cloud  23M Nov  6 02:12
/var/log/cloudstack/management/apilog.log.2013-11-05.gz
-rw-rw-r--. 1 cloud cloud  11M Nov  7 02:34
/var/log/cloudstack/management/apilog.log.2013-11-06.gz
-rw-rw-r--. 1 cloud cloud  20M Nov  8 02:54
/var/log/cloudstack/management/apilog.log.2013-11-07.gz
-rw-rw-r--. 1 cloud cloud 1.9M Nov  9 09:07
/var/log/cloudstack/management/apilog.log.2013-11-08.gz
-rw-rw-r--. 1 cloud cloud 268K Nov 10 05:11
/var/log/cloudstack/management/apilog.log.2013-11-09.gz
-rw-rw-r--. 1 cloud cloud  12M Nov 11 00:00
/var/log/cloudstack/management/apilog.log.2013-11-10.gz
-rw-rw-r--. 1 cloud cloud 7.2M Nov 12 00:00
/var/log/cloudstack/management/apilog.log.2013-11-11.gz
-rw-rw-r--. 1 cloud cloud 3.7M Nov 13 03:13
/var/log/cloudstack/management/apilog.log.2013-11-12.gz
*-rw-rw-r--. 1 cloud cloud 608M Nov 13 22:11
/var/log/cloudstack/management/apilog.log.2013-11-13*

For solving this issue I have to:

# kill -9 30869

# /etc/init.d/cloudstack-management restart
Stopping cloudstack-management:                            [  OK  ]
Starting cloudstack-management:                            [  OK  ]

And after that action web interface is working fine again.

Could you please tell me how to resolve or avoid this issue.

Thank you!




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message