cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anirban Chakraborty <abc...@juniper.net>
Subject Re: system vm disk space issue in ACS 4.3
Date Mon, 10 Mar 2014 23:30:39 GMT
Thanks for all the responses. I do not see cloud.log and cloud.out logs are zipped in /var/log
and /var/log/cloud respectively. Only file that was zipped was cron.log. The two largest files
are:
cloud.out.2 with following:
—
+ keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 console=hvc0 template=domP
type=secstorage host=10.84.58.252 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource
instance=SecStorage sslcopy=true role=templateProcessor mtu=1500'
+ for i in '$CMDLINE'
++ cut -s -d= -f1
++ echo eth2ip=10.84.59.176
+ KEY=eth2ip
++ cut -s -d= -f2
++ echo eth2ip=10.84.59.176
+ VALUE=10.84.59.176
+ '[' eth2ip == '' ']'
+ case $KEY in
+ keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 console=hvc0 template=domP
type=secstorage host=10.84.58.252 port=8250 name=s-1-VM zone=1 pod=1 guid=s-1-VM resource=org.apache.cloudstack.sJava
HotSpot(TM) Client VM warning: Insufficient space for shared memory file:
   /tmp/hsperfdata_root/8004
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:WARN No such property [maxFileSize] in org.apache.log4j.rolling.RollingFileAppender.
log4j:WARN No such property [maxBackupIndex] in org.apache.log4j.rolling.RollingFileAppender.
log4j:WARN Please set a rolling policy for the RollingFileAppender named 'FILE3'
08:31:55,320  INFO AgentShell:318 - Agent started
log4j:ERROR Failed to flush writer,
java.io.IOException: No space left on device
—
and, repetition of following in /var/log/cloud.log.1 
—
2014-03-01 07:05:08,607 DEBUG [cloud.utils.ProcessUtil] (main:null)   PID TTY          TIME
CMD
 3938 ?        00:00:36 java

2014-03-01 07:05:08,607 ERROR [cloud.agent.AgentShell] (main:null) Unable to start agent:
Java process is being started twice.  If this is not true, remove /var/run/agent.SecStorage.pid
2014-03-01 07:05:19,028 INFO  [cloud.agent.AgentShell] (main:null) Agent started
2014-03-01 07:05:19,030 INFO  [cloud.agent.AgentShell] (main:null) Implementation Version
is 4.3.0-SNAPSHOT
2014-03-01 07:05:19,030 INFO  [cloud.agent.AgentShell] (main:null) agent.properties found
at /usr/local/cloud/systemvm/conf/agent.properties
2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found property: instance
2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found property: resource
2014-03-01 07:05:19,038 INFO  [cloud.agent.AgentShell] (main:null) Defaulting to using properties
file for storage
2014-03-01 07:05:19,039 INFO  [cloud.agent.AgentShell] (main:null) Defaulting to the constant
time backoff algorithm
2014-03-01 07:05:19,048 INFO  [cloud.utils.LogUtils] (main:null) log4j configuration found
at /usr/local/cloud/systemvm/conf/log4j-cloud.xml
2014-03-01 07:05:19,062 DEBUG [cloud.agent.AgentShell] (main:null) Checking to see if agent.SecStorage.pid
exists.
2014-03-01 07:05:19,064 DEBUG [cloud.utils.ProcessUtil] (main:null) environment.properties
could not be opened
2014-03-01 07:05:19,071 DEBUG [cloud.utils.ProcessUtil] (main:null) Executing: bash -c ps
-p 3938 
2014-03-01 07:05:19,077 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution is successful.

eventually the log gets full with,
2014-03-10 09:54:09,420 INFO  [utils.nio.NioClient] (Agent-Selector:null) Connecting to 10.84.58.252:8250
2014-03-10 09:54:36,916 WARN  [utils.nio.NioConnection] (Agent-Selector:null) Unable to connect
to remote: is there a server running on port 8250 
—
Looks like the agent on s-1-vm could not connect to the management server at some point of
time after the system vm startup and the log gets filled with above ‘Unable to connect’
messages.

Anirban

On Mar 7, 2014, at 11:37 PM, Rajesh Battala <rajesh.battala@citrix.com> wrote:

> AFAIK, log roation is enabled in the systemvm.
> Can you check whether the logs are getting zipped .?
> 
> -----Original Message-----
> From: Anirban Chakraborty [mailto:abchak@juniper.net] 
> Sent: Saturday, March 8, 2014 12:46 PM
> To: dev@cloudstack.apache.org
> Subject: system vm disk space issue in ACS 4.3
> 
> Hi All,
> 
> I am seeing system vm disk has no space left after running for few days. Cloudstack UI
shows the agent in v-2-VM in alert state, while agent state of s-1-VM shows blank (hyphen
in the UI).
> Both the system vms are running and ssh-able from the host. The log in s-1-Vm shows following
errors:
> 
> root@s-1-VM:~# grep 'Exception' /var/log/cloud/*.*
> /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device
> /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device
> 
> whereas logs in v-1-VM shows
> /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device
> /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device
> /var/log/cloud/cloud.out.3:07:18:00,547  INFO CSExceptionErrorCode:87 - Could not find
exception: com.cloud.exception.AgentControlChannelException in error code list for exceptions
> /var/log/cloud/cloud.out.3:com.cloud.exception.AgentControlChannelException: Unable to
post agent control request as link is not available
> 
> Looks like cloud agent is filling up the log, which is leading to the disk full state.
> 
> Is this a known issue? Thanks.
> 
> Anirban
> 
> 



Mime
View raw message