Mailing-List: contact users-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@cloudstack.apache.org
From: "Vladimir Melnik" <v.melnik@uplink.ua>
To: <users@cloudstack.apache.org>
References: <20150512145638.GL17652@shagomer.uplink.tucha13.net>
In-Reply-To: <20150512145638.GL17652@shagomer.uplink.tucha13.net>
Subject: RE: The agent doesn't restart :(
Date: Tue, 12 May 2015 18:33:44 +0300
Message-ID: <060f01d08cc9$08753ac0$195fb040$@uplink.ua>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Thread-Index: AQG7w3+0yI0Zj0UxvwqIpfMnbHs4DJ2iCmKw
Content-Language: uk

Ah, now I see what's going on. The agent starts, but it doesn't work at all!

2015-05-12 18:12:49,493 INFO  [cloud.agent.AgentShell] (Thread-1:null) Agent
started
2015-05-12 18:12:49,495 INFO  [cloud.agent.AgentShell] (Thread-1:null)
Implementation Version is 4.2.1
2015-05-12 18:12:49,496 INFO  [cloud.agent.AgentShell] (Thread-1:null)
agent.properties found at /etc/cloudstack/agent/agent.properties
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: workers
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: port
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: pod
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: resource
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: private.network.device
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: zone
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: guid
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: guest.network.device
2015-05-12 18:12:49,498 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: cluster
2015-05-12 18:12:49,499 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: domr.scripts.dir
2015-05-12 18:12:49,499 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: local.storage.uuid
2015-05-12 18:12:49,499 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: public.network.device
2015-05-12 18:12:49,499 DEBUG [cloud.agent.AgentShell] (Thread-1:null) Found
property: host
2015-05-12 18:12:49,499 INFO  [cloud.agent.AgentShell] (Thread-1:null)
Defaulting to using properties file for storage
2015-05-12 18:12:49,500 INFO  [cloud.agent.AgentShell] (Thread-1:null)
Defaulting to the constant time backoff algorithm
2015-05-12 18:12:49,501 INFO  [cloud.utils.LogUtils] (Thread-1:null) log4j
configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2015-05-12 18:12:49,614 DEBUG [cloud.agent.AgentShell] (Thread-1:null)
Checking to see if agent.pid exists.
2015-05-12 18:12:49,622 DEBUG [cloud.utils.ProcessUtil] (Thread-1:null)
Executing: bash -c echo $PPID

And this is the last line I see in the log-file. The file /var/log/agent.pid
is being created, but it has zero length. So, the agent's process is still
"working" (I can see it by "ps"), but it doesn't do anything.

Any ideas on the cause of such an odd behavior?

-----Original Message-----
From: Vladimir Melnik [mailto:v.melnik@uplink.ua] 
Sent: Tuesday, May 12, 2015 5:57 PM
To: users@cloudstack.apache.org
Subject: The agent doesn't restart :(

Hello!

I encountered quite an odd problem: the agent doesn't restart on KVM host.

Here is what's going on.

When it starts for the first time it creates /var/run/agent.pid file, but
there are no numbers, the file has zero length.

When I restart it (e.g. by service cloudstack-agent restart command), it's
being shut down, but file isn't being removed. So it can't start again, here
is what I see in /var/log/cloudstack/agent/cloudstack-agent.out file:

2015-05-12 17:04:37,504{GMT} INFO  [cloud.agent.AgentShell] (Thread-1:)
Agent started
2015-05-12 17:04:37,506{GMT} INFO  [cloud.agent.AgentShell] (Thread-1:)
Implementation Version is 4.2.1
2015-05-12 17:04:37,507{GMT} INFO  [cloud.agent.AgentShell] (Thread-1:)
agent.properties found at /etc/cloudstack/agent/agent.properties
2015-05-12 17:04:37,508{GMT} INFO  [cloud.agent.AgentShell] (Thread-1:)
Defaulting to using properties file for storage
2015-05-12 17:04:37,509{GMT} INFO  [cloud.agent.AgentShell] (Thread-1:)
Defaulting to the constant time backoff algorithm
2015-05-12 17:04:37,510{GMT} INFO  [cloud.utils.LogUtils] (Thread-1:) log4j
configuration found at /etc/cloudstack/agent/log4j-cloud.xml
2015-05-12 17:04:37,626{GMT} ERROR [cloud.agent.AgentShell] (Thread-1:)
Unable to start agent: Java process is being started twice.  If this is not
true, remove /var/run/agent.pid Unable to start agent: Java process is being
started twice.  If this is not true, remove /var/run/agent.pid

I remove this file by hands and do "restart" again. The agent starts, but
/var/run/agent.pid has zero length again.

This server is just the same as other CentOS hosts (there are many of them
in my farm), but other hosts don't seem to be having this issue.

I opened a ticket (https://issues.apache.org/jira/browse/CLOUDSTACK-8456,
there are more details about the environment), but if anyone has some clues,
hints or ideas, please, share your thoughts on this topic. Any help will be
greatly appreciated! Thanks!

--
V.Melnik