cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: Strange bug? "spam" in management log files...
Date Fri, 12 Jun 2015 14:09:59 GMT
I'm sorry for spaming,

it seems that in my db.properties file on second MGMT srever, I had
127.0.0.1 as the Cluster IP.
After this was changed to real IP address, it seems now that I dont have
those "spam" log messages, seems all fine for some hours.



On 5 June 2015 at 16:24, Andrija Panic <andrija.panic@gmail.com> wrote:

> Hi,
>
> any hint on how to proceed ?
>
> on haproxy I see rougly 50%/50% sessions across 2 backend servers.
>  But inside DB, it all points to the one mgmt_server_ip...
>
> Thanks,
> Andrija
>
> On 4 June 2015 at 19:27, Andrija Panic <andrija.panic@gmail.com> wrote:
>
>> And if of any help another hint:
>>
>> while Im having this lines sent to logs in high volume...if I stop second
>> mgmt server, first one (that is making all these lines, doesnt stop to make
>> them), so log is still heavily writen to - only when I also restart mgmt on
>> 1st node (2nd node is down), then these log lines dissapear.
>>
>> Thx
>>
>> On 4 June 2015 at 19:19, Andrija Panic <andrija.panic@gmail.com> wrote:
>>
>>> And I could add - these lines (in this volume) only appears on first
>>> mgmt server (Actually I have 2 separate, but identical ACS installations,
>>> and same behaviour).
>>>
>>> On 4 June 2015 at 19:18, Andrija Panic <andrija.panic@gmail.com> wrote:
>>>
>>>> Just checked, in the HOSTS table, all agents are connected (via
>>>> haproxy) to the first mgmt server...I just restarted haproxy, and still
>>>> inside the DB, it says same mgmt_server_id for all agents - which is not
>>>> really true.
>>>>
>>>> Actually, on the haproxy itslef (statistics page) I can see almoust
>>>> 50%-50% distribution across 2 backends - which means by haproxy it should
>>>> be fine.
>>>> total 18 agents, 10 goes to 1 backend, 8 goes to other backend (ACS
>>>> mgmt server)
>>>>
>>>> This is our haproxy config, I think it's fine, but... DB says
>>>> differently, althouh haproxy statistick say all fine
>>>>
>>>> ### ACS 8250
>>>> #######################################################################################
>>>> frontend front_ACS_8250 10.20.10.100:8250
>>>>         option tcplog
>>>>         mode tcp
>>>>         default_backend back_8250
>>>> backend back_8250
>>>>         mode tcp
>>>>         balance source
>>>>         server acs1_8250 10.20.10.7:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>         server acs2_8250 10.20.10.8:8250 check port 8250 inter 2000
>>>> rise 3 fall 3
>>>>
>>>> ##################################################################################################
>>>>
>>>> Any info on how to proceed with this, since because of these lines, it
>>>> makes mgmt logs almoust unreadable... :(
>>>>
>>>> Thanks,
>>>> Andrija
>>>>
>>>> On 4 June 2015 at 19:00, Andrija Panic <andrija.panic@gmail.com> wrote:
>>>>
>>>>> Thanks Koushik,
>>>>>
>>>>> I will check and let you know - but 11GB log file for 10h ?  I dont
>>>>> expect this is expected :)
>>>>> I understand that the message is there because of setup, just an awful
>>>>> lot of lines....
>>>>>
>>>>> Will check thx for the help !
>>>>>
>>>>> Andrija
>>>>>
>>>>> On 4 June 2015 at 18:53, Koushik Das <koushik.das@citrix.com> wrote:
>>>>>
>>>>>> This is expected in a clustered MS setup. What is the distribution
of
>>>>>> HV hosts across these MS (check host table in db for MS id)? MS owning
the
>>>>>> HV host processes all commands for that host.
>>>>>> Grep for the sequence numbers (for e.g. 73-7374644389819187201) in
>>>>>> both MS logs to correlate.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 04-Jun-2015, at 8:30 PM, Andrija Panic <andrija.panic@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> > Hi,
>>>>>> >
>>>>>> > I have 2 ACS MGMT servers, loadbalanced properly (AFAIK), and
>>>>>> sometimes it
>>>>>> > happens that on the first node, we have extremem number of folowing
>>>>>> line
>>>>>> > entries in the log fie, which causes many GB log in just few
hours
>>>>>> or less:
>>>>>> > (as you can see here they are not even that frequent, but
>>>>>> sometimes, it
>>>>>> > gets really crazy with the speed/numer logged per seconds:
>>>>>> >
>>>>>> > 2015-06-04 16:55:04,089 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-29:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-28:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,129 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-8:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-26:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,169 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-30:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-27:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,209 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-2:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-4:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,249 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-7:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-3:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,289 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-5:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,329 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-1:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,330 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-15:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-11:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,369 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-17:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-14:null) Seq 1-3297479352165335041: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> > 2015-06-04 16:55:04,409 DEBUG [c.c.a.m.ClusteredAgentManagerImpl]
>>>>>> > (AgentManager-Handler-12:null) Seq 73-7374644389819187201: MgmtId
>>>>>> > 90520745449919: Resp: Routing to peer
>>>>>> >
>>>>>> >
>>>>>> > We have haproxy VIP, to which SSVM connects, and all cloudstack
>>>>>> agents
>>>>>> > (agent.properties file).
>>>>>> >
>>>>>> > Any suggestions, how to avoid this - I noticed when I turn off
>>>>>> second ACS
>>>>>> > MGMT server, and then reboot first one (restart
>>>>>> cloudstack-management) it
>>>>>> > stops and behaves nice :)
>>>>>> >
>>>>>> > This is ACS 4.5.1, Ubuntu 14.04 for mgmt nodes.
>>>>>> >
>>>>>> > Thanks,
>>>>>> > --
>>>>>> >
>>>>>> > Andrija Panić
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Andrija Panić
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Andrija Panić
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>



-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message