qpid-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Moravec (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (QPID-5072) [C++ broker] SessionManager does not forget sessions when broker drops connection after journal error, leading to memory leak
Date Fri, 04 Oct 2013 13:13:43 GMT

     [ https://issues.apache.org/jira/browse/QPID-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pavel Moravec resolved QPID-5072.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.25
         Assignee: Pavel Moravec  (was: Kim van der Riet)

This issue does not exhibit in upstream qpid (revision 1528730 checked), where the broker
allows the new session to be attached (but disconnects the connection as attempting to enqueue
to full journal). Some independent change since 0.22 apparently fixed the issue, in another
way than my proposed patch.

There is a "false alarm/error" observed here, as errors "error Channel exception: not-attached:
Channel 2 is not attached" are still raised. But that is proper behaviour, as:

1) client sends session.attach
2) immediatelly it sends some messages
3) it sets its command point
4) ..
5) broker gets the frames in one bunch, so it:
  a) attaches the session,
  b) gets some command point update (ok)
  c) gets some message - and drops the connection due to journal full
  d) gets another session-related data but on already detached channel - so now the error
is raised

The 5d) producing the errors should not be "fixed" in not raising the errors. As the sequence
could be also:
1) client sends some message
2) broker drops the connection
3) clients sends something more

The broker can't distinguish between these two scenarios, so it makes sense to raise the error
log in every such case.

> [C++ broker] SessionManager does not forget sessions when broker drops connection after
journal error, leading to memory leak
> -----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: QPID-5072
>                 URL: https://issues.apache.org/jira/browse/QPID-5072
>             Project: Qpid
>          Issue Type: Bug
>          Components: C++ Broker
>    Affects Versions: 0.22
>            Reporter: Pavel Moravec
>            Assignee: Pavel Moravec
>             Fix For: 0.25
>
>         Attachments: bz991413.patch, memoryLeak-in-sessionManager.sh, test_00903524_threaded.java
>
>
> Description of problem:
> Broker rejects session re-subscription (on a new AMQP+TCP connection) after the previous
was dropped due to a journal error. This bug in broker:
> - is in fact a memory leak, as session manager map is not cleared when requested
> - it affects Java client during failover, when original journal problem has been already
resolved, but the client can't even attach the same session.
> Version-Release number of selected component (if applicable):
> any (teste 0.18 and also 0.22 broker)
> How reproducible:
> 100%
> Steps to Reproduce:
> 1) run attached Java reproducer on a fresh broker:
> java test_00903524_threaded 1
> 2) check qpidd logs for errors
> Actual results:
> 2013-08-02 11:39:04 [Protocol] error Unexpected exception: Enqueue capacity threshold
exceeded on queue "testQueue". (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/legacystore/JournalImpl.cpp:594)
> 2013-08-02 11:39:04 [Protocol] error Connection qpid.10.34.1.141:5672-10.34.1.141:42635
closed by error: Enqueue capacity threshold exceeded on queue "testQueue". (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/legacystore/JournalImpl.cpp:594)(501)
> 2013-08-02 11:39:14 [Protocol] error Channel exception: session-busy: Session already
attached: guest@QPID.36332225-5c5b-4077-b8ec-820555253a89 (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/broker/SessionManager.cpp:55)
> 2013-08-02 11:39:14 [Protocol] error Channel exception: not-attached: Channel 1 is not
attached (/builddir/build/BUILD/qpid-0.22/cpp/src/qpid/amqp_0_10/SessionHandler.cpp:39)
> ..
> (few thousands of the latest error)
> Expected results:
> just the journal error be seen (and multiple times), not the "Session already attached"
one or "Channel 1 is not attached" one.
> Additional info:
> The broker management _deletes_ the session, as the broker logs:
> 2013-08-02 11:39:04 [Management] trace Management object marked deleted: org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89
>    org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89 (deleted)
> 2013-08-02 11:39:10 [Management] trace Deleting V1 properties 0-68-1--14(org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89)
len=164
> 2013-08-02 11:39:10 [Management] trace Deleting V1 statistics 0-68-1--14(org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89)
len=127
> 2013-08-02 11:39:10 [Management] trace Deleting V2 map={_create_ts:1375436343054221056,
_delete_ts:1375436344249095890, _object_id:{_agent_epoch:68, _object_name:org.apache.qpid.broker:session:36332225-5c5b-4077-b8ec-820555253a89},
_schema_id:{_class_name:session, _hash:1aaa08d0-c118-ff78-0956-47b9ac9c6849, _package_name:org.apache.qpid.broker,
_type:_data}, _update_ts:1375436343054221056, _values:{TxnCommits:0, TxnCount:0, TxnRejects:0,
TxnStarts:0, attached:True, channelId:0, clientCredit:0, connectionRef:{_agent_epoch:68, _object_name:org.apache.qpid.broker:connection:qpid.10.34.1.141:5672-10.34.1.141:42635},
detachedLifespan:0, name:36332225-5c5b-4077-b8ec-820555253a89, unackedMessages:0, vhostRef:{_object_name:org.apache.qpid.broker:vhost:org.apache.qpid.broker:broker:amqp-broker,/}}}
> But session manager inside does not forget the sessions/channels..
> Reproduction for memory leak:
> run attached script
> Weird is, even fixing the bug for Java client, some steady memory increase is still present..



--
This message was sent by Atlassian JIRA
(v6.1#6144)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org


Mime
View raw message