qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eagy, Taylor" <te...@BlackbirdTech.com>
Subject RE: Python Connections Hang from Subprocesses
Date Thu, 08 Aug 2013 13:42:33 GMT
Like I said in the original post this is in a multiprocessing.Process() object which uses the
subprocess module to create a subprocess and runs the run() function in that subprocess.
You can see the documentation here: http://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process

I'm not using threads or the threading module. Thanks for the reminder about multiple sessions.
I had forgotten about that. This means I will have to share the same Connection object between
multiple processes. Does the Connection object serialize fine when transferring via the pipe?

-----Original Message-----
From: Bill Freeman [mailto:ke1g.nh@gmail.com]
Sent: Wednesday, August 07, 2013 5:51 PM
To: users
Subject: Re: Python Connections Hang from Subprocesses

By the way, I presume that you know that you can have multiple sessions on one connection?


On Wed, Aug 7, 2013 at 5:49 PM, Bill Freeman <ke1g.nh@gmail.com> wrote:

>
>
>
> On Wed, Aug 7, 2013 at 5:35 PM, Eagy, Taylor <teagy@blackbirdtech.com>wrote:
>
>>  What O/S are you running this on?
>>
>>
>>
>> Ubuntu 12.04 LTS
>>
>> What does the code that spawns the subprocess look like?
>>
>>
>>
>> So my Process objects have a run() method that is called as the
>> subprocess.
>>
>
> Ok.  Unless I miss my guess, this is not a subprocess.  This is a thread.
> Your object is probably a subclass or threading.Thread, and you never
> import subprocess anywhere.  If you ran "ps -alxww" in another
> terminal, there wouldn't be a line for more than one process.
>
>>
>>
>>
>> import qpid.messaging # at the top of the Python module
>>
>>
>>
>> def run():
>>
>> // some other non-qpid setup
>>
>>
>>
>> conn = qpid.messaging.Connection(qpid_endpoint)
>>
>> conn.open() <-- hangs here
>>
>> self.sess = conn.session()
>>
>> I have done this when there was already a connection to the broker,
>> but
> that other connection was a qmf.console connection, which does not use
> qpid.messaging under the hood.
>
> Someone else will have to speak to the thread safety of
> qpid.messaging, because we have now exceeded my experience in that area.
>
>
>>
>> Can you, even when the subprocess is hung, run something like spout,
>> drain, or qpid-config, e.g.;
>>
>> qpid-config queues
>>
>> from another console window on the same box?
>>
>>
>>
>> Yes, I can connect to the queue if I try to connect from another
>> module (which runs a new copy of the Python interpreter).
>>
>> ________________________________
>> From: Bill Freeman [ke1g.nh@gmail.com]
>> Sent: Wednesday, August 07, 2013 4:21 PM
>> To: users
>> Subject: Re: Python Connections Hang from Subprocesses
>>
>> Just in case it makes me (or someone else) think of something else:
>>
>>   What O/S are you running this on?
>>
>>   What does the code that spawns the subprocess look like?
>>
>>   Can you, even when the subprocess is hung, run something like
>> spout, drain, or qpid-config, e.g.;
>>
>>       qpid-config queues
>>
>>   from another console window on the same box?
>>
>> Bill
>>
>>
>> On Wed, Aug 7, 2013 at 3:13 PM, Eagy, Taylor <teagy@blackbirdtech.com
>> >wrote:
>>
>> > Oops. Let me clean it up a bit.
>> >
>> > def attach(self):
>> >
>> > """
>> > Attach to the remote endpoint.
>> > """
>> > if not self._connected:
>> >     self._connected = True
>> >     self._driver.start()
>> >     self._wakeup()
>> > self._ewait(lambda: self._transport_connected and not
>> > self._unlinked())
>> >
>> > def _ewait(self, predicate, timeout=None):
>> >
>> >
>> >     result = self._wait(lambda: self.error or predicate(), timeout)
>> > <--
>> It
>> > times out here.
>> >     self.check_error()
>> >     return result
>> >
>> >
>> > ________________________________
>> > From: Eagy, Taylor [teagy@BlackbirdTech.com]
>> > Sent: Wednesday, August 07, 2013 2:50 PM
>> > To: users@qpid.apache.org
>> > Subject: RE: Python Connections Hang from Subprocesses
>> >
>> > Bill thanks for your response. The subprocess definitely runs when
>> > I
>> don't
>> > try to connect to the broker. Running a HTTPServer in it works too.
>> > It
>> just
>> > doesn't connect to the qpid broker. I did a little digging and I
>> > believe this is where it is hanging:
>> >
>> > (In Connection class)
>> >
>> >  270    @synchronized
>> >
>> >  271 -<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints-pysrc.html#
>> >
>> >  def attach<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints.Connection-class.html#attach
>> > >(self):
>> >
>> >  272      """
>> >  273      Attach to the remote endpoint.
>> >  274      """
>> >  275      if not self._connected:
>> >  276        self._connected = True
>> >  277        self._driver.start<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints-pysrc.html#
>> > >()
>> >  278        self._wakeup()
>> >  279      self._ewait(lambda: self._transport_connected and not
>> > self._unlinked())
>> >
>> > The parent Python process can connect successfully, but the
>> > self._transport_connected never gets set to True for all of the new
>> > Connection objects created in the subprocesses that are trying to
>> connect
>> > to the same broker. Where does this get set to True?
>> > Could this have something to do with the predicate because it
>> > doesn't return an error it just times out?
>> >
>> >
>> >  212 -<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints-pysrc.html#
>> >
>> >  def _ewait<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints.Connection-class.html#_ewait
>> >(self,
>> > predicate, timeout=None):
>> >
>> >  213      result = self._wait(lambda: self.error or predicate(),
>> timeout<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints-pysrc.html#
>> > >)
>> >  214      self.check_error<
>> >
>> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
>> id.messaging.endpoints-pysrc.html#
>> > >()
>> >  215      return result
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > Thanks,
>> > Taylor
>> > ________________________________
>> > From: Bill Freeman [ke1g.nh@gmail.com]
>> > Sent: Wednesday, August 07, 2013 2:00 PM
>> > To: users
>> > Subject: Re: Python Connections Hang from Subprocesses
>> >
>> > Subprocesses (using the subprocess module, or even the older exec
>> stuff, as
>> > opposed to threads, or even forked clones) are relatively trouble
>> > free
>> in
>> > python (except maybe on Windows, whose process model has that
>> > Microsoft difference).  I've certainly made multiple connections to
>> > a broker from
>> one
>> > python process, as well as using tools like spout and drain, which
>> > are
>> both
>> > written in python, while my main development project is running
>> connected.
>> >
>> > Maybe there are broker configuration items which can affect this.
>> > If
>> so, I
>> > hope that someone knowledgeable will speak up.  But I doubt that
>> > this is the problem.
>> >
>> > Are you sure that your subprocess runs?  It might be trying to
>> > report an error to you.  Of, if you have pipes configured for
>> > interaction with the invoking processor, it might be waiting on one of those.
>> >
>> > You could, temporarily, instead of your intended code, have the
>> subprocess
>> > invoke something like BasicHTTPServer, and see if you can interact
>> > with
>> it
>> > using your browser.  If that also fails, it leaves the broker
>> > connection out as the source of your problems.
>> >
>> > Possibly easier is to have it log its arrival at various points, so
>> > you
>> can
>> > be sure where it is getting stuck.  If you've already confirmed
>> > that
>> it's
>> > in the broker connect, forgive me, and wait for a better answer.
>> >
>> > Bill
>> >
>> >
>> > On Wed, Aug 7, 2013 at 12:55 PM, Eagy, Taylor
>> > <teagy@blackbirdtech.com
>> > >wrote:
>> >
>> > > Hello,
>> > >
>> > >
>> > >
>> > > I'm having an issue connecting multiple Python subprocesses to
>> > > the
>> qpid
>> > > C++ broker. I have a main Python process that imports
>> > > C++ qpid.messaging
>> at
>> > the
>> > > top and spawns a few multiprocessing.Process objects that try to
>> create a
>> > > Qpid connection to the broker. Before I spawn the processes
>> > > though,
>> the
>> > > main process creates a connection to the Qpid broker just fine,
>> > > but
>> all
>> > of
>> > > the subprocess code hangs when it trys to connect to the qpid broker.
>> I
>> > > know qpid is supposed to be multi-threaded, but what about
>> multiprocess
>> > > with Python? What could be causing the connections to hang from
>> > > the subprocesses? Is there some singleton object that isn't
>> > > getting
>> released
>> > > that's blocking the other subprocesses from connecting?
>> > >
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > TJ
>> > >
>> >
>>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


Mime
View raw message