qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Freeman <ke1g...@gmail.com>
Subject Re: Python Connections Hang from Subprocesses
Date Wed, 07 Aug 2013 21:49:16 GMT
On Wed, Aug 7, 2013 at 5:35 PM, Eagy, Taylor <teagy@blackbirdtech.com>wrote:

>  What O/S are you running this on?
>
>
>
> Ubuntu 12.04 LTS
>
> What does the code that spawns the subprocess look like?
>
>
>
> So my Process objects have a run() method that is called as the subprocess.
>

Ok.  Unless I miss my guess, this is not a subprocess.  This is a thread.
Your object is probably a subclass or threading.Thread, and you never
import subprocess anywhere.  If you ran "ps -alxww" in another terminal,
there wouldn't be a line for more than one process.

>
>
>
> import qpid.messaging # at the top of the Python module
>
>
>
> def run():
>
> // some other non-qpid setup
>
>
>
> conn = qpid.messaging.Connection(qpid_endpoint)
>
> conn.open() <-- hangs here
>
> self.sess = conn.session()
>
> I have done this when there was already a connection to the broker, but
that other connection was a qmf.console connection, which does not use
qpid.messaging under the hood.

Someone else will have to speak to the thread safety of qpid.messaging,
because we have now exceeded my experience in that area.


>
> Can you, even when the subprocess is hung, run something like spout,
> drain, or qpid-config, e.g.;
>
> qpid-config queues
>
> from another console window on the same box?
>
>
>
> Yes, I can connect to the queue if I try to connect from another module
> (which runs a new copy of the Python interpreter).
>
> ________________________________
> From: Bill Freeman [ke1g.nh@gmail.com]
> Sent: Wednesday, August 07, 2013 4:21 PM
> To: users
> Subject: Re: Python Connections Hang from Subprocesses
>
> Just in case it makes me (or someone else) think of something else:
>
>   What O/S are you running this on?
>
>   What does the code that spawns the subprocess look like?
>
>   Can you, even when the subprocess is hung, run something like spout,
> drain, or qpid-config, e.g.;
>
>       qpid-config queues
>
>   from another console window on the same box?
>
> Bill
>
>
> On Wed, Aug 7, 2013 at 3:13 PM, Eagy, Taylor <teagy@blackbirdtech.com
> >wrote:
>
> > Oops. Let me clean it up a bit.
> >
> > def attach(self):
> >
> > """
> > Attach to the remote endpoint.
> > """
> > if not self._connected:
> >     self._connected = True
> >     self._driver.start()
> >     self._wakeup()
> > self._ewait(lambda: self._transport_connected and not self._unlinked())
> >
> > def _ewait(self, predicate, timeout=None):
> >
> >
> >     result = self._wait(lambda: self.error or predicate(), timeout) <--
> It
> > times out here.
> >     self.check_error()
> >     return result
> >
> >
> > ________________________________
> > From: Eagy, Taylor [teagy@BlackbirdTech.com]
> > Sent: Wednesday, August 07, 2013 2:50 PM
> > To: users@qpid.apache.org
> > Subject: RE: Python Connections Hang from Subprocesses
> >
> > Bill thanks for your response. The subprocess definitely runs when I
> don't
> > try to connect to the broker. Running a HTTPServer in it works too. It
> just
> > doesn't connect to the qpid broker. I did a little digging and I believe
> > this is where it is hanging:
> >
> > (In Connection class)
> >
> >  270    @synchronized
> >
> >  271 -<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints-pysrc.html#
> >
> >  def attach<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints.Connection-class.html#attach
> > >(self):
> >
> >  272      """
> >  273      Attach to the remote endpoint.
> >  274      """
> >  275      if not self._connected:
> >  276        self._connected = True
> >  277        self._driver.start<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints-pysrc.html#
> > >()
> >  278        self._wakeup()
> >  279      self._ewait(lambda: self._transport_connected and not
> > self._unlinked())
> >
> > The parent Python process can connect successfully, but the
> > self._transport_connected never gets set to True for all of the new
> > Connection objects created in the subprocesses that are trying to connect
> > to the same broker. Where does this get set to True?
> > Could this have something to do with the predicate because it doesn't
> > return an error it just times out?
> >
> >
> >  212 -<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints-pysrc.html#
> >
> >  def _ewait<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints.Connection-class.html#_ewait
> >(self,
> > predicate, timeout=None):
> >
> >  213      result = self._wait(lambda: self.error or predicate(), timeout<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints-pysrc.html#
> > >)
> >  214      self.check_error<
> >
> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qpid.messaging.endpoints-pysrc.html#
> > >()
> >  215      return result
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Thanks,
> > Taylor
> > ________________________________
> > From: Bill Freeman [ke1g.nh@gmail.com]
> > Sent: Wednesday, August 07, 2013 2:00 PM
> > To: users
> > Subject: Re: Python Connections Hang from Subprocesses
> >
> > Subprocesses (using the subprocess module, or even the older exec stuff,
> as
> > opposed to threads, or even forked clones) are relatively trouble free in
> > python (except maybe on Windows, whose process model has that Microsoft
> > difference).  I've certainly made multiple connections to a broker from
> one
> > python process, as well as using tools like spout and drain, which are
> both
> > written in python, while my main development project is running
> connected.
> >
> > Maybe there are broker configuration items which can affect this.  If
> so, I
> > hope that someone knowledgeable will speak up.  But I doubt that this is
> > the problem.
> >
> > Are you sure that your subprocess runs?  It might be trying to report an
> > error to you.  Of, if you have pipes configured for interaction with the
> > invoking processor, it might be waiting on one of those.
> >
> > You could, temporarily, instead of your intended code, have the
> subprocess
> > invoke something like BasicHTTPServer, and see if you can interact with
> it
> > using your browser.  If that also fails, it leaves the broker connection
> > out as the source of your problems.
> >
> > Possibly easier is to have it log its arrival at various points, so you
> can
> > be sure where it is getting stuck.  If you've already confirmed that it's
> > in the broker connect, forgive me, and wait for a better answer.
> >
> > Bill
> >
> >
> > On Wed, Aug 7, 2013 at 12:55 PM, Eagy, Taylor <teagy@blackbirdtech.com
> > >wrote:
> >
> > > Hello,
> > >
> > >
> > >
> > > I'm having an issue connecting multiple Python subprocesses to the qpid
> > > C++ broker. I have a main Python process that imports qpid.messaging at
> > the
> > > top and spawns a few multiprocessing.Process objects that try to
> create a
> > > Qpid connection to the broker. Before I spawn the processes though, the
> > > main process creates a connection to the Qpid broker just fine, but all
> > of
> > > the subprocess code hangs when it trys to connect to the qpid broker. I
> > > know qpid is supposed to be multi-threaded, but what about multiprocess
> > > with Python? What could be causing the connections to hang from the
> > > subprocesses? Is there some singleton object that isn't getting
> released
> > > that's blocking the other subprocesses from connecting?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > TJ
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message