qpid-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Freeman <ke1g...@gmail.com>
Subject Re: Python Connections Hang from Subprocesses
Date Thu, 08 Aug 2013 17:24:04 GMT
I've got no experience with multiprocessing.Process.  My only thought at
the moment is to see if avoiding connecting to the broker in the parent
allows the child to run.  Maybe Process has some shared lock in its socket
code (though that should have prevented BasicHTTPServer too)?


On Thu, Aug 8, 2013 at 9:42 AM, Eagy, Taylor <teagy@blackbirdtech.com>wrote:

> Like I said in the original post this is in a multiprocessing.Process()
> object which uses the subprocess module to create a subprocess and runs the
> run() function in that subprocess.
> You can see the documentation here:
> http://docs.python.org/2/library/multiprocessing.html#multiprocessing.Process
>
> I'm not using threads or the threading module. Thanks for the reminder
> about multiple sessions. I had forgotten about that. This means I will have
> to share the same Connection object between multiple processes. Does the
> Connection object serialize fine when transferring via the pipe?
>
> -----Original Message-----
> From: Bill Freeman [mailto:ke1g.nh@gmail.com]
> Sent: Wednesday, August 07, 2013 5:51 PM
> To: users
> Subject: Re: Python Connections Hang from Subprocesses
>
> By the way, I presume that you know that you can have multiple sessions on
> one connection?
>
>
> On Wed, Aug 7, 2013 at 5:49 PM, Bill Freeman <ke1g.nh@gmail.com> wrote:
>
> >
> >
> >
> > On Wed, Aug 7, 2013 at 5:35 PM, Eagy, Taylor <teagy@blackbirdtech.com
> >wrote:
> >
> >>  What O/S are you running this on?
> >>
> >>
> >>
> >> Ubuntu 12.04 LTS
> >>
> >> What does the code that spawns the subprocess look like?
> >>
> >>
> >>
> >> So my Process objects have a run() method that is called as the
> >> subprocess.
> >>
> >
> > Ok.  Unless I miss my guess, this is not a subprocess.  This is a thread.
> > Your object is probably a subclass or threading.Thread, and you never
> > import subprocess anywhere.  If you ran "ps -alxww" in another
> > terminal, there wouldn't be a line for more than one process.
> >
> >>
> >>
> >>
> >> import qpid.messaging # at the top of the Python module
> >>
> >>
> >>
> >> def run():
> >>
> >> // some other non-qpid setup
> >>
> >>
> >>
> >> conn = qpid.messaging.Connection(qpid_endpoint)
> >>
> >> conn.open() <-- hangs here
> >>
> >> self.sess = conn.session()
> >>
> >> I have done this when there was already a connection to the broker,
> >> but
> > that other connection was a qmf.console connection, which does not use
> > qpid.messaging under the hood.
> >
> > Someone else will have to speak to the thread safety of
> > qpid.messaging, because we have now exceeded my experience in that area.
> >
> >
> >>
> >> Can you, even when the subprocess is hung, run something like spout,
> >> drain, or qpid-config, e.g.;
> >>
> >> qpid-config queues
> >>
> >> from another console window on the same box?
> >>
> >>
> >>
> >> Yes, I can connect to the queue if I try to connect from another
> >> module (which runs a new copy of the Python interpreter).
> >>
> >> ________________________________
> >> From: Bill Freeman [ke1g.nh@gmail.com]
> >> Sent: Wednesday, August 07, 2013 4:21 PM
> >> To: users
> >> Subject: Re: Python Connections Hang from Subprocesses
> >>
> >> Just in case it makes me (or someone else) think of something else:
> >>
> >>   What O/S are you running this on?
> >>
> >>   What does the code that spawns the subprocess look like?
> >>
> >>   Can you, even when the subprocess is hung, run something like
> >> spout, drain, or qpid-config, e.g.;
> >>
> >>       qpid-config queues
> >>
> >>   from another console window on the same box?
> >>
> >> Bill
> >>
> >>
> >> On Wed, Aug 7, 2013 at 3:13 PM, Eagy, Taylor <teagy@blackbirdtech.com
> >> >wrote:
> >>
> >> > Oops. Let me clean it up a bit.
> >> >
> >> > def attach(self):
> >> >
> >> > """
> >> > Attach to the remote endpoint.
> >> > """
> >> > if not self._connected:
> >> >     self._connected = True
> >> >     self._driver.start()
> >> >     self._wakeup()
> >> > self._ewait(lambda: self._transport_connected and not
> >> > self._unlinked())
> >> >
> >> > def _ewait(self, predicate, timeout=None):
> >> >
> >> >
> >> >     result = self._wait(lambda: self.error or predicate(), timeout)
> >> > <--
> >> It
> >> > times out here.
> >> >     self.check_error()
> >> >     return result
> >> >
> >> >
> >> > ________________________________
> >> > From: Eagy, Taylor [teagy@BlackbirdTech.com]
> >> > Sent: Wednesday, August 07, 2013 2:50 PM
> >> > To: users@qpid.apache.org
> >> > Subject: RE: Python Connections Hang from Subprocesses
> >> >
> >> > Bill thanks for your response. The subprocess definitely runs when
> >> > I
> >> don't
> >> > try to connect to the broker. Running a HTTPServer in it works too.
> >> > It
> >> just
> >> > doesn't connect to the qpid broker. I did a little digging and I
> >> > believe this is where it is hanging:
> >> >
> >> > (In Connection class)
> >> >
> >> >  270    @synchronized
> >> >
> >> >  271 -<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints-pysrc.html#
> >> >
> >> >  def attach<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints.Connection-class.html#attach
> >> > >(self):
> >> >
> >> >  272      """
> >> >  273      Attach to the remote endpoint.
> >> >  274      """
> >> >  275      if not self._connected:
> >> >  276        self._connected = True
> >> >  277        self._driver.start<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints-pysrc.html#
> >> > >()
> >> >  278        self._wakeup()
> >> >  279      self._ewait(lambda: self._transport_connected and not
> >> > self._unlinked())
> >> >
> >> > The parent Python process can connect successfully, but the
> >> > self._transport_connected never gets set to True for all of the new
> >> > Connection objects created in the subprocesses that are trying to
> >> connect
> >> > to the same broker. Where does this get set to True?
> >> > Could this have something to do with the predicate because it
> >> > doesn't return an error it just times out?
> >> >
> >> >
> >> >  212 -<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints-pysrc.html#
> >> >
> >> >  def _ewait<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints.Connection-class.html#_ewait
> >> >(self,
> >> > predicate, timeout=None):
> >> >
> >> >  213      result = self._wait(lambda: self.error or predicate(),
> >> timeout<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints-pysrc.html#
> >> > >)
> >> >  214      self.check_error<
> >> >
> >> http://qpid.apache.org/releases/qpid-0.22/messaging-api/python/api/qp
> >> id.messaging.endpoints-pysrc.html#
> >> > >()
> >> >  215      return result
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > Thanks,
> >> > Taylor
> >> > ________________________________
> >> > From: Bill Freeman [ke1g.nh@gmail.com]
> >> > Sent: Wednesday, August 07, 2013 2:00 PM
> >> > To: users
> >> > Subject: Re: Python Connections Hang from Subprocesses
> >> >
> >> > Subprocesses (using the subprocess module, or even the older exec
> >> stuff, as
> >> > opposed to threads, or even forked clones) are relatively trouble
> >> > free
> >> in
> >> > python (except maybe on Windows, whose process model has that
> >> > Microsoft difference).  I've certainly made multiple connections to
> >> > a broker from
> >> one
> >> > python process, as well as using tools like spout and drain, which
> >> > are
> >> both
> >> > written in python, while my main development project is running
> >> connected.
> >> >
> >> > Maybe there are broker configuration items which can affect this.
> >> > If
> >> so, I
> >> > hope that someone knowledgeable will speak up.  But I doubt that
> >> > this is the problem.
> >> >
> >> > Are you sure that your subprocess runs?  It might be trying to
> >> > report an error to you.  Of, if you have pipes configured for
> >> > interaction with the invoking processor, it might be waiting on one
> of those.
> >> >
> >> > You could, temporarily, instead of your intended code, have the
> >> subprocess
> >> > invoke something like BasicHTTPServer, and see if you can interact
> >> > with
> >> it
> >> > using your browser.  If that also fails, it leaves the broker
> >> > connection out as the source of your problems.
> >> >
> >> > Possibly easier is to have it log its arrival at various points, so
> >> > you
> >> can
> >> > be sure where it is getting stuck.  If you've already confirmed
> >> > that
> >> it's
> >> > in the broker connect, forgive me, and wait for a better answer.
> >> >
> >> > Bill
> >> >
> >> >
> >> > On Wed, Aug 7, 2013 at 12:55 PM, Eagy, Taylor
> >> > <teagy@blackbirdtech.com
> >> > >wrote:
> >> >
> >> > > Hello,
> >> > >
> >> > >
> >> > >
> >> > > I'm having an issue connecting multiple Python subprocesses to
> >> > > the
> >> qpid
> >> > > C++ broker. I have a main Python process that imports
> >> > > C++ qpid.messaging
> >> at
> >> > the
> >> > > top and spawns a few multiprocessing.Process objects that try to
> >> create a
> >> > > Qpid connection to the broker. Before I spawn the processes
> >> > > though,
> >> the
> >> > > main process creates a connection to the Qpid broker just fine,
> >> > > but
> >> all
> >> > of
> >> > > the subprocess code hangs when it trys to connect to the qpid
> broker.
> >> I
> >> > > know qpid is supposed to be multi-threaded, but what about
> >> multiprocess
> >> > > with Python? What could be causing the connections to hang from
> >> > > the subprocesses? Is there some singleton object that isn't
> >> > > getting
> >> released
> >> > > that's blocking the other subprocesses from connecting?
> >> > >
> >> > >
> >> > >
> >> > > Thanks,
> >> > >
> >> > > TJ
> >> > >
> >> >
> >>
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message