lucene-pylucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andi Vajda <>
Subject Re: call python from java - what strategy do you use?
Date Tue, 11 Jan 2011 19:13:06 GMT

  Hi Roman,

On Tue, 11 Jan 2011, Roman Chyla wrote:

> I have recently wrapped solr inside jetty with JCC (we need to access
> very big result sets quickly, via JNI, but also keep solr running as
> normal) and was wondering what strategies do you guys use to speak
> *from inside* Java towards the Python end.
> So far, I was able to think about these:
> - raise exceptions in java and catch in python (I think I have seen
> this in some posts from Bill Jansen)
> - communicate via sockets
> - wait passively - call some java method and wait for its return
> - monitor actively - in python check in loop some java object
> Is there something else?

I'm not sure I completely understand your questions but if what you're 
asking is how to run Python code from inside a Java servlet container, that
I've done with Tomcat and Lucene.

Basically, instead of embedding a JVM inside a Python VM - as is done for 
PyLucene - you do the opposite, you embed a Python VM inside a JVM.

For that purpose, see the org.apache.jcc.PythonVM class available in JCC's 
java tree. This class must be instantiated from the main thread at Java
servlet engine startup time. In Tomcat, I patched some startup code, in (see patches below) for this purpose.

Then, to make some Python code accessible from Java, use the usual way of
writing "extensions", the so-called JCC in reverse trick. Define a Java class
with some native methods implemented in Python; define a Python class that
"extends" it; build the Java class into a JAR; include it into a JCC-built
egg; install the egg into Python's env (site-packages, PYTHONPATH, whatever);
Then, write servlet code in Java that imports your Java class and calls it.

As you can see, this sounds simple but the devil is in the details. Of course,
bending Jetty for this may have different requirements but the code snippets
below should give you a good idea about what's required.

This approach has been in production running the's search server
for over two years now.

If you have questions, of course, please ask.
Good luck !


Patch to to use JCC's PythonVM (which initializes the embedded
Python VM)

--- apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/	2010-07-19 06:02:32.000000000
+++ apache-tomcat-6.0.29-src/java/org/apache/catalina/startup/	2010-08-04
08:49:05.000000000 -0700
@@ -30,16 +30,18 @@

  import org.apache.juli.logging.Log;
  import org.apache.juli.logging.LogFactory;

+import org.apache.jcc.PythonVM;

   * Boostrap loader for Catalina.  This application constructs a class loader
   * for use in loading the Catalina internal classes (by accumulating all of the
   * JAR files found in the "server" directory under "catalina.home"), and
   * starts the regular execution of the container.  The purpose of this
   * roundabout approach is to keep the Catalina internal classes (and any
   * other classes they depend on, such as an XML parser) out of the system
@@ -398,22 +400,24 @@
          try {
              String command = "start";
              if (args.length > 0) {
                  command = args[args.length - 1];

              if (command.equals("startd")) {
                  args[args.length - 1] = "start";
+                PythonVM.start("mql");
              } else if (command.equals("stopd")) {
                  args[args.length - 1] = "stop";
              } else if (command.equals("start")) {
+                PythonVM.start("mql");
              } else if (command.equals("stop")) {
              } else {
                  log.warn("Bootstrap: command \"" + command + "\" does not exist.");

Define a Java class:

package ....

public class EMQL {

     private long pythonObject;

     public EMQL()

     public void pythonExtension(long pythonObject)
         this.pythonObject = pythonObject;
     public long pythonExtension()
         return this.pythonObject;

     public void finalize()
         throws Throwable

     public native void pythonDecRef();

     // the methods implemented in python
     public native String init(ME me);
     public native String emql_refresh(String tid, String type);
     public native String emql_status();

     etc .......... etc

The corresponding Python class

import ......

from jemql import initVM, CLASSPATH, EMQL


class emql(EMQL):

     def __init__(self):
         super(emql, self).__init__()

     def init(self, me):
     def emql_refresh(self, tid, type):
     def emql_status(self):
        return "some status"

     etc ...... etc

Makefile rules to build this via JCC (the jemql.egg file is just an empty
target file for Makefile, it's not used for anything else):

default: jemql.egg

jemql.jar: java/org/blah/blah/
 	mkdir -p classes
 	javac -classpath $(CLASSPATH):$(MORE_CLASSPATH):$(etc..etc) -d classes $(JAVAC_FLAGS) $<
 	jar -cvf $@ -C classes .

jemql.egg: jemql.jar $(JMQL_JAR)
 	$(JCC) --version 1.0 --jar $< \
                --classpath $(CLASSPATH):$(JME_JAR):$(JMQL_JAR) \
                --package java.lang \
                --python jemql --build $(DBG_FLAGS) \
                --install \
                --module emql
 	touch $@
Patch to Tomcat's build.xml ANT script to add JCC's classes (like PythonVM) to
the build classpath.

--- apache-tomcat-6.0.29-src/build.xml	2010-07-19 06:02:31.000000000 -0700
+++ apache-tomcat-6.0.29-src/build.xml.patched	2010-08-04 09:30:24.000000000 -0700
@@ -95,16 +95,17 @@
    <property name="jasper-jdt.jar" value="${jasper-jdt.home}/jasper-jdt.jar"/>
    <available property="tomcat-dbcp.present" file="${tomcat-dbcp.jar}" />
    <available property="jdk16.present" classname="javax.sql.StatementEvent" />

    <!-- Classpath -->
    <path id="tomcat.classpath">
      <pathelement location="${ant.jar}"/>
      <pathelement location="${jdt.jar}"/>
+    <pathelement location="${jcc.egg}/jcc/classes"/>

    <!-- Version info filter set -->
      <format property="TODAY" pattern="MMM d yyyy" locale="en"/>
      <format property="TSTAMP" pattern="hh:mm:ss"/>
    <filterset id="version.filters">
@@ -148,16 +149,25 @@
  <!-- Comment this in to show unchecked warnings:
        <compilerarg value="-Xlint:unchecked"/>
        <classpath refid="tomcat.classpath" />
        <exclude name="org/apache/naming/factory/webservices/**" />
+    <javac srcdir="${extras.path}" destdir="${tomcat.classes}"
+           debug="${compile.debug}"
+           deprecation="${compile.deprecation}"
+           source="${compile.source}"
+           optimize="${compile.optimize}"
+           excludes="**/CVS/**,**/.svn/**">
+<!-- Comment this in to show unchecked warnings:     <compilerarg value="-Xlint:unchecked"/>
+      <classpath refid="tomcat.classpath" />
+    </javac>
      <!-- Copy static resource files -->
      <copy todir="${tomcat.classes}" encoding="ISO-8859-1">
        <filterset refid="version.filters"/>
        <fileset dir="java">
          <include name="**/*.properties"/>
          <include name="**/*.dtd"/>
          <include name="**/*.tasks"/>
          <include name="**/*.xsd"/>

Patch to, the Tomcat startup script to add JCC to LIBPATH and

--- apache-tomcat-6.0.29-src/output/build/bin/	2010-08-04 09:57:27.000000000 -0700
+++ apache-tomcat-6.0.29-src/output/build/bin/	2010-08-04 09:57:47.000000000
@@ -162,16 +162,30 @@
      exit 1

  if [ -z "$CATALINA_BASE" ] ; then

+if [ -n "$JCC_EGG" ]; then
+  CLASSPATH="$CLASSPATH":"$JCC_EGG"/jcc/classes
+if [ -n "$TOMCAT_APR_LIB_PATH" ]; then
+if [ -n "$JAVA_LIB_PATH" ]; then
+  JAVA_OPTS="$JAVA_OPTS -Djava.library.path=$JAVA_LIB_PATH"
+if [ -n "EXTRA_CLASSPATH" ]; then
  # Add tomcat-juli.jar and bootstrap.jar to classpath
  # tomcat-juli.jar can be over-ridden per instance
  if [ ! -z "$CLASSPATH" ] ; then
  if [ "$CATALINA_BASE" != "$CATALINA_HOME" ] && [ -r "$CATALINA_BASE/bin/tomcat-juli.jar"
] ; then

These EGG paths are long, complicated and OS-specific, the trick below
generates them programmatically (from inside a Makefile):

JCC_EGG:=$(shell $(PYTHON) -c "import os, jcc; print os.path.dirname(os.path.dirname(jcc.__file__))")
JEMQL_EGG:=$(shell $(PYTHON) -c "import os, jemql; print os.path.dirname(os.path.dirname(jemql.__file__))")

Then, the CLASSPATH addition during _build_ time:
   CLASSPATH = $(CLASSPATH):$(JEMQL_EGG)/jemql/jemql.jar
and so on...
At runtime, JCC takes care of adding your eggs to the startup CLASSPATH.

Last but not least, if you use Python's thread local storage in your threads, 
Python threads when embedded inside a JVM are 'dummy', that is, while they're
backed by the actual Java thread (a pthread), the Python VM is not managing
them and a thread state object is created each and every time a Python thread
is entered and released when exited back to the JVM. This has two problems:
  1. it's a bit wasteful
  2. python thread local storage gets lost

The Java class below works this around by incrementing the refcount that
controls this:

package org.apache.catalina.core;

import org.apache.jcc.PythonVM;

public class TerminatingThread extends Thread {
     protected Runnable runnable;

     public TerminatingThread(ThreadGroup group, Runnable runnable, String name)
         super(group, name);
         this.runnable = runnable;

     public void run()
         PythonVM vm = PythonVM.get();

         try {
         } finally {

Then, there is some trickery to get Tomcat to use this class for its threads
instead of the default one:

--- apache-tomcat-6.0.29-src/java/org/apache/catalina/core/	2010-07-19
06:02:32.000000000 -0700
+++ apache-tomcat-6.0.29-src/java/org/apache/catalina/core/
2010-08-04 08:56:02.000000000 -0700
@@ -44,17 +44,17 @@
      protected int minSpareThreads = 25;

      protected int maxIdleTime = 60000;

      protected ThreadPoolExecutor executor = null;

      protected String name;

-    private LifecycleSupport lifecycle = new LifecycleSupport(this);
+    protected LifecycleSupport lifecycle = new LifecycleSupport(this);
      // ---------------------------------------------- Constructors
      public StandardThreadExecutor() {
          //empty constructor for the digester

      // ---------------------------------------------- Public Methods

In Tomcat's server.xml, use this executor (and code below for it)
     <Executor name="relThreadPool"
 	      maxThreads="2" />

package org.apache.catalina.core;

import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
import org.apache.catalina.LifecycleException;

public class TerminatingThreadExecutor extends StandardThreadExecutor {

     public void start()
         throws LifecycleException
         lifecycle.fireLifecycleEvent(BEFORE_START_EVENT, null);

         TaskQueue taskqueue = new TaskQueue();
         TaskThreadFactory tf = new TerminatingTaskThreadFactory(namePrefix);

         lifecycle.fireLifecycleEvent(START_EVENT, null);
         executor = new ThreadPoolExecutor(getMinSpareThreads(), getMaxThreads(),
                                           maxIdleTime, TimeUnit.MILLISECONDS,
                                           taskqueue, tf);
         lifecycle.fireLifecycleEvent(AFTER_START_EVENT, null);

     protected class TerminatingTaskThreadFactory
         extends StandardThreadExecutor.TaskThreadFactory {

         protected TerminatingTaskThreadFactory(String namePrefix)

         public Thread newThread(Runnable runnable)
             Thread t = new TerminatingThread(group, runnable, namePrefix + threadNumber.getAndIncrement());


             return t;

View raw message