Mailing-List: contact pylucene-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: pylucene-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <4BF2C2C0.30601@cheimes.de>
Date: Tue, 18 May 2010 18:39:28 +0200
From: Christian Heimes <lists@cheimes.de>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.1.11pre) Gecko/20100513 Shredder/3.0.6pre
MIME-Version: 1.0
To: pylucene-dev@lucene.apache.org
Subject: Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse
 api
References: <4BEAD7AF.5020504@semantics.de>
 <alpine.OSX.2.01.1005122248150.36736@yuzu.local>
 <alpine.OSX.2.01.1005131118400.37716@dhcp-wired-157.corp.631h.metaweb.com>
 <4BEC8E27.8010406@cheimes.de>
 <alpine.OSX.2.01.1005131919470.38748@yuzu.local>
 <alpine.OSX.2.01.1005161919480.7167@yuzu.local>
 <alpine.OSX.2.01.1005162003230.7350@yuzu.local>
In-Reply-To: <alpine.OSX.2.01.1005162003230.7350@yuzu.local>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Andi Vajda wrote:
>> On Linux, there's got to be a way to make this work but I haven't found how
>> yet. I seem to be having trouble with linking to the Python Type objects. On
>> Windows, I expect this to be even more difficult.
>
> Adding a call to dlopen("_lucene.so", RTLD_NOW | RTLD_GLOBAL) solves the
> problem on Linux. I'm thinking of adding this to the initVM() call with an
> optional 'export=true/false' parameter... unless there is a better way to go
> about this.

Dear Andi,

thanks for your hard work! So far we have have found multiple smaller 
and larger issues with your experimental feature.

Setup
=====
I'm running the tests on an Ubuntu 10.04 X86_64 box with Python 2.6.5, 
Sun Java 1.6.0_20, GCC 4.4.3 and patched setuptools 0.6c11 (all 64bit).

JCC:
$ svn co 
http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x/jcc 
jcc_3x
$ cd jcc_3x
$ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py build
$ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py install

Small issue: The install target always rewrites the jcc.config module. 
This is a problem when I install JCC with "sudo python2.6 setup.py 
install" since it's not trivial to keep the environment variable JCC_JDK 
across a sudo call. How do you like the idea to bind the creation of 
jcc.config to the build or build_ext target?

LUCENE:
$ cd ../pylucene-2.9.2-1
$ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
NUM_FILES=2
$ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
NUM_FILES=2 install

Suggestion: How about providing sensible defaults for ANT, PYTHON, 
NUM_FILES and JCC? It's annoying that I have to type the arguments over 
and over again. ;)

BOBO-BROWSE:
$ cd ../bobobrowse
$ make
python2.6 -m jcc.__main__ --shared --jar 
bobo-browse-2.5.0-rc1/dist/bobo-browse-2.5.0-rc1.jar --include 
bobo-browse-2.5.0-rc1/lib/master/servlet-api.jar --include 
bobo-browse-2.5.0-rc1/lib/master/spring-webmvc.jar --include 
bobo-browse-2.5.0-rc1/lib/master/spring.jar --include 
bobo-browse-2.5.0-rc1/lib/master/log4j.jar --include 
bobo-browse-2.5.0-rc1/lib/master/protobuf-java-2.2.0.jar --include 
bobo-browse-2.5.0-rc1/lib/master/xstream.jar --include 
bobo-browse-2.5.0-rc1/lib/master/fastutil.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/ant.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/xercesImpl.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-collections.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-cli.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-configuration.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-logging.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-lang.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-digester.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-httpclient.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/zoie-2.0.0-rc1.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/json.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/dwr.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/xmlParserAPIs.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/kamikaze-2.0.0.jar --import lucene 
--package java.lang java.lang.System java.lang.Runtime --package 
java.util java.util.Arrays java.text.SimpleDateFormat --package java.io 
java.io.StringReader java.io.InputStreamReader java.io.FileInputStream 
--python bobobrowse --version 2.5.0-rc1 --files 2 --build
...
$ make install

As you can see I'm using "--import lucene" here.

$ python2.6
>>> import lucene
>>> lucene.initVM(export=True)
<jcc.JCCEnv object at 0x7f405de450d8>
>>> import bobobrowse
>>> bobobrowse.initVM()
<jcc.JCCEnv object at 0x7f405de451c8>


Issues
======

- I have to follow a strict order (first import lucene, than 
lucene.initVM(export=True), finally import bobobrowse). When I try to 
import bobobrowse before the call to initVM(export=True), the import 
fails with unresolved symbols. Have you tried to add _lucene.so as a 
dependency to _bobobrowse.so? I believe a dependency should make the 
dlopen() call obsolete.

$ ldd 
/usr/local/lib/python2.6/dist-packages/bobobrowse-2.5.0_rc1-py2.6-linux-x86_64.egg/bobobrowse/_bobobrowse.so
         linux-vdso.so.1 =>  (0x00007fffc37ff000)
         libjcc.so => 
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/libjcc.so 
(0x00007fbbdc9f8000)
         libjava.so => /usr/lib/jvm/java-6-sun/jre/lib/amd64/libjava.so 
(0x00007fbbdc8c8000)
         libjvm.so => 
/usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so (0x00007fbbdbe50000)
         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fbbdbb35000)
         libm.so.6 => /lib/libm.so.6 (0x00007fbbdb8b1000)
         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fbbdb69a000)
         libpthread.so.0 => /lib/libpthread.so.0 (0x00007fbbdb47d000)
         libc.so.6 => /lib/libc.so.6 (0x00007fbbdb0fa000)
         libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 
(0x00007fbbdac49000)
         libverify.so => 
/usr/lib/jvm/java-6-sun/jre/lib/amd64/libverify.so (0x00007fbbdab3a000)
         libnsl.so.1 => /lib/libnsl.so.1 (0x00007fbbda91f000)
         libdl.so.2 => /lib/libdl.so.2 (0x00007fbbda71b000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fbbdd1af000)
         libssl.so.0.9.8 => /lib/libssl.so.0.9.8 (0x00007fbbda4cb000)
         libcrypto.so.0.9.8 => /lib/libcrypto.so.0.9.8 (0x00007fbbda13a000)
         libz.so.1 => /lib/libz.so.1 (0x00007fbbd9f23000)
         libutil.so.1 => /lib/libutil.so.1 (0x00007fbbd9d20000)


- With "--import lucene" JCC gets the class_ attribute of all types 
wrong, that are part of the bobobrowse package. It either shows 
java.lang.Object as class_ or the first lucene class in the hierarchy. 
Without "--import lucene" it works as expected.

>>> bobobrowse.FacetSpec.class_
<Class: class java.lang.Object>

BoboBrowser is a subclass of MultiBoboBrowser which extends lucene's 
MultiSearcher
>>> bobobrowse.MultiBoboBrowser.class_
<Class: class org.apache.lucene.search.MultiSearcher>

>>> bobobrowse.BoboBrowser.class_
<Class: class org.apache.lucene.search.MultiSearcher>

>>> bobobrowse.Tokenizer.class_
<Class: class org.apache.lucene.analysis.Tokenizer>


- With "--import lucene" JCC doesn't wrap certain classes and it's 
missing bobobrowse.FacetSpec.FacetSortSpec.OrderHitsDesc. Without 
"--import lucene" the enum is wrapped correctly.

>>> bobobrowse.FacetSpec.FacetSortSpec.OrderHitsDesc
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: type object 'FacetSpec$FacetSortSpec' has no attribute 
'OrderHitsDesc'


- I've run into a rather strange bug with our custom builds of Python 
2.6. It differs from the stock build of Ubuntu in two ways. The 
python2.6 binary is linked against libpython2.6.so (Ubuntu's python2.6 
binary has libpython2.6.a statically linked in) and I've set the 
environment variable LD_RUN_PATH which causes the linker to add a 
"library rpath" to all ELF binaries. I wasn't able to find the reason 
for the issue so far. Any hints are appreciated.

Christian