lucene-pylucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andi Vajda <va...@apache.org>
Subject Re: AW: AW: [VOTE] Release PyLucene 4.2.1-1
Date Fri, 19 Apr 2013 18:24:56 GMT

Thank you, Thomas, for debugging this.
I'm going to add PREFIX to the built-in JCC reserved word list.

More comments inline...

On Fri, 19 Apr 2013, Thomas Koch wrote:

> The PREFIX seems to be the only issue - using:
> JCC=$(PYTHON) -m jcc --reserved PREFIX --shared --find-jvm-dll
> 'make' and 'make install' did pass.
> 'make test' yields the usual windows specific issues (due to some file
> locking issues) - there are a total of 45 errors in "make test", as said
> most due to windows not being able to remove a locked resource, e.g.
>
>  File "C:\Python27\lib\shutil.py", line 247, in rmtree
>    os.remove(fullname)
> WindowsError: [Error 32] Der Prozess kann nicht auf die Datei zugreifen, da
> sie von einem anderen Prozess verwendet wird: 'testrepo\\_0.fdt'
>
> Most errors seem to be follow-up issues (test-repo locked) - this also
> occurred in PyLucene 3.6.
>
> If anyone is interested in details - the output of 'make test' is available
> here:
> https://dl.dropboxusercontent.com/u/4384120/pylucene-4.2.1-test-20130419.txt
>
> I debugged this (once again) for test_PyLucene and found that
> - there is no write.lock pending on the store (directory 'testrepo')
> - the store is closed properly
> - there is a windows lock on a  Field Data file that therefore cannot be
> deleted: 'testrepo\_0.fdt'
> - even timeout-retry on shutil.rmtree does not fix it
> - even passing an onerror-handler to shutil.rmtree() that tries to force
> remove this file (using os.chmod before) does not help
>
> I'm clueless here. It could be that the JVM actually holds the lock and a
> JVM shutdown would help here. I therefore tried
>
>    def setUp(self):
>        self.jvm = lucene.initVM()
> 		...
> and in
>
>    def tearDown(self):
>        ...
>        self.jvm.detachCurrentThread()

This doesn't close the VM, it just detaches the current thread (and is a 
rather risky operation). If I remember correctly, the closing and reopening 
of a VM in a given is not supported by the JNI interface (or the interface 
exists but it's not implemented). (I could be wrong, of course).

> But this did crash my python.exe ,-( Is there any other way to safely 
> re-init jcc during one process loop? Here's the test_PyLucene.py with all 
> my attempts: 
> https://dl.dropboxusercontent.com/u/4384120/test_PyLucene-win-20130419.py
>
> I then ended up in using a 'fresh' test-repository (i.e. new dir name) if
> rmtree fails. The test_PyLucene test then passes on windows - the downside
> is that you end up with some testrepo.1, testrepo.2 etc. dirs - and of
> course an attempt to cleanup these dirs after unittest.main(exit=False) also
> fails...
>
> There's a patch against the pylucene_4_2 for test_PyLucene with this "fix":
> https://dl.dropboxusercontent.com/u/4384120/pylucene_4_2_test_PyLucene_WinFi
> x_20130419.patch
>
> Note: this should not affect linux tests and fixes the tests for windows.
> Though the test_PyLucene now passes (with the fix), still 20 tests fail ,
> e.g. test_PythonDirectory  and test_FieldEnumeration. Possibly similar
> workarounds could be applied here.  As the main problem seems to be mixing
> up several lucene-tests in one process call, alternatively one could write a
> batch script, that calls all those tests one by one (in a new windows
> process) - or even split test_-files into several smaller tests (python
> files). Don't know if these are better ideas though.
>
> So in summary: if you add the PREFIX to reserved words it will build and run
> on windows.

Thanks again. A trick - on Windows - might be to rename the testrepo dir to 
testrepo.<current datetime> and move to the next test. And clean them all up 
at the end. Or create a testrepos directory tree where each 
testrepo.<currentTime> is going to be created and remove the whole tree at 
the end.

GIven that Lucene 4.3 is about to be released, I think I'm going to drop the 
4.2.1 PyLucene release and move to proposing a PyLucene 4.3 release 
candidate next.

Andi..

>
> Regards,
> Thomas
>
>
>
> -----Urspr√ľngliche Nachricht-----
> Von: Andi Vajda [mailto:vajda@apache.org]
> Gesendet: Donnerstag, 18. April 2013 19:09
> An: pylucene-dev@lucene.apache.org
> Betreff: Re: AW: [VOTE] Release PyLucene 4.2.1-1
>
>
> On Thu, 18 Apr 2013, Thomas Koch wrote:
>
>> Andi,
>> I now get a different error while compiling __init__.cpp:
>>
>> org/apache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42
>> )
> :
>> error C2059: Syntaxfehler: 'Zeichenfolge'
>> org/apache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42
>> )
> :
>> error C2238: Unerwartete(s) Token vor ';'
>>
>> The line complained about is #42
>>
>> 40            static CompiledAutomaton$AUTOMATON_TYPE *NONE;
>> 41            static CompiledAutomaton$AUTOMATON_TYPE *NORMAL;
>> 42            static CompiledAutomaton$AUTOMATON_TYPE *PREFIX;
>> 43            static CompiledAutomaton$AUTOMATON_TYPE *SINGLE;
>>
>> PREFIX seems to be another reserved word ... I could compile
> __init__.cpp
>> after renaming PREFIX to PREFIX1.
>
> Instead of renaming PREFIX, could you please have JCC do it for you by
> adding it to the list of reserved words in the JCC invocation via the
> --reserved command line flag ? and rinse and repeat until all such conficts
> due to macro definitions are solved ?
>
> Or were you able to complete the build already once PREFIX was renamed ?
>
>> I tried to google a list of reserved words used by VS C++ compiler,
>> but
> had
>> no luck...
>
> These are not reserved words but macro definitions that conflict with the
> generated code. If PREFIX is, say, defined to 1, line 42 becomes:
>
>   static CompiledAutomaton$AUTOMATON_TYPE *1;
>
> and that doesn't compile.
>
>> There are some predefined macros -but none that match our issues
>> http://msdn.microsoft.com/en-us/library/b0084kay(v=vs.100).aspx
>
> Andi..
>
>>
>>
>>
>> Make output details:
>>
>> C:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\cl.exe /c /nologo
> /Ox
>> /MD /W3 /GS- /DNDEBUG -DPYTHON -DJCC_VER="2.16" -D_jcc_shared
>> -D_java_generics -D_dll_lucene=__declspec(dllexport) "-IC:\Program
>> Files\Java\jdk1.6.0_06/include" "-IC:\Program
>> Files\Java\jdk1.6.0_06/include/win32" -Ibuild\_lucene
>> -IC:\Python27\lib\site-packages\jcc-2.16-py2.7-win32.egg\jcc\sources
>> -IC:\Python27\include -IC:\Python27\PC /Tpbuild\_lucene\__init__.cpp
>> /Fobuild\temp.win32-2.7\Release\build\_lucene\__init__.obj /EHsc
>> /D_CRT_SECURE_NO_WARNINGS __init__.cpp
>>
> C:\Python27\lib\site-packages\jcc-2.16-py2.7-win32.egg\jcc\sources\JCCEnv.
> h(
>> 118) : warning C4251: 'JCCEnv::refs': class 'std::multimap<_Kty,_Ty>'
>> erfordert eine DLL-Schnittstelle, die von Clients von class 'JCCEnv'
>> verwendet wird
>>        with
>>        [
>>            _Kty=int,
>>            _Ty=countedRef
>>        ]
>>
> f:\devel\workspaces\workspace.pylucene\pylucene-4.2.1-1\build\_lucene\org/
> ap
>> ache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42) :
> error
>> C2059: Syntaxfehler: 'Zeichenfolge'
>>
> f:\devel\workspaces\workspace.pylucene\pylucene-4.2.1-1\build\_lucene\org/
> ap
>> ache/lucene/util/automaton/CompiledAutomaton$AUTOMATON_TYPE.h(42) :
> error
>> C2238: Unerwartete(s) Token vor ';'
>> error: command '"C:\Program Files\Microsoft Visual Studio
>> 9.0\VC\BIN\cl.exe"' failed with exit status 2
>> make: *** [compile] Error 1
>>
>>
>> regards,
>> Thomas
>>
>> -----Urspr√ľngliche Nachricht-----
>> Von: Andi Vajda [mailto:vajda@apache.org]
>> Gesendet: Mittwoch, 17. April 2013 22:11
>> An: pylucene-dev@lucene.apache.org
>> Cc: general@lucene.apache.org
>> Betreff: [VOTE] Release PyLucene 4.2.1-1
>>
>>
>> The PyLucene 4.2.1-0 release candidate had a number of problems
> preventing
>> its release. A PyLucene 4.2.1-1 release candidate is now ready for
> review
>> from:
>>
>>   http://people.apache.org/~vajda/staging_area/
>>
>> A list of changes in this release can be seen at:
>>
> http://svn.apache.org/repos/asf/lucene/pylucene/branches/pylucene_4_2/CHAN
> GE
>> S
>>
>> PyLucene 4.2.1 is built with JCC 2.16 included in these release
> artifacts:
>> http://svn.apache.org/repos/asf/lucene/pylucene/trunk/jcc/CHANGES
>>
>> A list of Lucene Java changes can be seen at:
>>
> http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_4_2_1/lucene/C
> HA
>> NGES.txt
>>
>> Please vote to release these artifacts as PyLucene 4.2.1-1.
>>
>> Thanks !
>>
>> Andi..
>>
>> ps: the KEYS file for PyLucene release signing is at:
>> http://svn.apache.org/repos/asf/lucene/pylucene/dist/KEYS
>> http://people.apache.org/~vajda/staging_area/KEYS
>>
>> pps: here is my +1
>>
>>
>
>
Mime
  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message