incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: [Lucy] Threads, revisited
Date Wed, 06 Jan 2010 06:24:14 GMT
On Tue, Jan 05, 2010 at 09:50:07PM -0600, Peter Karman wrote:
> Marvin Humphrey wrote on 1/5/10 9:28 PM:
> > There are still a couple of TODO items:
> >
> >  * JSON::XS is not thread safe, and we need to replace it with our own 
> >  custom JSON parsing module.
> isn't that necessary anyway for non-Perl host languages?

Yes, it's just been a low priority because JSON::XS is a solid and
high-performing module.

I also don't have a burning desire to implement threads, especially since
Perl's ithreads have a lot of downsides and aren't very popular.  Thread
support will be crucial for Java bindings, though -- JVM startup time is
significant enough that using processes as your concurrency model has serious
drawbacks compared to threads.  And other languages have better thread support
than Perl.

The main reason I went and worked on threads now was because McCandless and I
arrived at something like consensus over in the LUCENE-2026 thread.  I've
always thought it was important to have a concurrency plan.  Now that plan is
firming up.

> Is there enough C in JSON::XS we could borrow as a starting point?

Possible, but if we roll our own I think we should just work from scratch off
the spec, avoiding potential IP issues.

> Or, is the MIT license compatible with the Apache license, making it 
> possible to use something like ?

Possible.  It wouldn't necessarily be tailored to our data types, so we might
not be able to squeeze out absolute maximum performance.  However, it's the
next logical intermediate step after moving away from host-specific libraries
like JSON::XS.

OTOH, I hate dependencies, because it so hard to find something suitable
that's at least as platform compatible or more so than the library it would be
supporting.  (CPAN::Testers results for KinoSearch 0.30_072 so far: 32 passes,
0 fails.)  And in the C world, build scripts are wildly diverse and wildly
platform incompatible.

> Just seems like there should be a standalone, Apache-friendly JSON 
> reader/writer available in C.

There probably is, but the JSON spec is awesome for its simplicity.  It's not
that hard to write a parser.

> >In my opinion, we should stop there.
> ++

Cool, looks like consensus on the way.

Marvin Humphrey

View raw message