lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From swsong_dev <swsong_...@websqrd.com>
Subject Re: How can I make better project than Lucene?
Date Sat, 15 Nov 2014 23:23:06 GMT
Thank you for your sincere reply, Mr. McCandless.

When I posted an email in a mailing-list, I was afraid for not getting a considerable reply,
but I’m now so glad I might find a way.

I agree that a new search engine in Go would be competitive. I think we all need a next generation
search engine core redesigned from the start.

And, I understand Lucene’s limitations you mentioned. They are good points to get started.

I have been used a search engine for first 4 years and developed a search engines for last
6 years from the bottom, and I got feedback “It’s faster than Solr in indexing and searching”.
(http://ddakker.tistory.com/248 <http://ddakker.tistory.com/248>)

===Result===
Data size : 529,188
Fastcat indexing time : 1m 26s
Solr3.5 indexing time : 5m 30s
Fastcat searching time : 48ms
Solr3.5 searching time : 73ms

It applied to Korea’s greatest shopping service(http://danawa.com/ <http://danawa.com/>)
a month ago to my delight.

But my goal has been making a globally-used open source search engine.

As you suggested, now I want to make a whole-new search engine in Go.

I have made my first search engine alone, but I would not make a new search engine alone.
I want to make it with global developers together.

If you plan to make a new search engine in Go, or know someone around you, could you help
me gathering members for a new search engine, and guide us technically(feature requirement,
efficient design)?

Or if there is already a new search engine project in Go, could you let me know?

In Korea, no one develops a search engine except people who work at a search engine solution
company, and even they are very few and do not spend time to an open source project.

In my case, I found a tiny venture company for making time to develop an open source search
engine 4 year ago.

I want to be involved in a next-generation search engine project. I would be happy to make
a new search engine itself.

Your little help could be great for me.

Thank you.

Sang Song


> 2014. 11. 15., 오후 8:22, Michael McCandless <lucene@mikemccandless.com> 작성:
> 
> Actually I think competing projects is very healthy for open source development.
> 
> There are many things you could explore to "contrast" with Lucene,
> e.g. write your new search engine in Go not Java: Java has many
> problems, maybe Go fixes them.  Go also has a low-latency garbage
> collector in development ... and Java's GC options still can't scale
> to the heap sizes that are practical now.
> 
> Lucene has many limitations, so your competing engine could focus on
> them.  E.g. the "schemalessness" of Lucene has become a big problem,
> and near impossible to fix at this point, and prevents new important
> features like LUCENE-5879 from being possible, so you could give your
> engine a "gentle" schema from the start.
> 
> The Lucene Filter/Query situation is a mess: one should extend the other.
> 
> Lucene has weak support for proximity queries (SpanQuery is slow and
> does not get much attention).
> 
> Lucene is showing its age, missing some compelling features like a
> builtin transaction log, "core" support for numerics (they are sort of
> hacked on top), optimistic concurrency support (sequence ids,
> versions, something), distributed support (near real time replication,
> etc.), multi-tenancy, an example server implementation, so the search
> servers on top of Lucene have had to fill these gaps.  Maybe you could
> make your engine distributed from the start (Go is a great match for
> that, from what little I know).
> 
> All 3 highlighter options have problems.
> 
> The analysis chain (attributes) is overly complex.
> 
> In your competing engine you can borrow/copy/steal from Lucene's good
> parts to get started...
> 
> 
> Mike McCandless
> 
> http://blog.mikemccandless.com
> 
> 
> On Fri, Nov 14, 2014 at 8:43 PM, swsong_dev <swsong_dev@websqrd.com> wrote:
>> I’m developing search engine, Fastcatsearch. http://github <hthttp://githubtp//github>.com/fastcatsearch/fastcatsearch
>> 
>> Lucene is widely known and famous project and I cannot beat Lucene for now.
>> 
>> But is there any chance to beat Lucene?
>> 
>> Anything like features, performance.
>> 
>> Please, let me know what to do to make better product than Lucene.
>> 
>> Thank you.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message