Thanks a lot for you response !!

For the first solution :

I need to index all the content of my websites and I want just tika ignore
<meta name="id"> because I have already an id
I'll try monday and tell you if it works

The second solution :
Are your sure Tika use the HTML Tokenizer ? I'll check

2009/12/5 Raghuveer Kancherla <raghuveer.kancherla@aplopio.com>

> 2 ways I can think of ...
>
>   - ExtractingRequestHandler (this is what I am guessing you are using now)
>
> Set extractOnly=true while making a request to the extractingRequestHandler
> and get the parsed content back. Now make a post request on update request
> handler with what ever fields and field values you want.
>



>
>   - Use HTMLStripWhiteSpaceTokenizer factory. This article may be helpful
>   to explain what I mean.
>
> http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripWhitespaceTokenizerFactory
> .
>
>
>
> - Raghu
>
>
>
> On Sat, Dec 5, 2009 at 3:44 AM, khalid y <kernity@gmail.com> wrote:
>
> > Hi,
> >
> > I have a problem with solr. I'm indexing some html content and solr crash
> > because my id field is multivalued.
> > I found that Tika read the html and extract metadata like <meta name="id"
> > content="12"> from my htmls but my documents has an already an id setted
> by
> > literal.id=10.
> >
> > I tried to map the id from Tika by fmap.id=ignored_ but it ignore also
> my
> > literal.id
> >
> > I'm using solr 1.4 and tika 0.5
> >
> > Someone can explain to me how I can ignore this the Tika id metadata ??
> >
> > Thanks
> >
>

On Dec 4, 2009, at 1:56 AM, brad anderson wrote:

> Hi,
> 
> I'm looking for a way to have the score of documents decay over time. I want
> older documents to have a lower score than newer documents.
> 
> I noted the ReciprocalFloatFunction class. In an example it seemed to be
> doing just this when you set the function to be:
> 
>     recip(ms(NOW,mydatefield),3.16e-11,1,1)
> 
> This is supposed to degrade the score to half its value if the mydatefield
> is 1 year older  than the current date. My question with this is, is it
> making the document score go down to 0.5 or is it making the document score
> 1/2 of its original value.
> 
> i.e.
> 
> The document has score 0.8
> 
> Will the score be 0.4 or 0.5 after using this function?


Actually, the value of the function gets added (it can be multiplied, too, with other params)
to the score for the document.  You can see this by adding a &debugQuery=true value to
your request which allows you to examine the explains.

> 
> Also, are there better alternatives to deal with document decay?

Some people like a different decay that does something like:  today is better than yesterday,
yesterday is better than last week and last week is better than last month, etc. (in a non-linear
way).  Todo this, you would need to implement your own function, I think.

--------------------------
Grant Ingersoll
http://www.lucidimagination.com/

Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids) using Solr/Lucene:
http://www.lucidimagination.com/search

2 ways I can think of ...

   - ExtractingRequestHandler (this is what I am guessing you are using now)

Set extractOnly=true while making a request to the extractingRequestHandler
and get the parsed content back. Now make a post request on update request
handler with what ever fields and field values you want.


   - Use HTMLStripWhiteSpaceTokenizer factory. This article may be helpful
   to explain what I mean.
   http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.HTMLStripWhitespaceTokenizerFactory.



- Raghu



On Sat, Dec 5, 2009 at 3:44 AM, khalid y <kernity@gmail.com> wrote:

> Hi,
>
> I have a problem with solr. I'm indexing some html content and solr crash
> because my id field is multivalued.
> I found that Tika read the html and extract metadata like <meta name="id"
> content="12"> from my htmls but my documents has an already an id setted by
> literal.id=10.
>
> I tried to map the id from Tika by fmap.id=ignored_ but it ignore also my
> literal.id
>
> I'm using solr 1.4 and tika 0.5
>
> Someone can explain to me how I can ignore this the Tika id metadata ??
>
> Thanks
>

Hi Otis,
I think my experiments are not conclusive about reduction in search time. I
was playing around with various configurations to reduce the time to
retrieve documents from Solr. I am sure that making the two multi valued
text fields from stored to un-stored, retrieval time (query time + time to
load the stored fields) became very fast. I was expecting the
lazyfieldloading setting in solrconfig to take care of this but apparently
it is not working as expected.

Out of curiosity, I removed these 2 fields from the index (this time I am
not even indexing them) and my search time got better (10 times better).
However, I am still trying to isolate the reason for the search time
reduction. It may be either because of 2 less fields to search in or because
of the reduction in size of the index or may be something else. I am not
sure if lazyfieldloading has any part in explaining this.

- Raghu



On Fri, Dec 4, 2009 at 3:07 AM, Otis Gospodnetic <otis_gospodnetic@yahoo.com
> wrote:

> Hm, hm, interesting.  I was looking into something like this the other day
> (BIG indexed+stored text fields).  After seeing enableLazyFieldLoading=true
> in solrconfig and after seeing "fl" didn't include those big fields, I
> though "hm, so Lucene/Solr will not be pulling those large fields from disk,
> OK".
>
> You are saying that this may not be true based on your experiment?
> And what I'm calling your "experiment" means that you reindexed the same
> data, but without the 2 multi-valued text fields... .and that was the only
> change you made and got cca x10 search performance improvement?
>
> Sorry for repeating your words, just trying to confirm and understand.
>
> Thanks,
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>
>
>
> ----- Original Message ----
> > From: Raghuveer Kancherla <raghuveer.kancherla@aplopio.com>
> > To: solr-user@lucene.apache.org
> > Sent: Thu, December 3, 2009 8:43:16 AM
> > Subject: Re: Retrieving large num of docs
> >
> > Hi Hoss,
> >
> > I was experimenting with various queries to solve this problem and in one
> > such test I remember that requesting only the ID did not change the
> > retrieval time. To be sure, I tested it again using the curl command
> today
> > and it confirms my previous observation.
> >
> > Also, enableLazyFieldLoading setting is set to true in my solrconfig.
> >
> > Another general observation (off topic) is that having a moderately large
> > multi valued text field (~200 entries) in the index seems to slow down
> the
> > search significantly. I removed the 2 multi valued text fields from my
> index
> > and my search got ~10 time faster. :)
> >
> > - Raghu
> >
> >
> > On Thu, Dec 3, 2009 at 2:14 AM, Chris Hostetter wrote:
> >
> > >
> > > : I think I solved the problem of retrieving 300 docs per request for
> now.
> > > The
> > > : problem was that I was storing 2 moderately large multivalued text
> fields
> > > : though I was not retrieving them during search time.  I reindexed all
> my
> > > : data without storing these fields. Now the response time (time for
> Solr
> > > to
> > > : return the http response) is very close to the QTime Solr is showing
> in
> > > the
> > >
> > > Hmmm....
> > >
> > > two comments:
> > >
> > > 1) the example URL from your previous mail...
> > >
> > > : >
> > >
> >
> http://localhost:1212/solr/select/?rows=300&q=%28ResumeAllText%3A%28%28%28%22java+j2ee%22+%28java+j2ee%29%29%29%5E4%29%5E1.0%29&start=0&wt=python
> > >
> > > ...doesn't match your earlier statemnet that you are only returning hte
> id
> > > field (there is no "fl" param in that URL) ... are you certain you
> werent'
> > > returning those large stored fields in teh response?
> > >
> > > 2) assuming you were actually using an fl param to limit the fields,
> make
> > > sure you have this setting in your solrconfig.xml...
> > >
> > >    true
> > >
> > > ..that should make it pretty fast to return only a few fields of each
> > > document, even if you do have some jumpto stored fields that aren't
> being
> > > returned.
> > >
> > >
> > >
> > > -Hoss
> > >
> > >
>
>

Yonik Seeley wrote:
> Is BaseCharFilter required for the html strip filter?
>
> -Yonik
> http://www.lucidimagination.com
>
>   
It could be if HTMLStripCharFilter is reverted to first version.
The first version of HTMLStripCharFilter, for example,
if we have "<p>aaa", it produces "   aaa" (3 space chars prior
to aaa). But after committed SOLR-1394, it produces " aaa"
(1 space) and now it uses correct() method of BaseCharFilter
to correct offsets.

Koji

-- 
http://www.rondhuit.com/en/

Is there any update on this requirement??


Britske wrote:
> 
> Is there way to exclude filters from a stats field, like it is possible to
> exclude filters from a facet.field? It didn't work for me. 
> 
> i.e: I have a field price, and although I filter on price, I would like to
> be able to get the entire range (min,max) of prices as if I didn't specify
> the filter. Obviously without excluding the filter the min,max range is
> constrained by [50,100]
> 
> Part of query: 
> stats=true&stats.field={!ex=p1}price&fq={!tag=p1}price:[50 TO 100]
> 
> USE-CASE:
> I show a double-slider using javascript to display possible prices. (2
> handles, one allowing to set min-price and the other to set max-price) 
> The slider has a range of [0,maxprice without price filter set]. maxprice
> is inserted by getting info from 'stats.price&stats=true'
> 
> When the user sets the slider a filter (fq) is set constraining the
> resultset the set min and max-prices. 
> After the page updates, I still want to show the price-slider, with the
> min and max handles set to the prices as selected by the user, so the user
> can alter his filter quickly.
> 
> However (and here it comes) I would also be able to get the 'maxprice
> without price filter set' because I need this to set the max-range of the
> slider. 
> 
> Is there any (undocumented) feature that makes this possible? If not,
> would it be easy to add?
> 
> Thanks, 
> Britske
> 
> 

-- 
View this message in context: http://old.nabble.com/solr-1.4%3A-multi-select-for-statscomponent-tp22202971p26656565.html
Sent from the Solr - User mailing list archive at Nabble.com.

Robin Wojciki wrote:
> I am running a search in Solr 1.4 and I am getting the
> StringIndexOutOfBoundsException pasted below. The spell check field
> uses HTMLStripCharFilterFactory. However, the search works fine if I
> do not use the HTMLStripCharFilterFactory.
>
> If I set a breakpoint at SpellCheckComponent.java: 248, the value of
> the variable "best" is as shown in the screenshot:
> http://yfrog.com/j5solrdebuginspectp
>
> At the end of first iteration, offset = 5 - (24 - 0) = -19
> This causes the index out of bounds exception.
>
> The spell check field is defined as:
>
>         <fieldType name="text_spell" class="solr.TextField"
> positionIncrementGap="100" >
>             <analyzer>
>                 <charFilter class="solr.HTMLStripCharFilterFactory"/>
>                 <tokenizer class="solr.StandardTokenizerFactory"/>
>                 <filter class="solr.StandardFilterFactory"/>
>                 <filter class="solr.LowerCaseFilterFactory"/>
>                 <filter class="solr.StopFilterFactory"
> ignoreCase="true" words="stopwords.txt"
> enablePositionIncrements="true"/>
>                 <filter class="solr.SynonymFilterFactory"
> synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
>                 <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>             </analyzer>
>         </fieldType>
>
>
>
> Stack Trace:
> =========
> String index out of range: -19
>
> java.lang.StringIndexOutOfBoundsException: String index out of range: -19
> 	at java.lang.AbstractStringBuilder.replace(Unknown Source)
> 	at java.lang.StringBuilder.replace(Unknown Source)
> 	at org.apache.solr.handler.component.SpellCheckComponent.toNamedList(SpellCheckComponent.java:248)
> 	at org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:143)
> 	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195)
> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)
> 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
> 	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338)
> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241)
> 	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
> 	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
> 	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> 	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> 	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
> 	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
> 	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
> 	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> 	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
> 	at org.mortbay.jetty.Server.handle(Server.java:285)
> 	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:502)
> 	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:821)
> 	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
> 	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:208)
> 	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:378)
> 	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:226)
> 	at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
>
>   
I couldn't reproduce it with simple test data.
Can you open a JIRA and attach a test case that reproduces
the problem with spellchecker definition in solrconfig.xml.

Koji

-- 
http://www.rondhuit.com/en/

Checking it further by looking at the code, it seems that in most cases 
it indeed adds the boost queries as SHOULD. But if you define *one* bq 
parameter which contains a boolean query, then each clause in this 
boolean query will be added to the query as is. Therefore:

This set up will filter the query:
<str name="bq">
        +category:Audio +name:black
</str>

This set up will *not* filter the query:
<str name="bq">
        +category:Audio
</str>
<str name="bq">
        +name:black
</str>

So, in the first set up, the default operator as defined in the schema 
plays a role.

Cheers,
Uri

Erik Hatcher wrote:
> Are you sure about the default operator and bq?  I assume we're 
> talking about the setting in schema.xml.
>
> I think boosting queries are OR'd in automatically to the main query:
>
> From DismaxQParser#addBoostQuery()
>   ... query.add(f, BooleanClause.Occur.SHOULD);...
>
> There is one case where query.add((BooleanClause) c); is used though.
>
>     Erik
>
>
> On Dec 5, 2009, at 6:54 AM, Uri Boness wrote:
>
>> You can actually define boost queries to do that (bq parameter). 
>> Boost queries accept the standard Lucene query syntax and eventually 
>> appended to the user query. Just make sure that the default operator 
>> is set to OR other wise these boost queries will not only influence 
>> the boosts but also filter out some of the results.
>>
>> Otis Gospodnetic wrote:
>>> Terms no, but fields (with terms) and phrases, yes.
>>>
>>>
>>> Otis
>>> -- 
>>> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>>> From: Girish Redekar <girish.redekar@aplopio.com>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Fri, December 4, 2009 11:42:16 PM
>>>> Subject: Query time boosting with dismax
>>>>
>>>> Hi,
>>>>
>>>> Is it possible to weigh specific query terms with a Dismax query 
>>>> parser? Is
>>>> it possible to write queries of the sort ...
>>>> field1:(term1)^2.0 + (term2^3.0)
>>>> with dismax?
>>>>
>>>> Thanks,
>>>> Girish Redekar
>>>> http://girishredekar.net
>>>>
>>>
>>>
>>>
>
>

Well.. this is mainly based on some experiments I did (not based on the 
code base). It appeared as if the boost queries were appended to the 
generated dismax query and if the default operator (in the schema) was 
set to AND it actually filtered out the request. For example, here's a 
dismax config:

<requestHandler name="dismax" class="solr.SearchHandler" default="true">
    <lst name="defaults">
     <str name="defType">dismax</str>
     <str name="qf">
        text^0.5 name^1.0 category^1.2
     </str>
     <str name="bq">
        *category:Audio name:black*
     </str>
     <str name="fl">
        *,score
     </str>
     ...
  </requestHandler>

When searching with a default OR operator, you receive more results than 
with an AND operator. Checking out the generated query using 
debugQuery=true reviles the following:

Generated query with default OR operator:
+DisjunctionMaxQuery((category:black^1.2 | text:black^0.5 | 
name:black)~0.01) DisjunctionMaxQuery((category:black^1.5 | 
text:black^0.5 | name:black^1.2)~0.01) *category:Audio name:black* 
FunctionQuery((product(sint(rating),const(-1.0)))^0.5)

Generated query with default AND operator:
+DisjunctionMaxQuery((category:black^1.2 | text:black^0.5 | 
name:black)~0.01) DisjunctionMaxQuery((category:black^1.5 | 
text:black^0.5 | name:black^1.2)~0.01) *+category:Audio +name:black* 
FunctionQuery((product(sint(rating),const(-1.0)))^0.5)

So when it's an AND, both clauses are marked as MUST in the overall 
query, which in turn filters the query. Indeed, I would expect it to add 
these queries as SHOULD and then the generated query would look like:
+DisjunctionMaxQuery((category:black^1.2 | text:black^0.5 | 
name:black)~0.01) DisjunctionMaxQuery((category:black^1.5 | 
text:black^0.5 | name:black^1.2)~0.01) (*+category:Audio +name:black*) 
FunctionQuery((product(sint(rating),const(-1.0)))^0.5)

Cheers,
Uri

Erik Hatcher wrote:
> Are you sure about the default operator and bq?  I assume we're 
> talking about the setting in schema.xml.
>
> I think boosting queries are OR'd in automatically to the main query:
>
> From DismaxQParser#addBoostQuery()
>   ... query.add(f, BooleanClause.Occur.SHOULD);...
>
> There is one case where query.add((BooleanClause) c); is used though.
>
>     Erik
>
>
> On Dec 5, 2009, at 6:54 AM, Uri Boness wrote:
>
>> You can actually define boost queries to do that (bq parameter). 
>> Boost queries accept the standard Lucene query syntax and eventually 
>> appended to the user query. Just make sure that the default operator 
>> is set to OR other wise these boost queries will not only influence 
>> the boosts but also filter out some of the results.
>>
>> Otis Gospodnetic wrote:
>>> Terms no, but fields (with terms) and phrases, yes.
>>>
>>>
>>> Otis
>>> -- 
>>> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>>>
>>>
>>>
>>> ----- Original Message ----
>>>
>>>> From: Girish Redekar <girish.redekar@aplopio.com>
>>>> To: solr-user@lucene.apache.org
>>>> Sent: Fri, December 4, 2009 11:42:16 PM
>>>> Subject: Query time boosting with dismax
>>>>
>>>> Hi,
>>>>
>>>> Is it possible to weigh specific query terms with a Dismax query 
>>>> parser? Is
>>>> it possible to write queries of the sort ...
>>>> field1:(term1)^2.0 + (term2^3.0)
>>>> with dismax?
>>>>
>>>> Thanks,
>>>> Girish Redekar
>>>> http://girishredekar.net
>>>>
>>>
>>>
>>>
>
>

Are you sure about the default operator and bq?  I assume we're  
talking about the setting in schema.xml.

I think boosting queries are OR'd in automatically to the main query:

 From DismaxQParser#addBoostQuery()
   ... query.add(f, BooleanClause.Occur.SHOULD);...

There is one case where query.add((BooleanClause) c); is used though.

	Erik


On Dec 5, 2009, at 6:54 AM, Uri Boness wrote:

> You can actually define boost queries to do that (bq parameter).  
> Boost queries accept the standard Lucene query syntax and eventually  
> appended to the user query. Just make sure that the default operator  
> is set to OR other wise these boost queries will not only influence  
> the boosts but also filter out some of the results.
>
> Otis Gospodnetic wrote:
>> Terms no, but fields (with terms) and phrases, yes.
>>
>>
>> Otis
>> --
>> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>>
>>
>>
>> ----- Original Message ----
>>
>>> From: Girish Redekar <girish.redekar@aplopio.com>
>>> To: solr-user@lucene.apache.org
>>> Sent: Fri, December 4, 2009 11:42:16 PM
>>> Subject: Query time boosting with dismax
>>>
>>> Hi,
>>>
>>> Is it possible to weigh specific query terms with a Dismax query  
>>> parser? Is
>>> it possible to write queries of the sort ...
>>> field1:(term1)^2.0 + (term2^3.0)
>>> with dismax?
>>>
>>> Thanks,
>>> Girish Redekar
>>> http://girishredekar.net
>>>
>>
>>
>>

On Sat, Dec 5, 2009 at 7:02 AM, Marc Sturlese <marc.sturlese@gmail.com> wrote:
>
> And what about:
> <fieldtype name="sint" class="solr.SortableIntField"
> sortMissingLast="true"/>
> vs.
> <fieldtype name="bcdint" class="solr.BCDIntField" sortMissingLast="true"/>
>
> Wich is the differenece between both? It's just bcdint always better?
> Thanks in advance

BCDInt was a very early attempt at a sortable int type that didnt go
through binary - it went directly from base 10 (the actual string
representation) to a sortable base 10000 (10K fits in a single char
and saves memory in the fieldCache), and it also had no size limit.
It's no longer referenced in any example schemas, and it doesn't have
support for function queries.

-Yonik
http://www.lucidimagination.com

And what about:
<fieldtype name="sint" class="solr.SortableIntField"
sortMissingLast="true"/>
vs.
<fieldtype name="bcdint" class="solr.BCDIntField" sortMissingLast="true"/>

Wich is the differenece between both? It's just bcdint always better?
Thanks in advance


Yonik Seeley-2 wrote:
> 
> On Fri, Dec 4, 2009 at 7:38 PM, Jay Hill <jayallenhill@gmail.com> wrote:
>> 1) Is there any benefit to using the "int" type as a TrieIntField w/
>> precisionStep=0 over the "pint" type for simple ints that won't be sorted
>> or
>> range queried?
> 
> No.  But given that people could throw in a random range query and
> have it work correctly with a trie based int (vs a plain int), seems
> reason enough to prefer it.
> 
>> 2) In 1.4, what type is now most efficient for sorting?
> 
> trie and plain should be pretty equivalent (trie might be slightly
> faster to uninvert the first time).  Both take up less memory in the
> field cache than sint.
> 
>> 3) The only reason to use a "sint" field is for backward compatibility
>> and/or to use sortMissingFirst/SortMissingLast, correct?
> 
> I believe so.
> 
> -Yonik
> http://www.lucidimagination.com
> 
> 

-- 
View this message in context: http://old.nabble.com/Sanity-check-on-numeric-types-and-which-of-them-to-use-tp26651725p26655009.html
Sent from the Solr - User mailing list archive at Nabble.com.

Hello,

 

Does anyone know of a way to tell an http SolrServer to reload its back-end index (mark cache
as dirty) periodically?

 

I have a scenario where an EmbeddedSolrServer is used for writing (for fast indexing), and
an

CommonsHttpSolrServer for reading (for remote access).

 

If the http server is used for writing, reading clients pick up any updates, as the /update
has gone 'through' the http server.

For very high indexing rates, I'd rather not have to build an http request for every write
(or group of writes), since the writer is always on the same machine as the index.

 

Any help on this is much appreciated.

 

Thanks,

Peter

 
 		 	   		  
_________________________________________________________________
View your other email accounts from your Hotmail inbox. Add them now.
http://clk.atdmt.com/UKM/go/186394592/direct/01/

You can actually define boost queries to do that (bq parameter). Boost 
queries accept the standard Lucene query syntax and eventually appended 
to the user query. Just make sure that the default operator is set to OR 
other wise these boost queries will not only influence the boosts but 
also filter out some of the results.

Otis Gospodnetic wrote:
> Terms no, but fields (with terms) and phrases, yes.
>
>
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
>
>
>
> ----- Original Message ----
>   
>> From: Girish Redekar <girish.redekar@aplopio.com>
>> To: solr-user@lucene.apache.org
>> Sent: Fri, December 4, 2009 11:42:16 PM
>> Subject: Query time boosting with dismax
>>
>> Hi,
>>
>> Is it possible to weigh specific query terms with a Dismax query parser? Is
>> it possible to write queries of the sort ...
>> field1:(term1)^2.0 + (term2^3.0)
>> with dismax?
>>
>> Thanks,
>> Girish Redekar
>> http://girishredekar.net
>>     
>
>
>   

Terms no, but fields (with terms) and phrases, yes.


Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Girish Redekar <girish.redekar@aplopio.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, December 4, 2009 11:42:16 PM
> Subject: Query time boosting with dismax
> 
> Hi,
> 
> Is it possible to weigh specific query terms with a Dismax query parser? Is
> it possible to write queries of the sort ...
> field1:(term1)^2.0 + (term2^3.0)
> with dismax?
> 
> Thanks,
> Girish Redekar
> http://girishredekar.net

Local Solr doesn't look like 64 bit.

rajan@rajan-desktop:~$ uname -a
Linux rajan-desktop 2.6.28-16-server #55-Ubuntu SMP Tue Oct 20 20:50:00 UTC
2009 i686 GNU/Linux


But the Xen Solr server does

rajan@rajan-desktop:~$ uname -a
Linux rajan-desktop 2.6.28-16-server #55-Ubuntu SMP Tue Oct 20 20:50:00 UTC
2009 i686 GNU/Linux


May be that is the reason why Server is taking more RAM.

Thanks all for your responses.

Regards
Rajan

On Sat, Dec 5, 2009 at 11:06 AM, rajan chandi <chandi.rajan@gmail.com>wrote:

> My local ubuntu 9.04 64 bit taking 1.5 GB is not a VM and Debian Lenny 64
> bit taking 2 GB is a Xen Instance.
>
> - Rajan
>
>
> On Sat, Dec 5, 2009 at 10:51 AM, rajan chandi <chandi.rajan@gmail.com>wrote:
>
>> We are using 64 bit VM with 64 bit JDK on it.
>> It is 2.00 GB RAM Zen instance.
>>
>> We're setting up max JVM heap size of 1800 MB max.
>>
>> - Rajan
>>
>>
>>
>> On Fri, Dec 4, 2009 at 8:19 PM, Yonik Seeley <yonik@lucidimagination.com>wrote:
>>
>>> Are you explicitly setting the heap sizes?  If not, the JVM is
>>> deciding for itself based on what the box looks like (ram, cpus, OS,
>>> etc).  Are they both the same architecture (32 bit or 64 bit?)
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>> p.s. in general cross-posting to both solr-user and solr-dev is
>>> discouraged.
>>>
>>>
>>> On Fri, Dec 4, 2009 at 5:27 AM, rajan chandi <chandi.rajan@gmail.com>
>>> wrote:
>>> > Hi All,
>>> >
>>> > We've deployed 4 instances of Solr on a debian server.
>>> >
>>> > It is taking only 1.5 GB of RAM on local ubuntu machine but it is
>>> taking 2.0
>>> > GB plus on Debian Lenny server.
>>> >
>>> > Any ideas/pointers will help.
>>> >
>>> > Regards
>>> > Rajan
>>> >
>>>
>>
>>
>

My local ubuntu 9.04 64 bit taking 1.5 GB is not a VM and Debian Lenny 64
bit taking 2 GB is a Xen Instance.

- Rajan

On Sat, Dec 5, 2009 at 10:51 AM, rajan chandi <chandi.rajan@gmail.com>wrote:

> We are using 64 bit VM with 64 bit JDK on it.
> It is 2.00 GB RAM Zen instance.
>
> We're setting up max JVM heap size of 1800 MB max.
>
> - Rajan
>
>
>
> On Fri, Dec 4, 2009 at 8:19 PM, Yonik Seeley <yonik@lucidimagination.com>wrote:
>
>> Are you explicitly setting the heap sizes?  If not, the JVM is
>> deciding for itself based on what the box looks like (ram, cpus, OS,
>> etc).  Are they both the same architecture (32 bit or 64 bit?)
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>> p.s. in general cross-posting to both solr-user and solr-dev is
>> discouraged.
>>
>>
>> On Fri, Dec 4, 2009 at 5:27 AM, rajan chandi <chandi.rajan@gmail.com>
>> wrote:
>> > Hi All,
>> >
>> > We've deployed 4 instances of Solr on a debian server.
>> >
>> > It is taking only 1.5 GB of RAM on local ubuntu machine but it is taking
>> 2.0
>> > GB plus on Debian Lenny server.
>> >
>> > Any ideas/pointers will help.
>> >
>> > Regards
>> > Rajan
>> >
>>
>
>

We are using 64 bit VM with 64 bit JDK on it.
It is 2.00 GB RAM Zen instance.

We're setting up max JVM heap size of 1800 MB max.

- Rajan


On Fri, Dec 4, 2009 at 8:19 PM, Yonik Seeley <yonik@lucidimagination.com>wrote:

> Are you explicitly setting the heap sizes?  If not, the JVM is
> deciding for itself based on what the box looks like (ram, cpus, OS,
> etc).  Are they both the same architecture (32 bit or 64 bit?)
>
> -Yonik
> http://www.lucidimagination.com
>
> p.s. in general cross-posting to both solr-user and solr-dev is
> discouraged.
>
>
> On Fri, Dec 4, 2009 at 5:27 AM, rajan chandi <chandi.rajan@gmail.com>
> wrote:
> > Hi All,
> >
> > We've deployed 4 instances of Solr on a debian server.
> >
> > It is taking only 1.5 GB of RAM on local ubuntu machine but it is taking
> 2.0
> > GB plus on Debian Lenny server.
> >
> > Any ideas/pointers will help.
> >
> > Regards
> > Rajan
> >
>

Hi,

Is it possible to weigh specific query terms with a Dismax query parser? Is
it possible to write queries of the sort ...
field1:(term1)^2.0 + (term2^3.0)
with dismax?

Thanks,
Girish Redekar
http://girishredekar.net

On Fri, Dec 4, 2009 at 7:38 PM, Jay Hill <jayallenhill@gmail.com> wrote:
> 1) Is there any benefit to using the "int" type as a TrieIntField w/
> precisionStep=0 over the "pint" type for simple ints that won't be sorted or
> range queried?

No.  But given that people could throw in a random range query and
have it work correctly with a trie based int (vs a plain int), seems
reason enough to prefer it.

> 2) In 1.4, what type is now most efficient for sorting?

trie and plain should be pretty equivalent (trie might be slightly
faster to uninvert the first time).  Both take up less memory in the
field cache than sint.

> 3) The only reason to use a "sint" field is for backward compatibility
> and/or to use sortMissingFirst/SortMissingLast, correct?

I believe so.

-Yonik
http://www.lucidimagination.com

Looking at the example version of schema.xml there seems to be some
confusion on which numeric field types are best used in different
situations. What confused me was that the type of "int" is now set to a
TrieIntField, but with a precisionStep of 0:
    <fieldType name="int" class="solr.TrieIntField" precisionStep="0"
omitNorms="true" positionIncrementGap="0"/>'
the "tint" type is set up as a TrieIntField with a precisionStep of 8:
    <fieldType name="tint" class="solr.TrieIntField" precisionStep="8"
omitNorms="true" positionIncrementGap="0"/>
the "sint" type is unchanged:
    <fieldType name="sint" class="solr.SortableIntField"
sortMissingLast="true" omitNorms="true"/>
and the old IntField is now of type "pint":
    <fieldType name="pint" class="solr.IntField" omitNorms="true"/>

It's obvious that the "tint" type would be preferred for range queries. But
these questions come to mind:
1) Is there any benefit to using the "int" type as a TrieIntField w/
precisionStep=0 over the "pint" type for simple ints that won't be sorted or
range queried?
2) In 1.4, what type is now most efficient for sorting?
3) The only reason to use a "sint" field is for backward compatibility
and/or to use sortMissingFirst/SortMissingLast, correct?

-Jay

On Dec 4, 2009, at 4:21 PM, javaxmlsoapdev wrote:

>
> how do I create a query string witih multiple fq params using solrj  
> SolrQuery
> API.
>
> e.g. I want to build a query as follow
>
> http://servername:port/solr/issues/select/?q=testing&fq=statusName: 
> (Female
> OR Male)&fq=name="Joe"
>
> I am using solrj client APIs to build query and using SolrQuery as  
> follow
>
> solrQuery.setParam("fq" statusString);
> solrQuery.setParam("fq", nameString);
>
> It only sets last "fq" (fq=nameString)in the string.. If I swich  
> abover
> setParam order it sets fq=statusString. How do I set muliple fq  
> params in
> SolrQuery object.

Use SolrQuery#add() instead.  Or SolrQuery#addFilterQuery()

	Erik

Hi,

I have a problem with solr. I'm indexing some html content and solr crash
because my id field is multivalued.
I found that Tika read the html and extract metadata like <meta name="id"
content="12"> from my htmls but my documents has an already an id setted by
literal.id=10.

I tried to map the id from Tika by fmap.id=ignored_ but it ignore also my
literal.id

I'm using solr 1.4 and tika 0.5

Someone can explain to me how I can ignore this the Tika id metadata ??

Thanks

Teruhiko Kurosaka wrote:
>> Aha!
>> Sounds like a job for a simple, custom 
>> UpdateRequestProcessor.  Actually, I think URP doesn't get 
>> access to the actual XML, but what it has access may be 
>> enough for you: http://wiki.apache.org/solr/UpdateRequestProcessor
>>     
>
> I added this to solrconfig.xml but I don't see any extra output 
> in the log file.
>
>   <updateRequestProcessorChain>
>     <processor class="solr.LogUpdateProcessorFactory" />
>     <processor class="solr.RunUpdateProcessorFactory" />
>   </updateRequestProcessorChain>
>
> Do I need to do something else to make this effective?
>
> The commented-out example in solrconfic.xml has 'name="dedupe"'
> attribute. Do I have to specify a name? If so, how do I use 
> that name in the request?
>
> Kuro
>   
Look at the comment above the dedupe declaration:

       You have to link the chain to an update handler above to use it ie:
         <requestHandler name="/update
"class="solr.XmlUpdateRequestHandler">
           <lst name="defaults">
             <str name="update.processor">dedupe</str>
           </lst>
         </requestHandler> 

-- 
- Mark

http://www.lucidimagination.com

> Aha!
> Sounds like a job for a simple, custom 
> UpdateRequestProcessor.  Actually, I think URP doesn't get 
> access to the actual XML, but what it has access may be 
> enough for you: http://wiki.apache.org/solr/UpdateRequestProcessor

I added this to solrconfig.xml but I don't see any extra output 
in the log file.

  <updateRequestProcessorChain>
    <processor class="solr.LogUpdateProcessorFactory" />
    <processor class="solr.RunUpdateProcessorFactory" />
  </updateRequestProcessorChain>

Do I need to do something else to make this effective?

The commented-out example in solrconfic.xml has 'name="dedupe"'
attribute. Do I have to specify a name? If so, how do I use 
that name in the request?

Kuro

What are the nature of the machines / VM run on ?  32-bit / 64-bit ?

rajan chandi wrote:
> Hi All,
>
> We've deployed 4 instances of Solr on a debian server.
>
> It is taking only 1.5 GB of RAM on local ubuntu machine but it is taking 2.0
> GB plus on Debian Lenny server.
>
> Any ideas/pointers will help.
>
> Regards
> Rajan
>
>   

Is BaseCharFilter required for the html strip filter?

-Yonik
http://www.lucidimagination.com


On Tue, Dec 1, 2009 at 1:17 AM, Koji Sekiguchi <koji@r.email.ne.jp> wrote:
> Robin,
>
> Thank you for reporting this. Performance degradation of HTML Stripper
> could be in 1.4. I opened a ticket in Lucene:
>
> https://issues.apache.org/jira/browse/LUCENE-2098
>
> Koji
>
> --
> http://www.rondhuit.com/en/
>
>

Thanks Lance, I appreciate your response.  

I know what a DIH is and have already written custom transformers.  I just
misunderstood your response to my message (I wasnt aware that we could use
JS to create transformers).

Anyhow, my intent is to change the tool (create a variation of hsin to
support degrees) rather than change the data (which introduces other issues,
such as having to support most systems in degrees and this one system in
radians)

any ideas/advice in that regard?
-- 
View this message in context: http://old.nabble.com/how-is-score-computed-with-hsin-functionquery--tp26504265p26638720.html
Sent from the Solr - User mailing list archive at Nabble.com.

how do I create a query string witih multiple fq params using solrj SolrQuery
API.

e.g. I want to build a query as follow

http://servername:port/solr/issues/select/?q=testing&fq=statusName:(Female
OR Male)&fq=name="Joe"

I am using solrj client APIs to build query and using SolrQuery as follow

solrQuery.setParam("fq" statusString);
solrQuery.setParam("fq", nameString);

It only sets last "fq" (fq=nameString)in the string.. If I swich abover
setParam order it sets fq=statusString. How do I set muliple fq params in
SolrQuery object.

Thanks,
-- 
View this message in context: http://old.nabble.com/how-to-set-multiple-fq-while-building-a-query-in-solrj-tp26638650p26638650.html
Sent from the Solr - User mailing list archive at Nabble.com.

Aha!
Sounds like a job for a simple, custom UpdateRequestProcessor.  Actually, I think URP doesn't
get access to the actual XML, but what it has access may be enough for you: http://wiki.apache.org/solr/UpdateRequestProcessor

Alternatively, unpack the war, add a custom logging servlet filter, chain it in web.xml and
that might do the trick.

Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Teruhiko Kurosaka <Kuro@basistech.com>
> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
> Sent: Fri, December 4, 2009 3:05:57 PM
> Subject: RE: Dumping solr requests for indexing
> 
> Log only tells high-level descriptions of what were done.
> I'd like to capture the exact XML requests with data, so that
> I could re-feed it to Solr to reproduce the issue my
> customer is encountering.
> 
> -kuro  
> 
> > -----Original Message-----
> > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
> > Sent: Friday, December 04, 2009 11:41 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Dumping solr requests for indexing
> > 
> > The solr log, as well as the servlet container log should 
> > have them all.
> > 
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
> > 
> > 
> > 
> > ----- Original Message ----
> > > From: Teruhiko Kurosaka 
> > > To: "solr-user@lucene.apache.org" 
> > > Sent: Fri, December 4, 2009 2:23:17 PM
> > > Subject: Dumping solr requests for indexing
> > > 
> > > Is there any way to dump all incoming requests to Solr into a file?
> > > 
> > > My customer is seeing a strange problem of disappearing docs from 
> > > index and I'd like to ask them to capture all incoming requests.
> > > 
> > > Thanks.
> > > 
> > > -kuro
> > 
> > 

Oops, of course the answer was staring me in the face!

   --> Use the EmbeddedSolrServer, rather than the CommonsHttpSolrServer.

 

Live and learn. Live. and learn.

 

Thanks,

Peter

 


 
> From: peter4u@hotmail.com
> To: solr-user@lucene.apache.org
> Subject: Question: Write to Solr but not via http, and still store date_format
> Date: Fri, 4 Dec 2009 20:09:19 +0000
> 
> 
> Hi Solr team,
> 
> 
> 
> Has anyone been able to write to Solr, keeping things like 'date_format', but indexing
directly, rather than via http?
> 
> 
> 
> I've been indexing using Lucene Java, and this works well and is very fast, except that
any data indexed this way doesn't store date_format et al information (date.format resuts
always return 0).
> 
> I like indexing directly into Lucene, rather than via http requests, as it is much faster,
particularly at very high input rates.
> 
> 
> 
> Anyone encountered this and managed to solve it?
> 
> 
> 
> Many thanks,
> 
> peter
> 
> 
> 
> _________________________________________________________________
> Got more than one Hotmail account? Save time by linking them together
> http://clk.atdmt.com/UKM/go/186394591/direct/01/
 		 	   		  
_________________________________________________________________
Got more than one Hotmail account? Save time by linking them together
 http://clk.atdmt.com/UKM/go/186394591/direct/01/

Are you looking for http://wiki.apache.org/solr/EmbeddedSolr ?

 Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Peter 4U <peter4u@hotmail.com>
> To: Solr <solr-user@lucene.apache.org>
> Sent: Fri, December 4, 2009 3:09:19 PM
> Subject: Question: Write to Solr but not via http, and still store date_format
> 
> 
> Hi Solr team,
> 
> 
> 
> Has anyone been able to write to Solr, keeping things like 'date_format', but 
> indexing directly, rather than via http?
> 
> 
> 
> I've been indexing using Lucene Java, and this works well and is very fast, 
> except that any data indexed this way doesn't store date_format et al 
> information (date.format resuts always return 0).
> 
> I like indexing directly into Lucene, rather than via http requests, as it is 
> much faster, particularly at very high input rates.
> 
> 
> 
> Anyone encountered this and managed to solve it?
> 
> 
> 
> Many thanks,
> 
> peter
> 
> 
>                           
> _________________________________________________________________
> Got more than one Hotmail account? Save time by linking them together
> http://clk.atdmt.com/UKM/go/186394591/direct/01/

Hi Solr team,

 

Has anyone been able to write to Solr, keeping things like 'date_format', but indexing directly,
rather than via http?

 

I've been indexing using Lucene Java, and this works well and is very fast, except that any
data indexed this way doesn't store date_format et al information (date.format resuts always
return 0).

I like indexing directly into Lucene, rather than via http requests, as it is much faster,
particularly at very high input rates.

 

Anyone encountered this and managed to solve it?

 

Many thanks,

peter

 
 		 	   		  
_________________________________________________________________
Got more than one Hotmail account? Save time by linking them together
 http://clk.atdmt.com/UKM/go/186394591/direct/01/

Log only tells high-level descriptions of what were done.
I'd like to capture the exact XML requests with data, so that
I could re-feed it to Solr to reproduce the issue my
customer is encountering.

-kuro  

> -----Original Message-----
> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com] 
> Sent: Friday, December 04, 2009 11:41 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Dumping solr requests for indexing
> 
> The solr log, as well as the servlet container log should 
> have them all.
> 
> Otis
> --
> Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch
> 
> 
> 
> ----- Original Message ----
> > From: Teruhiko Kurosaka <Kuro@basistech.com>
> > To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
> > Sent: Fri, December 4, 2009 2:23:17 PM
> > Subject: Dumping solr requests for indexing
> > 
> > Is there any way to dump all incoming requests to Solr into a file?
> > 
> > My customer is seeing a strange problem of disappearing docs from 
> > index and I'd like to ask them to capture all incoming requests.
> > 
> > Thanks.
> > 
> > -kuro
> 
> 

Would http://wiki.apache.org/solr/FunctionQuery#fieldvalue help?

 Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: William Pierce <evalsinca@hotmail.com>
> To: solr-user@lucene.apache.org
> Sent: Fri, December 4, 2009 2:43:25 PM
> Subject: Best way to handle bitfields in solr...
> 
> Folks:
> 
> In my db I currently have fields that represent bitmasks.   Thus, for example, a 
> value of the mask of 48 might represent an "undergraduate" (value = 16) and 
> "graduate" (value = 32).   Currently,  the corresponding field in solr is a 
> multi-valued string field called "EdLevel" which will have 
> Undergraduate and Graduate  as its two values (for 
> this example).   I do the conversion from the int into the list of values as I 
> do the indexing.
> 
> Ideally, I'd like solr to have bitwise operations so that I could store the int 
> value, and then simply search by using bit operations.  However, given that this 
> is not possible,  and that there have been recent threads speaking to 
> performance issues with multi-valued fields,  is there something better I could 
> do?
> 
> TIA,
> 
> - Bill

Folks:

In my db I currently have fields that represent bitmasks.   Thus, for example, a value of
the mask of 48 might represent an "undergraduate" (value = 16) and "graduate" (value = 32).
  Currently,  the corresponding field in solr is a multi-valued string field called "EdLevel"
which will have <value>Undergraduate</value> and <value>Graduate</value>
 as its two values (for this example).   I do the conversion from the int into the list of
values as I do the indexing.

Ideally, I'd like solr to have bitwise operations so that I could store the int value, and
then simply search by using bit operations.  However, given that this is not possible,  and
that there have been recent threads speaking to performance issues with multi-valued fields,
 is there something better I could do?

TIA,

- Bill

The solr log, as well as the servlet container log should have them all.

Otis
--
Sematext -- http://sematext.com/ -- Solr - Lucene - Nutch



----- Original Message ----
> From: Teruhiko Kurosaka <Kuro@basistech.com>
> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
> Sent: Fri, December 4, 2009 2:23:17 PM
> Subject: Dumping solr requests for indexing
> 
> Is there any way to dump all incoming requests to Solr
> into a file?
> 
> My customer is seeing a strange problem of disappearing
> docs from index and I'd like to ask them to capture all
> incoming requests.
> 
> Thanks.
> 
> -kuro 

Is there any way to dump all incoming requests to Solr
into a file?

My customer is seeing a strange problem of disappearing
docs from index and I'd like to ask them to capture all
incoming requests.

Thanks.

-kuro 

Have you gone through the solr tomcat wiki?

http://wiki.apache.org/solr/SolrTomcat

I found this very helpful when I did our solr installation on tomcat.

- Bill

--------------------------------------------------
From: "Jill Han" <jill.han@alverno.edu>
Sent: Friday, December 04, 2009 8:54 AM
To: <solr-user@lucene.apache.org>
Subject: RE: search on tomcat server

> I went through all the links on 
> http://wiki.apache.org/solr/#Search_and_Indexing
> And still have no clue as how to proceed.
> 1. do I have to do some implementation in order to get solr to search doc. 
> on tomcat server?
> 2. if I have files, such as .doc, docx, .pdf, .jsp, .html, etc under 
> window xp, c:/tomcat/webapps/test1, /webapps/test2,
>   What should I do to make solr search those directories
> 3. since I am using tomcat, instead of jetty, is there any demo that shows 
> the solr searching features, and real searching result?
>
> Thanks,
> Jill
>
>
> -----Original Message-----
> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
> Sent: Monday, November 30, 2009 10:40 AM
> To: solr-user@lucene.apache.org
> Subject: Re: search on tomcat server
>
> On Mon, Nov 30, 2009 at 9:55 PM, Jill Han <jill.han@alverno.edu> wrote:
>
>> I got solr running on the tomcat server,
>> http://localhost:8080/solr/admin/
>>
>> After I enter a search word, such as, solr, then hit Search button, it
>> will go to
>>
>> http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&in
>> dent=on<http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&in%0Adent=on>
>>
>>  and display
>>
>>   <?xml version="1.0" encoding="UTF-8" ?>
>>
>> -
>> <http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i
>> ndent=on<http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i%0Andent=on>>
>>  <<response>
>>
>> -
>> <http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i
>> ndent=on<http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i%0Andent=on>>
>>  <  <lst name="responseHeader">
>>
>>  <    <int name="status">0</int>
>>
>>  <    <int name="QTime">0</int>
>>
>> -
>> <http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i
>> ndent=on<http://localhost:8080/solr/select/?q=solr&version=2.2&start=0&rows=10&i%0Andent=on>>
>>  <    <lst name="params">
>>
>>  <      <str name="rows">10</str>
>>
>>  <      <str name="start">0</str>
>>
>>  <      <str name="indent">on</str>
>>
>>  <      <str name="q">solr</str>
>>
>>  <      <str name="version">2.2</str>
>>
>>     </lst>
>>
>>   </lst>
>>
>>  <  <result name="response" numFound="0" start="0" />
>>
>>  </response>
>>
>>  My question is what is the next step to search files on tomcat server?
>>
>>
>>
> Looks like you have not added any documents to Solr. See the "Indexing
> Documents" section at http://wiki.apache.org/solr/#Search_and_Indexing
>
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 

Hi everyone,

New to forum and to Solr, doing my first major project with it and enjoying
it so far, great software.

In my web application I want to set up auto-suggest as you type
functionality which will search case-insensitively yet return the original
case terms.  It doesn't seem like TermsComponent can do this as it can only
return the lowercase indexed terms your are searching against, not the
original case terms.

There was one post on this forum 
http://old.nabble.com/Auto-suggest..-how-to-do-mixed-case-td24106666.html#a24143981
http://old.nabble.com/Auto-suggest..-how-to-do-mixed-case-td24106666.html#a24143981 
where someone asked the same question, and what someone said is to

There is no way to do this right now using TermsComponent. You can index
lower case terms and store the mixed case terms. Then you can use a prefix
query which will return documents (and hence stored field values).

So this got me started, I set out to use Solr Query instead of
TermsComponent to try to do this.  I did the following as mentioned:

<fieldType name="test" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.KeywordTokenizerFactory"/>
  </analyzer>
</fieldType>

<fieldType name="test_lc" class="solr.TextField" positionIncrementGap="100">
  <analyzer>
    <tokenizer class="solr.KeywordTokenizerFactory"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

<field name="test" type="test" indexed="false" stored="true"
multiValued="true" />
<field name="test_lc" type="test_lc" indexed="true"  stored="false"
multiValued="true" />

And used copyField to populate the test_lc field:

<copyField source="test" dest="test_lc"/>

This is the easy part (the forum user didn't explain the hard part!) It is
very hard to get the same information that TermsComponent returns using the
regular Solr Query functionality!  For example:

http://localhost:8983/solr/terms?terms.fl=test_lc&terms.prefix=a&terms.sort=count&terms.limit=5&omitHeader=true

<int name="a-kinase anchor protein 13">15</int>
<int name="accn5">6</int>
<int name="actin-binding">3</int>
<int name="activator">1</int>
<int name="agie-bp1">1</int>

which provides useful sorting by and returning of term frequency counts in
your index.  How does one get this same information with regular Solr Query? 
I set up the following prefix query, searching by the indexed lowercased
field and returning the other:

http://localhost:8983/solr/select?fl=test&q=test_lc%3Aa*&sort=score+desc&rows=5&omitHeader=true

<doc>
  <arr name="test">
    <str>3D-structure</str>
    <str>acetylation</str>
    <str>alternative promoter usage</str>
    <str>HLC-7</str>
  </arr>
</doc>
<doc>
  <arr name="test">
    <str>alternative splicing</str>
    <str>complete proteome</str>
    <str>DNA-binding</str>
    <str>RACK1</str>
  </arr>
</doc>
<doc>
  <arr name="test">
    <str>acetylation</str>
    <str>AIG21</str>
    <str>WD repeat</str>
    <str>GNB2L1</str>
  </arr>
</doc>
<doc>
</arr>
  <arr name="test">
    <str>3D-structure</str>
    <str>apoptosis</str>
    <str>cathepsin G-like 1</str>
    <str>ATSGL1</str>
    <str>CTLA-1</str>
  </arr>
</doc>
<doc>
  <arr name="test">
    <str>autoantigen Ge-1</str>
    <str>autoantigen RCD-8</str>
    <str>HERV-H LTR-associating protein 3</str>
    <str>HHLA3</str>
  </arr>
</doc>

I can see how to process this in my front-end app to extract the original
terms starting with the prefix letter(s) used in the query, but there are
still some major problems when compared to TermsComponent:

- How do I make sure my auto-suggest list is at least a certain number of
terms long?  Using rows of course doesn't work like terms.limit, because
between returned docs there can be the same term and these will get
collapsed.
- How do I get term frequency counts like TermsComponent does?  I looked at
faceting but I don't understand how to get the TermsComponent behavior using
it.

Sorry for the long message, just wanted to fully explain, thanks for any
help!

leandro

-- 
View this message in context: http://old.nabble.com/how-to-do-auto-suggest-case-insensitive-match-and-return-original-case-field-values-tp26636365p26636365.html
Sent from the Solr - User mailing list archive at Nabble.com.