lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Henri Biestro (JIRA)" <>
Subject [jira] Commented: (SOLR-646) Configuration properties in multicore.xml
Date Sun, 03 Aug 2008 14:27:44 GMT


Henri Biestro commented on SOLR-646:

About documentation:
I'll try to come with better (ie more meaningful) examples - although I kinda feel "impaired"
to describe the 3 tags (property/import/fragment) if what I've already written does not convey
90% of how things would/should work already. I welcome any help/hint on what a useful documentation
should look like (any template I could/should follow)?
On the documentation editing/releasing process, I'd tend to go for refining this issue description
till the code gets committed at which time I'd copy/paste it in a Wiki page if that's OK.

About "automatic" properties:
Those reflect the Core descriptor values or in more general terms, any Solr install/multicore/core
configuration/schema value used in the system but defined through another API; the core name,
instance directory, configuration file name (defaults to solrconfig.xml), schema file name
(defaults to schema.xml) all come form the core descriptor .
solr.core.instanceDir is strictly what the core descriptor instance directory is; solr.core.instancePath
is the non-relative version of it. When solr.core.instanceDir is *not* an absolute path, is
is relative to the solr.home instance directory; in this case, solr.core.instancePath is the
non relative version. Otherwise, they are just the same; overall, this just reflect what the
code does but removes any ambiguous implicits (which is what the PropertyMap.imports() also

I've also added the solr.core.schemaDir & solr.core.configDir- the directories where the
config & schema files reside. We have no way yet to specify those last two but they are
inferred from the other values - which would be their defaults if un-specified.There is a
functional choice to make about how we want to complete the "core" properties; the "missing"
parts are the config & schema directories. My preference would be to extend the core descriptor
and reflect those as properties but we might want&try to go the other way, put everything
as properties & retain the API as shortcuts.

Regarding naming conventions/protection, I would tend to go for the natural attribute-like
convention & protect those; the automatic/reserved namespaces would be any property starting
with 'solr.' (including 'solr.core' and 'solr.multicore') and disallow/warn about any attempt
to override. Would that be acceptable/enough ?

About files/substituted properties & admin JSPs:
As is, nothing has changed; getting the schema/config return the file as it exists in the
file-system. The first natural extension would be that for config & schema, when allowed,
the imported files should be downloadable too. It might be nice to have the download expanded
version of each file for debugging purpose as well if this is actually useful and/or the expanded
version of config & schema; any functional suggestions is welcome. Lance, what do you
think would be good to have?

About PropertyMap.imports():
The rationale behind this -besides debugging/logging purpose- would be "dynamic" replication
of a core (a slave Slr 'install' could automatically replicate new "master' cores. The goal
of this API is to have the canonical (local) path of each file - so we can read them- and
their logical/symbolic counterparts so their destination (remote) path can be determined,
each core's solrconfig.xml itself being "replicable".

About the issue itself:
How much of these should be specified/implemented before the patch can be committed and/or
is there any chance to put this in the 1.3 scope ?

> Configuration properties in multicore.xml
> -----------------------------------------
>                 Key: SOLR-646
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>    Affects Versions: 1.3
>            Reporter: Henri Biestro
>             Fix For: 1.3
>         Attachments: solr-646.patch, solr-646.patch
> This patch refers to 'generalized configuration properties' as specified by [HossMan|]
> This means configuration & schema files can use expression based on properties defined
in multicore.xml.
> h3. Use cases:
> Share the same schema and/or config file between multiple cores but point to different
dataDirs using a <dataDir>${core.datadir}</dataDir>
> Change the general layout between data, config & schema directories.
> Define 'installation' wide properties (for replication for instance) in
(different base/install/data directories on different hosts).
> h3. Syntax:
> h4. Defining properties:
> Either in the file (usual property format):
> {code:xml}
> env.value=an installation value
> env.other_value=another installation value
> {code}
> In the multicore.xml:
> {code:xml}
> <multicore'>
>   <property name='mp0'>a value</property>  <!-- a basic property -->
>   <property name='mp1'>${env.value}</property>  }<!-- a property whose
value is an expression -->
>   <core name='core_name'>
>      <property name='p0'>some value</property>
>      <property name='p1'>${mp0}-and some data</property>
>   </core>
> </multicore>
> {code}
> h4. Using properties:
> Besides used defined properties, the following core descriptor properties are automatically
defined in each core context, namely:
> {code}
> solr.core.instanceDir
> solr.core.instancePath
> solr.core.configName
> solr.core.schemaName
> {code}
> All properties can be used in any attribute or text node of a solrconfig.xml or schema.xml
as in:
> {code:xml}
> <dataDir>${core.dataDir}</dataDir>
> {code}
> h4. Technical specifications
> Multicore.xml can define properties at the multicore & each core level.
> Properties defined in the multicore scope can override system properties.
> Properties defined in a core scope can override multicore & system properties.
> Property definitions can use expressions to define their name & value; these expressions
are evaluated in their outer scope context .
> Multicore serialization keeps properties as written (ie as expressions if they were defined
> The core descriptor properties are automatically defined in each core context, namely:
> solr.core.instanceDir
> solr.core.instancePath
> solr.core.configName
> solr.core.schemaName
> h4. Test:
> The following (contrived) multicore.xml:
> {code:xml}
> <multicore adminPath='/admin/multicore' persistent='true'>
>   <property name='revision'>33</property>  <!-- a basic property -->
>   <property name='zero'>0</property>  <!-- used to expand the core0 name
>   <property name='one'>1</property>  <!-- used to expand the core1 name
>   <property name='id_type'>bogus</property>  <!-- a bogus type that will
be overriden  -->
>   <property name='updateHandler'>bogus</property>  <!-- a bogus updateHandler
that will be overriden  -->
>   <core name='core${zero}' instanceDir='core0/'>    <!-- the name is expanded
>     <property name='id_type'>core${zero}_id</property> <!-- so is a text
node -->
>     <property name='updateHandler'>solr.DirectUpdateHandler2</property> <!--
a property can be overriden -->
>     <property name='revision'>11</property>
>   </core>
>   <core name='core${one}' instanceDir='core1/'>
>     <property name='id_type'>core${one}_id</property>
>     <property name='updateHandler'>solr.DirectUpdateHandler2</property>
>     <property name='revision'>22</property>
>   </core>
> </multicore>
> {code}
> allows this config.xml:
> {code:xml}
> <config>
> <!-- use the defined update handler property -->
>   <updateHandler class="${updateHandler}" />
>   <requestDispatcher handleSelect="true" >
>     <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048"
>   </requestDispatcher>
>   <requestHandler name="standard" class="solr.StandardRequestHandler" default="true"
>   <requestHandler name="/update" class="solr.XmlUpdateRequestHandler" />
>   <requestHandler name="/admin/luke"       class="org.apache.solr.handler.admin.LukeRequestHandler"
>   <!-- config for the admin interface --> 
>   <admin>
>     <defaultQuery>solr</defaultQuery>
>     <gettableFiles>solrconfig.xml schema.xml admin-extra.html</gettableFiles>
>     <pingQuery>
>      qt=standard&amp;q=solrpingquery
>     </pingQuery>
>   </admin>
> </config>
> {code}
> and this schema.xml:
> {code:xml}
> <schema name="example core zero" version="1.1">
>   <types>
>    <!-- define a type name dynamically -->
>     <fieldtype name="${id_type:id_t}"  class="solr.StrField" sortMissingLast="true"
>     <fieldtype name="string"  class="solr.StrField" sortMissingLast="true" omitNorms="true"/>
>   </types>
>  <fields>   
>   <!-- the type of unique key defined above -->
>   <field name="id"      type="${id_type:id_t}"   indexed="true"  stored="true"  multiValued="false"
>   <field name="type"    type="string"   indexed="true"  stored="true"  multiValued="false"
>   <field name="name"    type="string"   indexed="true"  stored="true"  multiValued="false"
>   <field name="${}"   type="string"   indexed="true"  stored="true"
 multiValued="false" /> 
>  </fields>
>  <uniqueKey>id</uniqueKey>
>  <defaultSearchField>name</defaultSearchField>
>  <solrQueryParser defaultOperator="OR"/>
> </schema>
> {code}
> Allow the trunk test to work.
> h3. Coding notes:
> The code itself refactored some of DOMUtil (the ant based property substitution) into
one added class (PropertyMap & PropertyMap.Evaluator).
> The PropertyMap are chained (one link chain between core to multicore map); those maps
are owned by each core's ResourceLoader.
> Config is modified a little to accommodate delaying & specializing property expansions.
> Multicore is modified so it properly parses & serializes.
> Tested against the example above.
> Reviews & comments more than welcome.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message