lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julio Castillo" <jcasti...@edgenuity.com>
Subject Re: How to describe 2 entities in dataConfig for the DataImporter?
Date Mon, 02 Jun 2008 21:58:32 GMT
Shalin,
I experimented with it, and the null pointer exception has been taken care
of. Thank you.

I have a different problem now. I believe it is a syntax/specification
problem.

When importing data, I got the following exceptions:
SEVERE: Exception while adding:
SolrInputDocumnt[{comboId=comboId(1.0)={owners-9},
userName=userName(1.0)={[David, Schroeder]}}]

org.apache.solr.common.SolrException: Document [null] missing required
field: id
	at
org.apache.solr.update.DocumentBuilder.toDocument(DocumentBuilder.java:289)
	at
org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHand
ler.java:263)
	...

The problem arises the moment I try to include nested entities (e.g. pets
-the problem does not occur if I don't use the transformer, but I have to
use the transformer because other unrelated entities also have id's).
My data config file looks as follows.

<dataConfig>
  <document name="doc-1">
    <entity name="owners" pk="id"
	    query="select id,first_name,last_name FROM owners"
	    transformer="TemplateTransformer">
	<field column="id"  name="comboId"  template="owners-${owners.id}"/>
	<field column="first_name" name="userName"/>
	<field column="last_name"  name="userName"/>

	<entity name="pets" pk="id"
		query="SELECT id,name,birth_date,type_id FROM pets WHERE
owner_id='${owners.id}'"
		parentDeltaQuery="SELECT id FROM owners WHERE
id=${pets.owner_id}">
	    <field column="id"		name="id"/>
	    <field column="name"	name="name"/>
	    <field column="birth_date" name="birthDate"/>
	</entity>
    </entity>
  </document>
</dataConfig>

The debug output of the data import looks as follows:

....
- <lst name="verbose-output">
  - <lst name="entity:owners">
    - <lst name="document#1">
      <str name="query">select id,first_name,last_name FROM owners</str> 
      <str name="time-taken">0:0:0.15</str> 
      <str>----------- row #1-------------</str> 
      <int name="id">1</int> 
      <str name="first_name">George</str> 
      <str name="last_name">Franklin</str> 
      <str>---------------------------------------------</str> 
      - <lst name="transformer:TemplateTransformer">
        <str>---------------------------------------------</str> 
        <str name="id">owners-1</str> 
        <str name="first_name">George</str> 
        <str name="last_name">Franklin</str> 
        <str>---------------------------------------------</str> 
        - <lst name="entity:pets">
          <str name="query">SELECT id,name,birth_date,type_id FROM pets
WHERE owner_id='owners-1'</str> 
          <str name="time-taken">0:0:0.0</str> 
          </lst>
      </lst>
  </lst>
+ <lst name="document#1">
....

Thanks again

** julio


-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com] 
Sent: Saturday, May 31, 2008 10:26 AM
To: solr-user@lucene.apache.org
Subject: Re: How to describe 2 entities in dataConfig for the DataImporter?

Hi Julio,

I've fixed the bug, can you please replace the exiting
TemplateTransformer.java in the SOLR-469.patch and use the attached
TemplateTransformer.java file. We'll add the changes to our next patch.
Sorry for all the trouble.

On Sat, May 31, 2008 at 10:31 PM, Noble Paul ??????? ??????
<noble.paul@gmail.com> wrote:
> julio,
> Looks like it is a bug.
> We can give u a new TemplateTransformer.java which we will incorporate 
> in the next patch --Noble
>
> On Sat, May 31, 2008 at 12:24 AM, Julio Castillo 
> <jcastillo@edgenuity.com> wrote:
>> I'm sorry Shalin, but I still get the same Null Pointer exception. 
>> This is my complete dataconfig.xml (I remove the parallel entity to 
>> narrow down the scope of the problem).
>> <dataConfig>
>>  <document name="doc-1">
>>    <entity name="vets" pk="id"
>>            query="select id as idAlias,first_name,last_name FROM vets"
>>            deltaQuery="SELECT id as idAlias FROM vets WHERE 
>> last_modified > '${dataimporter.last_index_time}'"
>>            transformer="TemplateTransformer">
>>        <field column="id"              name="id"
>> template="vets-${vets.idAlias}"/>
>>        <field column="first_name"      name="userName"/>
>>        <field column="last_name"       name="userName"/>
>>    </entity>
>>  </document>
>> </dataConfig>
>>
>> Thanks again.
>>
>> ** julio
>>
>> -----Original Message-----
>> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
>> Sent: Friday, May 30, 2008 11:38 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: How to describe 2 entities in dataConfig for the
DataImporter?
>>
>> The surname is used just as an example of a field.
>>
>> The NullPointerException is because the same field "id" tries to use 
>> it's own value in a template. The template cannot contain the same 
>> field on which it is being applied. I'd suggest that you get the id 
>> aliased to another name, for example using a query "select id as 
>> idAlias from vets" and then
>> use:
>> <field column="id" template="vets-${vets.idAlias}" />
>>
>> That should work, let me know if you face a problem.
>>
>> On Fri, May 30, 2008 at 10:40 PM, Julio Castillo 
>> <jcastillo@edgenuity.com>
>> wrote:
>>> Thanks for all the leads.
>>> I did get however a null pointer exception while implementing it:
>>>
>>> May 30, 2008 9:57:50 AM
>>> org.apache.solr.handler.dataimport.EntityProcessorBase
>>> applyTransformer
>>> WARNING: transformer threw error
>>> java.lang.NullPointerException
>>>   at
>>> org.apache.solr.handler.dataimport.TemplateTransformer.transformRow(
>>> Te
>>> mplate
>>> Transformer.java:55)
>>>   at
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransfor
>>> me
>>> r(Enti
>>> tyProcessorBase.java:186)
>>>
>>> Looking at the source code, it appears that the resolverMap is null.
>>> The resolver returned null given the entityName.
>>>
>>> Looking at the documentation, there is the reference to a
eparent.surname.
>>> The example says:
>>> <field column="namedesc" template="hello${e.name},${eparent.surname}"
>>> />
>>>
>>> I'm afraid, I don't know what an eparent.surname is. This is my 
>>> current dataconfig.xml configuration excerpt:
>>>
>>> <entity name="vets" pk="id"  query="..." deltaQuery="..."
>>> transformer="TemplateTransformer">
>>>  <field column="id" name="id" template="vets-${vets.id}"/>  ...
>>>
>>> Am I missing a surname? Whatever that may be?
>>>
>>> Thanks
>>>
>>> ** julio
>>>
>>> -----Original Message-----
>>> From: Noble Paul ??????? ?????? [mailto:noble.paul@gmail.com]
>>> Sent: Thursday, May 29, 2008 11:10 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: How to describe 2 entities in dataConfig for the
>> DataImporter?
>>>
>>> Sorry I forgot to mention that.
>>> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a9
>>> 90
>>> fb03c4
>>> ff461b3736496a9
>>> --Noble
>>>
>>> On Fri, May 30, 2008 at 11:37 AM, Shalin Shekhar Mangar 
>>> <shalinmangar@gmail.com> wrote:
>>>> You need to enable TemplateTransformer for your entity. For example:
>>>> <entity name="owners" pk="id" query="...."
>>>> transformer="TemplateTransformer">
>>>>
>>>> On Fri, May 30, 2008 at 11:31 AM, Julio Castillo 
>>>> <jcastillo@edgenuity.com> wrote:
>>>>> Noble,
>>>>> I tried the template setting for the "id" field, but I didn't 
>>>>> notice any different behavior. I also didn't see where this would be
reflected.
>>>>> I looked at the fields and the debug output for the dataImporter 
>>>>> and couldn't see any reference to a modified id name (per the 
>>>>> template instructions).
>>>>>
>>>>> The behavior in the end seemed to be the same. Did I miss anything?
>>>>> I assume that the <uniqueKey>id</uniqueKey> setting in the

>>>>> schema.xml remains the same?
>>>>>
>>>>> Thanks again
>>>>>
>>>>> ** julio
>>>>>
>>>>> -----Original Message-----
>>>>> From: Noble Paul ??????? ?????? [mailto:noble.paul@gmail.com]
>>>>> Sent: Thursday, May 29, 2008 9:46 PM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>> DataImporter?
>>>>>
>>>>> Consider constructing the id concatenating an extra string for 
>>>>> each document . You can construct that field using the
TeplateTransformer.
>>>>> in the entity owners keep the id as
>>>>>
>>>>> <field column="id" name="id" template="owners-${owners.id}"/> and

>>>>> in vets <field column="id" name="id" template="vets-${vets.id}"/>
>>>>>
>>>>> or anything else which can make it unique
>>>>>
>>>>> --Noble
>>>>>
>>>>> On Fri, May 30, 2008 at 10:05 AM, Shalin Shekhar Mangar 
>>>>> <shalinmangar@gmail.com> wrote:
>>>>>> That will happen only if id is the uniqueKey in Solr and the id 
>>>>>> coming from both your tables have same values. In that case, they

>>>>>> will overwrite each other. You will need a separate uniqueKey (on

>>>>>> other than id field).
>>>>>>
>>>>>> On Fri, May 30, 2008 at 6:34 AM, Julio Castillo 
>>>>>> <jcastillo@edgenuity.com>
>>>>> wrote:
>>>>>>> Thanks Shalin,
>>>>>>> I tried putting everything under the same document (two 
>>>>>>> different unrelated entities), and got a bit further.
>>>>>>>
>>>>>>> My problem now appears to be both of them stepping on each other

>>>>>>> due to
>>>>> "id"
>>>>>>> conflicts. Currently my id is defined in my schema as <field

>>>>>>> name="id" type="long" indexed="true" stored="true"
>>>>>>> required="true"/>
>>>>>>>
>>>>>>> Do I have to create a new "id" field?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> ** julio
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
>>>>>>> Sent: Thursday, May 29, 2008 11:40 AM
>>>>>>> To: solr-user@lucene.apache.org
>>>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>>>> DataImporter?
>>>>>>>
>>>>>>> Hi Julio,
>>>>>>>
>>>>>>> The first data-config is correct.
>>>>>>>
>>>>>>> You're running DataImportHandler in debug mode which creates

>>>>>>> only the first 10 documents by default. You can also add count=N

>>>>>>> to index only the first N documents. But this is intended only

>>>>>>> for debugging purposes. If you want to do a full-import just
use 
>>>>>>> /dataimport?command=full-import. By default, a full-import 
>>>>>>> automatically
>>>>> cleans and commits.
>>>>>>>
>>>>>>> Hope that helps.
>>>>>>>
>>>>>>> On Thu, May 29, 2008 at 11:42 PM, Julio Castillo 
>>>>>>> <jcastillo@edgenuity.com>
>>>>>>> wrote:
>>>>>>>> I have 2 dB tables unrelated to each other that I want to
index.
>>>>>>>>
>>>>>>>> I have tried 2 approaches for specifying them in my 
>>>>>>>> data-config.xml
>>>>> file.
>>>>>>>> None of them seem to work (it seems I can only get data for
the 
>>>>>>>> first one listed).
>>>>>>>>
>>>>>>>> CASE 1)
>>>>>>>> <dataConfig>
>>>>>>>>  <document name="test">
>>>>>>>>    <entity name="owners" pk="id" query="....">
>>>>>>>>      <field column="id" name="id"/>
>>>>>>>>      <field column=name" name="userName"/>
>>>>>>>>    </entity>
>>>>>>>>    <entity name="vets" pk="id" query="....">
>>>>>>>>      <field column="id" name="id"/>
>>>>>>>>      <field column=name" name="userName"/>
>>>>>>>>    </entity>
>>>>>>>>  </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> CASE 2)
>>>>>>>> <dataConfig>
>>>>>>>>  <document name="test-1">
>>>>>>>>    <entity name="owners" pk="id" query="....">
>>>>>>>>      <field column="id" name="id"/>
>>>>>>>>      <field column=name" name="userName"/>
>>>>>>>>    </entity>
>>>>>>>>  </document>
>>>>>>>>  <document name="test-2">
>>>>>>>>    <entity name="vets" pk="id" query="....">
>>>>>>>>      <field column="id" name="id"/>
>>>>>>>>      <field column=name" name="userName"/>
>>>>>>>>    </entity>
>>>>>>>>  </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> I ran the dataImporter as follows 
>>>>>>>> http://localhost:2455/solr/dataimport?command=full-import&debug
>>>>>>>> =o
>>>>>>>> n
>>>>>>>> &v
>>>>>>>> er
>>>>>>>> bose=t
>>>>>>>> rue&clean=true&commit=true
>>>>>>>>
>>>>>>>> After running Case 1 the output only shows documents under
>>>>> entity:owners.
>>>>>>>> And an empty reference to entity:vets (no data).
>>>>>>>>
>>>>>>>> After running Case 2, the output only shows documents under
>>>>> entity:owners.
>>>>>>>> No references to entity:vets.
>>>>>>>>
>>>>>>>> In either case, I have commented out the entity:owners and
I do 
>>>>>>>> see the entity:vets data then.
>>>>>>>>
>>>>>>>> Thanks for your assistance
>>>>>>>>
>>>>>>>> Julio Castillo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
>
> --
> --Noble Paul
>



--
Regards,
Shalin Shekhar Mangar.


Mime
View raw message