manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Indexing Solr documents with atomic updates using manifoldcf solr connector
Date Tue, 11 Aug 2015 06:06:44 GMT
Hi Dileepa,

The only current way for ManifoldCF to track documents that are related in
a parent-child relationship is using the "document component" mechanism.
This is appropriate when a repository is structured so that a single
document being processed results in multiple documents being indexed.
Document components are determined and managed by the repository connector,
NOT by a transformation connector or output connector.  Each
RepositoryDocument still represents a single document, never multiple
documents, and even if you could create document components in a
transformation connector, they would be tracked individually in MCF and
indexed completely independently in Solr.

So, to do what you want sounds like it would require a different approach,
specifically the extension of RepositoryDocument to handle multiple
atomically-related logical documents at one time.  This is something that
would require API changes.  However, if such a thing were attempted, the
entire set of related documents represented by a single RepositoryDocument
would all be indexed at one time, atomically, which sounds like it also
might not be what you want.  It sounds to me like you are still trying to
pursue your idea of indexing individual fields independently, is that
correct?

Karl


On Tue, Aug 11, 2015 at 12:44 AM, Dileepa Jayakody <djayakody@zaizi.com>
wrote:

> Hi Karl,
>
> Thanks for your response. My requirement is indexing child documents
> constructed from the content repo.document as separate Solr documents. So
> adding meta-data fields to the original repository document wouldn't help
> my scenario AFAIU.
>
> My transformation connector is somewhat similar to the Stanbol
> transformation connector proposed in manifoldcf jira [1].
> What I referred as meta-data are the Named Entity Recognition data (NER)
> extracted from the repository document. So each content repository document
> will have multiple NER child documents. These NERs are expected to be
> indexed as separate Solr documents having a mapping to the parent content
> repository document which the NERs were extracted from.
> So apart from indexing the content repository document in Solr, I need to
> index all NER child documents with their attributes as separate documents
> in Solr.
>
> Above example is how I create a child repo document for NER. I set the
> entire NER document as the binary stream of the child repository document
> which is then sent to mcf-solr connector.
>
> In the mcf-solr connector (In HttpPoster class) when building the solr
> document from the repository document's input stream, it adds the
> inputStream String as a field to the content field of the Solr document
> configured by solr-connector as below;
>
> buildSorDocument(long length, InputStream is){
>
> if (contentAttributeName != null)
>       {
>         Reader r = new InputStreamReader(is, Consts.UTF_8);
>         StringBuilder sb = new StringBuilder((int)length);
>         char[] buffer = new char[65536];
>         while (true)
>         {
>           int amt = r.read(buffer,0,buffer.length);
>           if (amt == -1)
>             break;
>           sb.append(buffer,0,amt);
>         }
>
>         outputDoc.addField( contentAttributeName, sb.toString() );
>       }
> ....
> }
>
> Therefore the solr-connector sends the JSON update request I constructed in
> my connector as a field value of the  Solr document, not as the whole Solr
> document.
>
> Can you please give me some advice on how to index nested child documents
> in Solr using Manifold?
>
> Thanks,
> Dileepa
>
> [1] https://issues.apache.org/jira/browse/CONNECTORS-1181
>
> On Mon, Aug 10, 2015 at 6:47 PM, Karl Wright <daddywri@gmail.com> wrote:
>
> > Hi Dileepa,
> >
> > In order for ManifoldCF to index metadata, you need to set metadata field
> > values in the RepositoryDocument object, not send Solr JSON as the
> > document's content.  In fact from your example it looks like you want
> zero
> > content.
> >
> > Please read the RepositoryDocument java doc to see how you set metadata.
> >
> > Karl
> >
> >
> > On Mon, Aug 10, 2015 at 9:05 AM, Dileepa Jayakody <djayakody@zaizi.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > We have a requirement to extract some meta-data from content documents
> > and
> > > index those meta-data as separate documents into a Solr index.
> > > I'm writing a transformation connector where I construct a new
> repository
> > > document adding the meta-data extracted by the connector and hand it
> over
> > > to mcf-solr-connector to index in Solr.
> > > Currently I face some difficulties with indexing these new documents in
> > > Solr properly using solr-connector.
> > >
> > > The new solr document should contain some atomic updates for certain
> > > fields. So in my connector I create a JSON to represent the Solr atomic
> > > update request and set is as the binaryStream of the repository
> > > document.The json string for the new solr document is as below;
> > >
> > > String jsonString = "[{"id":"http://dbpedia.org/resource/Africa
> > > ","label":"Africa","documents":{"add":"sample2.txt"}}]";
> > >
> > >
> > > Then, I add an id and set above jsonString as the binary input stream
> of
> > > the repo-document as follows;
> > >
> > > repoDoc.addField( "id", idString );
> > > InputStream inputStream = IOUtils.toInputStream( jsonString );
> > > repoDoc.setBinary(inputStream, jsonString.getBytes().length);
> > >
> > > The expected behavior is Solr connector sending the SolrInputDocument
> > > constructed from the inputStream I added to the repo-document from my
> > > connector. But instead it adds the JSON  string to the  'content' field
> > of
> > > the solr-document and sends to Solr.
> > >
> > > When I monitored the HTTP request from manifold to Solr I see below;
> > >
> > > POST /solr/core1/update?wt=xml&version=2.2 HTTP/1.1
> > > <add>
> > >    <doc boost="1.0">
> > >       <field name="id">http://dbpedia.org/resource/Africa</field>
> > >       <field name="_root_">[{"id":"http://dbpedia.org/resource/Africa
> > > ","label":"Africa","documents":{"add":"sample2.txt"}}]</field>
> > >       <field name="lcf_metadata_id">http://dbpedia.org/resource/Africa
> > > </field>
> > >    </doc></add>0
> > >
> > > Please note that the 'content' field configured in manifoldcf is
> > *_root_*.
> > >
> > > But the expected Solr update request from solr-connector should be as
> > > below;
> > > <add>
> > >    <doc boost="1.0">
> > >     <field name="id">http://dbpedia.org/resource/Africa</field>
> > >      <field name="label">Africa</field>
> > >       <field name="documents" update="add">sample2.txt</field>
> > >      <field name="lcf_metadata_id">http://dbpedia.org/resource/Africa
> > > </field>
> > >    </doc></add>0
> > >
> > >
> > > Can someone please give some advice on how to use solr atomic updates
> > with
> > > manifoldcf solr-connector? Have I missed some configurations/arguments?
> > >
> > > Thanks,
> > > Dileepa
> > >
> > > --
> > >
> > > ------------------------------
> > > This message should be regarded as confidential. If you have received
> > this
> > > email in error please notify the sender and destroy it immediately.
> > > Statements of intent shall only become binding when confirmed in hard
> > copy
> > > by an authorised signatory.
> > >
> > > Zaizi Ltd is registered in England and Wales with the registration
> number
> > > 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> > > London W6 7AN.
> > >
> >
>
> --
>
> ------------------------------
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number
> 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> London W6 7AN.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message