manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Indexing Solr documents with atomic updates using manifoldcf solr connector
Date Tue, 11 Aug 2015 10:07:41 GMT
That is definitely a back-door approach, in that your transformation
connector will work ONLY with the solr output connector, and may indeed
stop working if the details of the solr connector change in the future.
But I will take it to mean that any changes MCF does to its APIs are not
urgent.

Thanks,
Karl


On Tue, Aug 11, 2015 at 3:14 AM, Dileepa Jayakody <djayakody@zaizi.com>
wrote:

> Hi Karl,
>
> Thanks a lot for the detailed explanation.
>
> I was able to get my usecase working by configuring the solr-connector to
> create solrj ContentStreamUpdateRequest where it constructs the solr update
> request using the RepositoryDocumentStream.
>
> I just had to tick "Use the Extract Update Handler" option in
> solr-connector's schema configuration section and use /update/json handler
> to index the content in Solr.
>
> So I could create multiple child repository documents in my transformation
> connector, set the content of it to a JSON and use solrj
> ContentStreamUpdateRequest to send it to Solr and index the child documents
> separately using /update/json handler.
>
> Thank you very much for all the help.
>
> Regards,
> Dileepa
>
>
> On Tue, Aug 11, 2015 at 11:36 AM, Karl Wright <daddywri@gmail.com> wrote:
>
> > Hi Dileepa,
> >
> > The only current way for ManifoldCF to track documents that are related
> in
> > a parent-child relationship is using the "document component" mechanism.
> > This is appropriate when a repository is structured so that a single
> > document being processed results in multiple documents being indexed.
> > Document components are determined and managed by the repository
> connector,
> > NOT by a transformation connector or output connector.  Each
> > RepositoryDocument still represents a single document, never multiple
> > documents, and even if you could create document components in a
> > transformation connector, they would be tracked individually in MCF and
> > indexed completely independently in Solr.
> >
> > So, to do what you want sounds like it would require a different
> approach,
> > specifically the extension of RepositoryDocument to handle multiple
> > atomically-related logical documents at one time.  This is something that
> > would require API changes.  However, if such a thing were attempted, the
> > entire set of related documents represented by a single
> RepositoryDocument
> > would all be indexed at one time, atomically, which sounds like it also
> > might not be what you want.  It sounds to me like you are still trying to
> > pursue your idea of indexing individual fields independently, is that
> > correct?
> >
> > Karl
> >
> >
> > On Tue, Aug 11, 2015 at 12:44 AM, Dileepa Jayakody <djayakody@zaizi.com>
> > wrote:
> >
> > > Hi Karl,
> > >
> > > Thanks for your response. My requirement is indexing child documents
> > > constructed from the content repo.document as separate Solr documents.
> So
> > > adding meta-data fields to the original repository document wouldn't
> help
> > > my scenario AFAIU.
> > >
> > > My transformation connector is somewhat similar to the Stanbol
> > > transformation connector proposed in manifoldcf jira [1].
> > > What I referred as meta-data are the Named Entity Recognition data
> (NER)
> > > extracted from the repository document. So each content repository
> > document
> > > will have multiple NER child documents. These NERs are expected to be
> > > indexed as separate Solr documents having a mapping to the parent
> content
> > > repository document which the NERs were extracted from.
> > > So apart from indexing the content repository document in Solr, I need
> to
> > > index all NER child documents with their attributes as separate
> documents
> > > in Solr.
> > >
> > > Above example is how I create a child repo document for NER. I set the
> > > entire NER document as the binary stream of the child repository
> document
> > > which is then sent to mcf-solr connector.
> > >
> > > In the mcf-solr connector (In HttpPoster class) when building the solr
> > > document from the repository document's input stream, it adds the
> > > inputStream String as a field to the content field of the Solr document
> > > configured by solr-connector as below;
> > >
> > > buildSorDocument(long length, InputStream is){
> > >
> > > if (contentAttributeName != null)
> > >       {
> > >         Reader r = new InputStreamReader(is, Consts.UTF_8);
> > >         StringBuilder sb = new StringBuilder((int)length);
> > >         char[] buffer = new char[65536];
> > >         while (true)
> > >         {
> > >           int amt = r.read(buffer,0,buffer.length);
> > >           if (amt == -1)
> > >             break;
> > >           sb.append(buffer,0,amt);
> > >         }
> > >
> > >         outputDoc.addField( contentAttributeName, sb.toString() );
> > >       }
> > > ....
> > > }
> > >
> > > Therefore the solr-connector sends the JSON update request I
> constructed
> > in
> > > my connector as a field value of the  Solr document, not as the whole
> > Solr
> > > document.
> > >
> > > Can you please give me some advice on how to index nested child
> documents
> > > in Solr using Manifold?
> > >
> > > Thanks,
> > > Dileepa
> > >
> > > [1] https://issues.apache.org/jira/browse/CONNECTORS-1181
> > >
> > > On Mon, Aug 10, 2015 at 6:47 PM, Karl Wright <daddywri@gmail.com>
> wrote:
> > >
> > > > Hi Dileepa,
> > > >
> > > > In order for ManifoldCF to index metadata, you need to set metadata
> > field
> > > > values in the RepositoryDocument object, not send Solr JSON as the
> > > > document's content.  In fact from your example it looks like you want
> > > zero
> > > > content.
> > > >
> > > > Please read the RepositoryDocument java doc to see how you set
> > metadata.
> > > >
> > > > Karl
> > > >
> > > >
> > > > On Mon, Aug 10, 2015 at 9:05 AM, Dileepa Jayakody <
> djayakody@zaizi.com
> > >
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > We have a requirement to extract some meta-data from content
> > documents
> > > > and
> > > > > index those meta-data as separate documents into a Solr index.
> > > > > I'm writing a transformation connector where I construct a new
> > > repository
> > > > > document adding the meta-data extracted by the connector and hand
> it
> > > over
> > > > > to mcf-solr-connector to index in Solr.
> > > > > Currently I face some difficulties with indexing these new
> documents
> > in
> > > > > Solr properly using solr-connector.
> > > > >
> > > > > The new solr document should contain some atomic updates for
> certain
> > > > > fields. So in my connector I create a JSON to represent the Solr
> > atomic
> > > > > update request and set is as the binaryStream of the repository
> > > > > document.The json string for the new solr document is as below;
> > > > >
> > > > > String jsonString = "[{"id":"http://dbpedia.org/resource/Africa
> > > > > ","label":"Africa","documents":{"add":"sample2.txt"}}]";
> > > > >
> > > > >
> > > > > Then, I add an id and set above jsonString as the binary input
> stream
> > > of
> > > > > the repo-document as follows;
> > > > >
> > > > > repoDoc.addField( "id", idString );
> > > > > InputStream inputStream = IOUtils.toInputStream( jsonString );
> > > > > repoDoc.setBinary(inputStream, jsonString.getBytes().length);
> > > > >
> > > > > The expected behavior is Solr connector sending the
> SolrInputDocument
> > > > > constructed from the inputStream I added to the repo-document from
> my
> > > > > connector. But instead it adds the JSON  string to the  'content'
> > field
> > > > of
> > > > > the solr-document and sends to Solr.
> > > > >
> > > > > When I monitored the HTTP request from manifold to Solr I see
> below;
> > > > >
> > > > > POST /solr/core1/update?wt=xml&version=2.2 HTTP/1.1
> > > > > <add>
> > > > >    <doc boost="1.0">
> > > > >       <field name="id">http://dbpedia.org/resource/Africa</field>
> > > > >       <field name="_root_">[{"id":"
> > http://dbpedia.org/resource/Africa
> > > > > ","label":"Africa","documents":{"add":"sample2.txt"}}]</field>
> > > > >       <field name="lcf_metadata_id">
> > http://dbpedia.org/resource/Africa
> > > > > </field>
> > > > >    </doc></add>0
> > > > >
> > > > > Please note that the 'content' field configured in manifoldcf is
> > > > *_root_*.
> > > > >
> > > > > But the expected Solr update request from solr-connector should be
> as
> > > > > below;
> > > > > <add>
> > > > >    <doc boost="1.0">
> > > > >     <field name="id">http://dbpedia.org/resource/Africa</field>
> > > > >      <field name="label">Africa</field>
> > > > >       <field name="documents" update="add">sample2.txt</field>
> > > > >      <field name="lcf_metadata_id">
> > http://dbpedia.org/resource/Africa
> > > > > </field>
> > > > >    </doc></add>0
> > > > >
> > > > >
> > > > > Can someone please give some advice on how to use solr atomic
> updates
> > > > with
> > > > > manifoldcf solr-connector? Have I missed some
> > configurations/arguments?
> > > > >
> > > > > Thanks,
> > > > > Dileepa
> > > > >
> > > > > --
> > > > >
> > > > > ------------------------------
> > > > > This message should be regarded as confidential. If you have
> received
> > > > this
> > > > > email in error please notify the sender and destroy it immediately.
> > > > > Statements of intent shall only become binding when confirmed in
> hard
> > > > copy
> > > > > by an authorised signatory.
> > > > >
> > > > > Zaizi Ltd is registered in England and Wales with the registration
> > > number
> > > > > 6440931. The Registered Office is Brook House, 229 Shepherds Bush
> > Road,
> > > > > London W6 7AN.
> > > > >
> > > >
> > >
> > > --
> > >
> > > ------------------------------
> > > This message should be regarded as confidential. If you have received
> > this
> > > email in error please notify the sender and destroy it immediately.
> > > Statements of intent shall only become binding when confirmed in hard
> > copy
> > > by an authorised signatory.
> > >
> > > Zaizi Ltd is registered in England and Wales with the registration
> number
> > > 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> > > London W6 7AN.
> > >
> >
>
> --
>
> ------------------------------
> This message should be regarded as confidential. If you have received this
> email in error please notify the sender and destroy it immediately.
> Statements of intent shall only become binding when confirmed in hard copy
> by an authorised signatory.
>
> Zaizi Ltd is registered in England and Wales with the registration number
> 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> London W6 7AN.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message