Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@manifoldcf.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALUFAGBVUppTkwbzLyAzEaPPELeKCuzVWNfGnStyt5j3NC0Bzg@mail.gmail.com>
References: 
 <CAOug8noTKY4k827qfJd0Vqy4h6tpqZYTQueiSEP76eaa+X-2mA@mail.gmail.com>
	<CALUFAGBK9rOqR1mL65m5s-8HgXpq-a71X57dyJXmb38idrxFsA@mail.gmail.com>
	<CAOug8npukty-iNwfSs91tvtD4Rk6=O7FaFz_AVwjqy0wqJRu+Q@mail.gmail.com>
	<CALUFAGBVUppTkwbzLyAzEaPPELeKCuzVWNfGnStyt5j3NC0Bzg@mail.gmail.com>
Date: Tue, 11 Aug 2015 12:44:55 +0530
Message-ID: 
 <CAOug8noS6w3P_vNR+fp_5Mi7nW_yx=P5smt+x228wjwyKss3eg@mail.gmail.com>
Subject: Re: Indexing Solr documents with atomic updates using manifoldcf solr
 connector
From: Dileepa Jayakody <djayakody@zaizi.com>
To: dev@manifoldcf.apache.org
Content-Type: multipart/alternative; boundary=f46d04428132f0e742051d03db8a

--f46d04428132f0e742051d03db8a
Content-Type: text/plain; charset=UTF-8

Hi Karl,

Thanks a lot for the detailed explanation.

I was able to get my usecase working by configuring the solr-connector to
create solrj ContentStreamUpdateRequest where it constructs the solr update
request using the RepositoryDocumentStream.

I just had to tick "Use the Extract Update Handler" option in
solr-connector's schema configuration section and use /update/json handler
to index the content in Solr.

So I could create multiple child repository documents in my transformation
connector, set the content of it to a JSON and use solrj
ContentStreamUpdateRequest to send it to Solr and index the child documents
separately using /update/json handler.

Thank you very much for all the help.

Regards,
Dileepa


On Tue, Aug 11, 2015 at 11:36 AM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Dileepa,
>
> The only current way for ManifoldCF to track documents that are related in
> a parent-child relationship is using the "document component" mechanism.
> This is appropriate when a repository is structured so that a single
> document being processed results in multiple documents being indexed.
> Document components are determined and managed by the repository connector,
> NOT by a transformation connector or output connector.  Each
> RepositoryDocument still represents a single document, never multiple
> documents, and even if you could create document components in a
> transformation connector, they would be tracked individually in MCF and
> indexed completely independently in Solr.
>
> So, to do what you want sounds like it would require a different approach,
> specifically the extension of RepositoryDocument to handle multiple
> atomically-related logical documents at one time.  This is something that
> would require API changes.  However, if such a thing were attempted, the
> entire set of related documents represented by a single RepositoryDocument
> would all be indexed at one time, atomically, which sounds like it also
> might not be what you want.  It sounds to me like you are still trying to
> pursue your idea of indexing individual fields independently, is that
> correct?
>
> Karl
>
>
> On Tue, Aug 11, 2015 at 12:44 AM, Dileepa Jayakody <djayakody@zaizi.com>
> wrote:
>
> > Hi Karl,
> >
> > Thanks for your response. My requirement is indexing child documents
> > constructed from the content repo.document as separate Solr documents. So
> > adding meta-data fields to the original repository document wouldn't help
> > my scenario AFAIU.
> >
> > My transformation connector is somewhat similar to the Stanbol
> > transformation connector proposed in manifoldcf jira [1].
> > What I referred as meta-data are the Named Entity Recognition data (NER)
> > extracted from the repository document. So each content repository
> document
> > will have multiple NER child documents. These NERs are expected to be
> > indexed as separate Solr documents having a mapping to the parent content
> > repository document which the NERs were extracted from.
> > So apart from indexing the content repository document in Solr, I need to
> > index all NER child documents with their attributes as separate documents
> > in Solr.
> >
> > Above example is how I create a child repo document for NER. I set the
> > entire NER document as the binary stream of the child repository document
> > which is then sent to mcf-solr connector.
> >
> > In the mcf-solr connector (In HttpPoster class) when building the solr
> > document from the repository document's input stream, it adds the
> > inputStream String as a field to the content field of the Solr document
> > configured by solr-connector as below;
> >
> > buildSorDocument(long length, InputStream is){
> >
> > if (contentAttributeName != null)
> >       {
> >         Reader r = new InputStreamReader(is, Consts.UTF_8);
> >         StringBuilder sb = new StringBuilder((int)length);
> >         char[] buffer = new char[65536];
> >         while (true)
> >         {
> >           int amt = r.read(buffer,0,buffer.length);
> >           if (amt == -1)
> >             break;
> >           sb.append(buffer,0,amt);
> >         }
> >
> >         outputDoc.addField( contentAttributeName, sb.toString() );
> >       }
> > ....
> > }
> >
> > Therefore the solr-connector sends the JSON update request I constructed
> in
> > my connector as a field value of the  Solr document, not as the whole
> Solr
> > document.
> >
> > Can you please give me some advice on how to index nested child documents
> > in Solr using Manifold?
> >
> > Thanks,
> > Dileepa
> >
> > [1] https://issues.apache.org/jira/browse/CONNECTORS-1181
> >
> > On Mon, Aug 10, 2015 at 6:47 PM, Karl Wright <daddywri@gmail.com> wrote:
> >
> > > Hi Dileepa,
> > >
> > > In order for ManifoldCF to index metadata, you need to set metadata
> field
> > > values in the RepositoryDocument object, not send Solr JSON as the
> > > document's content.  In fact from your example it looks like you want
> > zero
> > > content.
> > >
> > > Please read the RepositoryDocument java doc to see how you set
> metadata.
> > >
> > > Karl
> > >
> > >
> > > On Mon, Aug 10, 2015 at 9:05 AM, Dileepa Jayakody <djayakody@zaizi.com
> >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > We have a requirement to extract some meta-data from content
> documents
> > > and
> > > > index those meta-data as separate documents into a Solr index.
> > > > I'm writing a transformation connector where I construct a new
> > repository
> > > > document adding the meta-data extracted by the connector and hand it
> > over
> > > > to mcf-solr-connector to index in Solr.
> > > > Currently I face some difficulties with indexing these new documents
> in
> > > > Solr properly using solr-connector.
> > > >
> > > > The new solr document should contain some atomic updates for certain
> > > > fields. So in my connector I create a JSON to represent the Solr
> atomic
> > > > update request and set is as the binaryStream of the repository
> > > > document.The json string for the new solr document is as below;
> > > >
> > > > String jsonString = "[{"id":"http://dbpedia.org/resource/Africa
> > > > ","label":"Africa","documents":{"add":"sample2.txt"}}]";
> > > >
> > > >
> > > > Then, I add an id and set above jsonString as the binary input stream
> > of
> > > > the repo-document as follows;
> > > >
> > > > repoDoc.addField( "id", idString );
> > > > InputStream inputStream = IOUtils.toInputStream( jsonString );
> > > > repoDoc.setBinary(inputStream, jsonString.getBytes().length);
> > > >
> > > > The expected behavior is Solr connector sending the SolrInputDocument
> > > > constructed from the inputStream I added to the repo-document from my
> > > > connector. But instead it adds the JSON  string to the  'content'
> field
> > > of
> > > > the solr-document and sends to Solr.
> > > >
> > > > When I monitored the HTTP request from manifold to Solr I see below;
> > > >
> > > > POST /solr/core1/update?wt=xml&version=2.2 HTTP/1.1
> > > > <add>
> > > >    <doc boost="1.0">
> > > >       <field name="id">http://dbpedia.org/resource/Africa</field>
> > > >       <field name="_root_">[{"id":"
> http://dbpedia.org/resource/Africa
> > > > ","label":"Africa","documents":{"add":"sample2.txt"}}]</field>
> > > >       <field name="lcf_metadata_id">
> http://dbpedia.org/resource/Africa
> > > > </field>
> > > >    </doc></add>0
> > > >
> > > > Please note that the 'content' field configured in manifoldcf is
> > > *_root_*.
> > > >
> > > > But the expected Solr update request from solr-connector should be as
> > > > below;
> > > > <add>
> > > >    <doc boost="1.0">
> > > >     <field name="id">http://dbpedia.org/resource/Africa</field>
> > > >      <field name="label">Africa</field>
> > > >       <field name="documents" update="add">sample2.txt</field>
> > > >      <field name="lcf_metadata_id">
> http://dbpedia.org/resource/Africa
> > > > </field>
> > > >    </doc></add>0
> > > >
> > > >
> > > > Can someone please give some advice on how to use solr atomic updates
> > > with
> > > > manifoldcf solr-connector? Have I missed some
> configurations/arguments?
> > > >
> > > > Thanks,
> > > > Dileepa
> > > >
> > > > --
> > > >
> > > > ------------------------------
> > > > This message should be regarded as confidential. If you have received
> > > this
> > > > email in error please notify the sender and destroy it immediately.
> > > > Statements of intent shall only become binding when confirmed in hard
> > > copy
> > > > by an authorised signatory.
> > > >
> > > > Zaizi Ltd is registered in England and Wales with the registration
> > number
> > > > 6440931. The Registered Office is Brook House, 229 Shepherds Bush
> Road,
> > > > London W6 7AN.
> > > >
> > >
> >
> > --
> >
> > ------------------------------
> > This message should be regarded as confidential. If you have received
> this
> > email in error please notify the sender and destroy it immediately.
> > Statements of intent shall only become binding when confirmed in hard
> copy
> > by an authorised signatory.
> >
> > Zaizi Ltd is registered in England and Wales with the registration number
> > 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> > London W6 7AN.
> >
>

-- 

------------------------------
This message should be regarded as confidential. If you have received this 
email in error please notify the sender and destroy it immediately. 
Statements of intent shall only become binding when confirmed in hard copy 
by an authorised signatory.

Zaizi Ltd is registered in England and Wales with the registration number 
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road, 
London W6 7AN. 

--f46d04428132f0e742051d03db8a--