Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@manifoldcf.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CALUFAGBK9rOqR1mL65m5s-8HgXpq-a71X57dyJXmb38idrxFsA@mail.gmail.com>
References: 
 <CAOug8noTKY4k827qfJd0Vqy4h6tpqZYTQueiSEP76eaa+X-2mA@mail.gmail.com>
	<CALUFAGBK9rOqR1mL65m5s-8HgXpq-a71X57dyJXmb38idrxFsA@mail.gmail.com>
Date: Tue, 11 Aug 2015 10:14:31 +0530
Message-ID: 
 <CAOug8npukty-iNwfSs91tvtD4Rk6=O7FaFz_AVwjqy0wqJRu+Q@mail.gmail.com>
Subject: Re: Indexing Solr documents with atomic updates using manifoldcf solr
 connector
From: Dileepa Jayakody <djayakody@zaizi.com>
To: dev@manifoldcf.apache.org
Content-Type: multipart/alternative; boundary=047d7b86df460abbd9051d01c2db

--047d7b86df460abbd9051d01c2db
Content-Type: text/plain; charset=UTF-8

Hi Karl,

Thanks for your response. My requirement is indexing child documents
constructed from the content repo.document as separate Solr documents. So
adding meta-data fields to the original repository document wouldn't help
my scenario AFAIU.

My transformation connector is somewhat similar to the Stanbol
transformation connector proposed in manifoldcf jira [1].
What I referred as meta-data are the Named Entity Recognition data (NER)
extracted from the repository document. So each content repository document
will have multiple NER child documents. These NERs are expected to be
indexed as separate Solr documents having a mapping to the parent content
repository document which the NERs were extracted from.
So apart from indexing the content repository document in Solr, I need to
index all NER child documents with their attributes as separate documents
in Solr.

Above example is how I create a child repo document for NER. I set the
entire NER document as the binary stream of the child repository document
which is then sent to mcf-solr connector.

In the mcf-solr connector (In HttpPoster class) when building the solr
document from the repository document's input stream, it adds the
inputStream String as a field to the content field of the Solr document
configured by solr-connector as below;

buildSorDocument(long length, InputStream is){

if (contentAttributeName != null)
      {
        Reader r = new InputStreamReader(is, Consts.UTF_8);
        StringBuilder sb = new StringBuilder((int)length);
        char[] buffer = new char[65536];
        while (true)
        {
          int amt = r.read(buffer,0,buffer.length);
          if (amt == -1)
            break;
          sb.append(buffer,0,amt);
        }

        outputDoc.addField( contentAttributeName, sb.toString() );
      }
....
}

Therefore the solr-connector sends the JSON update request I constructed in
my connector as a field value of the  Solr document, not as the whole Solr
document.

Can you please give me some advice on how to index nested child documents
in Solr using Manifold?

Thanks,
Dileepa

[1] https://issues.apache.org/jira/browse/CONNECTORS-1181

On Mon, Aug 10, 2015 at 6:47 PM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Dileepa,
>
> In order for ManifoldCF to index metadata, you need to set metadata field
> values in the RepositoryDocument object, not send Solr JSON as the
> document's content.  In fact from your example it looks like you want zero
> content.
>
> Please read the RepositoryDocument java doc to see how you set metadata.
>
> Karl
>
>
> On Mon, Aug 10, 2015 at 9:05 AM, Dileepa Jayakody <djayakody@zaizi.com>
> wrote:
>
> > Hi All,
> >
> > We have a requirement to extract some meta-data from content documents
> and
> > index those meta-data as separate documents into a Solr index.
> > I'm writing a transformation connector where I construct a new repository
> > document adding the meta-data extracted by the connector and hand it over
> > to mcf-solr-connector to index in Solr.
> > Currently I face some difficulties with indexing these new documents in
> > Solr properly using solr-connector.
> >
> > The new solr document should contain some atomic updates for certain
> > fields. So in my connector I create a JSON to represent the Solr atomic
> > update request and set is as the binaryStream of the repository
> > document.The json string for the new solr document is as below;
> >
> > String jsonString = "[{"id":"http://dbpedia.org/resource/Africa
> > ","label":"Africa","documents":{"add":"sample2.txt"}}]";
> >
> >
> > Then, I add an id and set above jsonString as the binary input stream of
> > the repo-document as follows;
> >
> > repoDoc.addField( "id", idString );
> > InputStream inputStream = IOUtils.toInputStream( jsonString );
> > repoDoc.setBinary(inputStream, jsonString.getBytes().length);
> >
> > The expected behavior is Solr connector sending the SolrInputDocument
> > constructed from the inputStream I added to the repo-document from my
> > connector. But instead it adds the JSON  string to the  'content' field
> of
> > the solr-document and sends to Solr.
> >
> > When I monitored the HTTP request from manifold to Solr I see below;
> >
> > POST /solr/core1/update?wt=xml&version=2.2 HTTP/1.1
> > <add>
> >    <doc boost="1.0">
> >       <field name="id">http://dbpedia.org/resource/Africa</field>
> >       <field name="_root_">[{"id":"http://dbpedia.org/resource/Africa
> > ","label":"Africa","documents":{"add":"sample2.txt"}}]</field>
> >       <field name="lcf_metadata_id">http://dbpedia.org/resource/Africa
> > </field>
> >    </doc></add>0
> >
> > Please note that the 'content' field configured in manifoldcf is
> *_root_*.
> >
> > But the expected Solr update request from solr-connector should be as
> > below;
> > <add>
> >    <doc boost="1.0">
> >     <field name="id">http://dbpedia.org/resource/Africa</field>
> >      <field name="label">Africa</field>
> >       <field name="documents" update="add">sample2.txt</field>
> >      <field name="lcf_metadata_id">http://dbpedia.org/resource/Africa
> > </field>
> >    </doc></add>0
> >
> >
> > Can someone please give some advice on how to use solr atomic updates
> with
> > manifoldcf solr-connector? Have I missed some configurations/arguments?
> >
> > Thanks,
> > Dileepa
> >
> > --
> >
> > ------------------------------
> > This message should be regarded as confidential. If you have received
> this
> > email in error please notify the sender and destroy it immediately.
> > Statements of intent shall only become binding when confirmed in hard
> copy
> > by an authorised signatory.
> >
> > Zaizi Ltd is registered in England and Wales with the registration number
> > 6440931. The Registered Office is Brook House, 229 Shepherds Bush Road,
> > London W6 7AN.
> >
>

-- 

------------------------------
This message should be regarded as confidential. If you have received this 
email in error please notify the sender and destroy it immediately. 
Statements of intent shall only become binding when confirmed in hard copy 
by an authorised signatory.

Zaizi Ltd is registered in England and Wales with the registration number 
6440931. The Registered Office is Brook House, 229 Shepherds Bush Road, 
London W6 7AN. 

--047d7b86df460abbd9051d01c2db--