Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@manifoldcf.apache.org
Received-SPF: pass (nike.apache.org: domain of daddywri@gmail.com designates
 209.85.213.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALUFAGCjmYcQic14qdB=6vXMO8tHdqMCHOV2fX8XTyCEa4L+VA@mail.gmail.com>
References: 
 <CAGwv_LeMhb0u=YKiVk8xF04Vj3L8Tw3mzyX+9boQ17Jq=L3OTg@mail.gmail.com>
	<CALUFAGCjmYcQic14qdB=6vXMO8tHdqMCHOV2fX8XTyCEa4L+VA@mail.gmail.com>
Date: Thu, 8 Jan 2015 11:36:25 -0500
Message-ID: 
 <CALUFAGC3jB1X_up9qgPqK=TsB2TBiZ9NS8HpuG720K9geAwQZw@mail.gmail.com>
Subject: Re: Metadata fields get lost in 1.7.2 with Sharepoint 2013 repository
 and Solr output connection
From: Karl Wright <daddywri@gmail.com>
To: dev <dev@manifoldcf.apache.org>
Cc: "fatih.cetin" <FATIH.CETIN@global-bilgi.com.tr>
Content-Type: multipart/alternative; boundary=001a113a3e1427194d050c26a478

--001a113a3e1427194d050c26a478
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Actually, I take some of this back.  Any SharePoint metadata that is
associated with a parent object rather than a child is represented in
RepositoryDocument as a Reader[] array.  So you should see
RepositoryDocumentFactory iterating through all such fields and making a
TempFileCharacterInput for each member of each field.  If you are seeing
only one iteration of the getFields() iterator, it means that the
RepositoryDocument object fields member is not properly being managed.  But
I'm looking that RepositoryDocument code, and addField() looks like it does
the right thing for all variations of data types.

Karl


On Thu, Jan 8, 2015 at 11:24 AM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Salih,
>
> The code you point at is designed to make copies of fields that are
> represented by Reader objects.  Most SharePoint fields are represented by
> String objects, so this code does not apply to them.
>
> The place you want to look is:
>
> >>>>>>
>     // Copy metadata fields (including minting new Readers where needed)
>     Iterator<String> iter =3D original.getFields();
>     if (iter.hasNext())
>     {
>       String fieldName =3D iter.next();
>       Object[] objects =3D original.getField(fieldName);
>       if (objects instanceof Reader[])
>       {
>         CharacterInput[] rts =3D metadataReaders.get(fieldName);
>         Reader[] newReaders =3D new Reader[rts.length];
>         for (int i =3D 0; i < rts.length; i++)
>         {
>           rts[i].doneWithStream();
>           newReaders[i] =3D rts[i].getStream();
>         }
>         rd.addField(fieldName,newReaders);
>       }
>       else if (objects instanceof Date[])
>       {
>         rd.addField(fieldName,(Date[])objects);
>       }
>       else if (objects instanceof String[])
>       {
>         rd.addField(fieldName,(String[])objects);
>       }
>       else
>         throw new RuntimeException("Unknown kind of metadata:
> "+objects.getClass().getName());
>     }
>
> <<<<<<
>
> This code should copy all fields to the new RepositoryDocument object
> (rd), and do the necessary special manipulation for Reader fields.
>
> If you'd be willing to send me a screen shot of your job (from your view
> job page), I can try to recreate your pipeline here and see what's going =
on.
>
> Thanks,
> Karl
>
>
>
> On Thu, Jan 8, 2015 at 11:13 AM, Salih Sen <salih@dilisim.com> wrote:
>
>> Hi,
>>
>> We've noticed that metadata of some documents aren't indexed in Solr.
>>
>> I tried tracking down to issue in source code and noticed that
>> RepositoryDocument
>> has around 25 fields until it reaches the RepositoryDocumentFactory.
>> =E2=80=8B =E2=80=8B
>> Document that returned from
>> =E2=80=8B =E2=80=8B
>> factory.createDocument()
>> =E2=80=8B =E2=80=8B
>> has only a single field in IncrementalIngester.java line 3089.
>>
>>
>>
>> I couldn't get the logic behind if (iter.hasNext()) in the code below
>> while
>> it has twenty something fields it "iterates" on only the first one.
>> Is is the expected behaviour?
>>
>> A similar code also exist in createDocument() method so I feel I might b=
e
>> looking at the wrong places but as far as I can see this part creates th=
e
>> difference between the document comes from Sharepoint repository and the
>> one posted to Solr.
>>
>> Thanks.
>>
>>
>> RepositoryDocumentFactory.java
>> ---------------------------------=E2=80=8B------------
>>
>> public RepositoryDocumentFactory(RepositoryDocument document)
>>   throws ManifoldCFException, IOException
>> {
>>   this.original =3D document;
>>
>>   try
>>   {
>>     this.binaryTracker =3D new TempFileInput(document.getBinaryStream())=
;
>>     // Copy all reader streams
>>     Iterator<String> iter =3D document.getFields();
>>     if (iter.hasNext())
>>     {
>>       String fieldName =3D iter.next();
>>       Object[] objects =3D document.getField(fieldName);
>>       if (objects instanceof Reader[])
>>       {
>>         CharacterInput[] newValues =3D new CharacterInput[objects.length=
];
>>         metadataReaders.put(fieldName,newValues);
>>         // Populate newValues
>>         for (int i =3D 0; i < newValues.length; i++)
>>         {
>>           newValues[i] =3D new TempFileCharacterInput((Reader)objects[i]=
);
>>         }
>>       }
>>     }
>>   }
>>   catch (Throwable e)
>>   {
>>     // Clean up everything we've done so far.
>>     if (this.binaryTracker !=3D null)
>>       this.binaryTracker.discard();
>>     for (String key : metadataReaders.keySet())
>>     {
>>       CharacterInput[] rt =3D metadataReaders.get(key);
>>       for (CharacterInput r : rt)
>>       {
>>         if (r !=3D null)
>>           r.discard();
>>       }
>>     }
>>     if (e instanceof IOException)
>>       throw (IOException)e;
>>     else if (e instanceof RuntimeException)
>>       throw (RuntimeException)e;
>>     else if (e instanceof Error)
>>       throw (Error)e;
>>     else
>>       throw new RuntimeException("Unknown exception type:
>> "+e.getClass().getName()+": "+e.getMessage(),e);
>>   }
>> }
>>
>>
>>
>> --
>>
>> Salih =C5=9Een
>>
>> Dili=C5=9Fim Bilgi Bilgisayar ve =C4=B0leti=C5=9Fim Teknolojileri Sanayi=
 ve Ticaret Ltd.
>> Sti.
>>
>> email: salih@dilisim.com
>>
>> Tel: 0 222 330 20 21
>>
>> GSM: 0 507 296 15 51
>>
>
>

--001a113a3e1427194d050c26a478--