manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Has anyone successfully populated Elastic Search properties from custom SharePoint Meta data fields?
Date Tue, 30 Aug 2016 18:55:22 GMT
Re: "all metadata": there was, I believe, a problem with this that was
fixed in MCF 2.5.  What version of MCF are you using now?

Karl


On Tue, Aug 30, 2016 at 2:54 PM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Mike,
>
> There are TWO versions for the plugin: (1) the version of ES it was
> written for (1.5), and the functional compatibility with MCF itself (that's
> where the 2.1 comes from).
>
> There is no version of the ES plugin that we built for ES 2.x.  That's
> what we're trying to create right now.
>
> So when you see:
>
> apache-manifoldcf-elasticsearch-1.5-plugin-2.2
>
> ... the 1.5 means the ES version and the 2.2 is the actual version of the
> ES-1.5 plugin.
>
> Karl
>
>
> On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
>> So the plugin install fails in Elastic Search 2.1.0 with the same error:
>>
>>
>>
>>
>>
>> C:\APPS\elasticsearch-2.1.0>bin\plugin install
>> file:///C:/Users/holtzm/Downloads/apache-manifoldcf-elastics
>> earch-plugin-2.1-bin.zip
>>
>> -> Installing from file:/C:/Users/holtzm/Download
>> s/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip...
>>
>> Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>> ...
>>
>> Downloading .DONE
>>
>> Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>> checksums if available ...
>>
>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>> .sha1 or .md5 file to verify)
>>
>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
>> plugin zip
>>
>>
>>
>>
>>
>> The documentation says for 2.1 though so I’m confused at this point.
>>
>>
>>
>> I will try adding the ‘plugin-descriptor.properties ‘ file
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>>
>>
>> *From:* Holtz, Michael (IT Consultant)
>> *Sent:* Tuesday, August 30, 2016 1:03 PM
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* RE: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> I think I’ll switch to Elastic Search version 2.1, just to see it work
>> first:
>>
>>
>>
>> *Plugin for Elastic Search (toolkit only)*
>>
>> Latest release (Apache ManifoldCF Plugin for Elastic Search, version 2.1,
>> 2015 Jul 25)
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> BTW I was able to see my first use case work. It turns out that I needed
>> to explicitly include the custom meta data fields, “include all metadata”
>> didn’t work.
>>
>>
>>
>>
>>
>> That leaves me with #4
>>
>>
>>
>> 1.       Index custom meta data fields from SharePoint into the
>> ElasticSearch index *done*
>>
>> 2.       Full text indexing of SharePoint content in ElasticSearch *done*
>>
>> 3.       SharePoint security information indexed in ElasticSearch* done*
>>
>> 4.       ElasticSearch Plugin to enforce SharePoint security settings
>> for search *todo*
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
>> *Sent:* Tuesday, August 30, 2016 12:34 PM
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> We'll probably need to build against ES 2.x, and will need a whole new
>> plugin.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Actually I see a classpath exception when starting elastic search so it
>> doesn’t appear to be that simple.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Tuesday, August 30, 2016 11:55 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Ok, you just included this in the jar and all was OK?  It should be
>> straightforward to set that up if so...
>>
>>
>>
>> Thanks!
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Karl,
>>
>>
>>
>> Ok, I will see what I can do.
>>
>>
>>
>> I just created the attached file for the 1.5 version of the plugin and
>> was able to install it. Let me know if you see any issues with it.
>>
>>
>>
>> I’m going to start with a clean version of the new 2.5 release for my use
>> cases just using the example setup, let me know if you don’t think that is
>> a good idea.
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Tuesday, August 30, 2016 11:17 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> The 2.0 spec has clearly changed from that needed for the 1.5-level
>> plugins.
>>
>>
>>
>> The plugin code is likely to be pretty much unchanged but the packaging
>> will need to change.  Now, I did not develop the 1.5-level plugin, and
>> don't know that much about ES in the first place, so if you want to
>> undertake the plugin transformation and contribute it, we'd be very
>> grateful.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> It looks like a ‘plugin-descriptor.properties’ file needs to be added to
>> the jar:
>>
>>
>>
>> https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/pl
>> ugin-authors.html
>>
>>
>>
>>
>>
>> *From:* Holtz, Michael (IT Consultant)
>> *Sent:* Tuesday, August 30, 2016 10:53 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* RE: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Karl,
>>
>>
>>
>>
>>
>> FYI, Now I’m seeing this:
>>
>>
>>
>>
>>
>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/
>> dist/plugins/elasticsearch/elasticsearch-1.5/target/elastics
>> earch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
>>
>> -> Installing from file:/C:/Users/holtzm/workspac
>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elastics
>> earch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-
>> with-dependencies.jar...
>>
>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/
>> plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch
>> -1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ...
>>
>> Downloading ...........................DONE
>>
>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/
>> plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch
>> -1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if available
>> ...
>>
>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>> .sha1 or .md5 file to verify)
>>
>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
>> plugin zip
>>
>>
>>
>> Mike H.
>>
>>
>>
>>
>>
>> *From:* Holtz, Michael (IT Consultant)
>> *Sent:* Tuesday, August 30, 2016 10:48 AM
>> *To:* user@manifoldcf.apache.org
>> *Subject:* RE: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Karl,
>>
>>
>>
>> I got what your’re saying, I should use the 1.5 plugin. Got it.
>>
>>
>>
>> I will try that
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Holtz, Michael (IT Consultant)
>> *Sent:* Tuesday, August 30, 2016 10:47 AM
>> *To:* user@manifoldcf.apache.org
>> *Subject:* RE: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Hmm that’s confusing, here is the documentation:
>>
>>
>>
>> *Apache ManifoldCF ElasticSearch plugin compatibility*
>>
>> *ManifoldCF versions*
>>
>> *Plugin version*
>>
>> 0.1.x-1.4.x
>>
>> 0.x
>>
>> 1.5.x
>>
>> 1.x
>>
>> >=1.6.x
>>
>> 2.x
>>
>>
>>
>>
>>
>>
>>
>> I’m running version 2.4 of manifold so based on this I went with the 2.1
>> plugin.
>>
>>
>>
>>
>>
>> Where can I get the 1.0 plugin? I don’t see it on the site or in the 2.4
>> build.
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
>> *Sent:* Tuesday, August 30, 2016 10:38 AM
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> You need the ES-1.5 plugin.  That's the older (deprecated) 1.0 plugin.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Elasticsearch 2.3.3
>>
>> Plugin 2.1
>>
>>
>>
>> Here is the current error
>>
>>
>>
>> C:\APPS\elasticsearch-2.3.3>bin\plugin install
>> file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/
>> dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>>
>> -> Installing from file:/C:/Users/holtzm/workspac
>> e/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elastics
>> earch/elasticsearch-plugin-mcf-2.1.jar...
>>
>> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/
>> plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar ...
>>
>> Downloading .DONE
>>
>> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/
>> plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>> checksums if available ...
>>
>> NOTE: Unable to verify checksum for downloaded plugin (unable to find
>> .sha1 or .md5 file to verify)
>>
>> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
>> plugin zip
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Tuesday, August 30, 2016 10:22 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Hi Mike,
>>
>>
>>
>> If you are just trying things out you can use the single-process example
>> and it will make your life easier for experimentation.
>>
>>
>>
>> As far as the ES plugin, there are two: one for ES up to 1.5, and one for
>> 1.5 onward.  Which did you try to install?
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Karl,
>>
>>
>>
>> You are correct.
>>
>>
>>
>> Before I setup zookeeper I would like to prove out a few use cases.
>>
>>
>>
>> 1.       Index custom meta data fields from SharePoint into the
>> ElasticSearch index
>>
>> 2.       Full text indexing of SharePoint content in ElasticSearch
>>
>> 3.       SharePoint security information indexed in ElasticSearch
>>
>> 4.       ElasticSearch Plugin to enforce SharePoint security settings
>> for search
>>
>>
>>
>> At this point I’ve only managed to see #2 and #3 work
>>
>>
>>
>> For #4 I was getting an error when installing the ElasticSearch plugin.
>>
>> I would like to see #1 work before I do anything else. Are there any
>> example configurations that show how to do this?
>>
>>
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Tuesday, August 30, 2016 9:49 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Hi Mike,
>>
>>
>>
>> This sounds like you're having a lock problem.
>>
>>
>>
>> Let me guess: you are using the multiprocess example with file-based
>> locking?  If so, you need to stop everything and do the "lock clean"
>> procedure.  I would also strongly recommend that you switch to the
>> zookeeper multiprocess example.
>>
>>
>>
>> Thanks,
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> I currently cannot view the job. The UI is hanging when I click the
>> “view” link for the job. I’m wondering if that has anything to do with the
>> “Metadata Adjuster” I added?
>>
>>
>>
>> I did not schedule the job.
>>
>>
>>
>> At this point the job has finished running and the Document Status and
>> Queue status reports. I do not see any errors reported there.
>>
>>
>>
>> Also I still do not see my custom meta data fields from SharePoint in the
>> ElasticSearch index.
>>
>>
>>
>> We are hoping to use this in production, but at this point it is not
>> looking too good.
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Tuesday, August 30, 2016 9:09 AM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> It is also important to know whether you have your job set up as
>> "continuous".  If so, it will run forever.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <furkankamaci@gmail.com>
>> wrote:
>>
>> Hi Mike,
>>
>>
>>
>> Could you tell us what "Document Status" and "Queue Status" says as Karl
>> mentioned?
>>
>>
>>
>> Kind Regards,
>>
>> Furkan KAMACI
>>
>>
>>
>> On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> I’m looking at manifoldcf.log and I see no errors.
>>
>>
>>
>> I have only 13 files and they are all very tiny < 15kb .docx files.
>>
>>
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Monday, August 29, 2016 5:26 PM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> You should be looking at the ManifoldCF.log file, and the examples have
>> logging for errors at least as being the default.
>>
>>
>>
>> The only other possibility is that you have a very large document that is
>> taking a long time to transfer/index.
>>
>>
>>
>> Also, you can generally see what MCF is up to using the "document status"
>> and "queue status" reports.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Karl,
>>
>>
>>
>> I see no errors in the log.
>>
>>
>>
>> I probably need to configure the logging I think.
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Monday, August 29, 2016 5:20 PM
>>
>>
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Hi Mike,
>>
>>
>>
>> Have a look in the log.  If there are errors there please send them to
>> us.  Generally, "hangs" are due to MCF retrying documents that have errors.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Karl,
>>
>>
>>
>> Yes, it shows up. I include all meta-data.
>>
>>
>>
>> I’m no trying to use a MetaData adjuster like this
>>
>>
>>
>> Parameter
>>
>> field1                     “${Field 1}”
>>
>>
>>
>> And now I see “field1”=”” in Elastic Search.
>>
>>
>>
>> Also it seems each time I make a change the job will hang and I need to
>> restart everything including ElasticSearch to get it to complete.
>>
>>
>>
>> Thanks,
>>
>> Mike H.
>>
>>
>>
>> *From:* Karl Wright [mailto:daddywri@gmail.com]
>> *Sent:* Monday, August 29, 2016 5:13 PM
>> *To:* user@manifoldcf.apache.org
>> *Subject:* Re: Has anyone successfully populated Elastic Search
>> properties from custom SharePoint Meta data fields?
>>
>>
>>
>> Hi Mike,
>>
>>
>>
>> If I were you I'd try to independently verify that the custom field shows
>> up.  Do you see it appear in the pulldown for selecting metadata?  If not,
>> it probably is not accessible through the inspection methods that give us
>> access into SharePoint fields.
>>
>>
>>
>> If it appears there then we should look at what ES gets sent.  It's
>> possible it appears but under a different field name.
>>
>>
>>
>> Karl
>>
>>
>>
>>
>>
>> On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <
>> MHoltz@consultantemail.com> wrote:
>>
>> Hi,
>>
>>
>>
>> I have created a custom content type with a custom field “Field 1”.
>>
>>
>>
>> I have configured the job to include all meta data.
>>
>>
>>
>> I do not see “Field 1” being populated in the Elastic Search index.
>>
>>
>>
>>
>>
>> Does anyone have any experience with this?
>>
>>
>>
>> Thanks in advance,
>>
>> Mike H
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>

Mime
View raw message