manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Has anyone successfully populated Elastic Search properties from custom SharePoint Meta data fields?
Date Tue, 30 Aug 2016 21:31:58 GMT
I set up an ES-2.0 plugin and pointed it at ES 2.0.2.  Build fails pretty
dramatically.  I'd say that pretty much all the API's changed.  I committed
the code if you want to try for yourself.

You can find the plugin at:

https://svn.apache.org/repos/asf/manifoldcf/integration/elasticsearch-2.0/trunk

Karl


On Tue, Aug 30, 2016 at 3:27 PM, Holtz, Michael (IT Consultant) <
MHoltz@consultantemail.com> wrote:

> Ah got it. I was wondering if it was something like that.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 2:55 PM
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hi Mike,
>
>
>
> There are TWO versions for the plugin: (1) the version of ES it was
> written for (1.5), and the functional compatibility with MCF itself (that's
> where the 2.1 comes from).
>
>
>
> There is no version of the ES plugin that we built for ES 2.x.  That's
> what we're trying to create right now.
>
>
>
> So when you see:
>
>
>
> apache-manifoldcf-elasticsearch-1.5-plugin-2.2
>
>
>
> ... the 1.5 means the ES version and the 2.2 is the actual version of the
> ES-1.5 plugin.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 1:25 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> So the plugin install fails in Elastic Search 2.1.0 with the same error:
>
>
>
>
>
> C:\APPS\elasticsearch-2.1.0>bin\plugin install file:///C:/Users/holtzm/
> Downloads/apache-manifoldcf-elasticsearch-plugin-2.1-bin.zip
>
> -> Installing from file:/C:/Users/holtzm/Downloads/apache-manifoldcf-
> elasticsearch-plugin-2.1-bin.zip...
>
> Trying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-
> elasticsearch-plugin-2.1-bin.zip ...
>
> Downloading .DONE
>
> Verifying file:/C:/Users/holtzm/Downloads/apache-manifoldcf-
> elasticsearch-plugin-2.1-bin.zip checksums if available ...
>
> NOTE: Unable to verify checksum for downloaded plugin (unable to find
> .sha1 or .md5 file to verify)
>
> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
> plugin zip
>
>
>
>
>
> The documentation says for 2.1 though so I’m confused at this point.
>
>
>
> I will try adding the ‘plugin-descriptor.properties ‘ file
>
>
>
>
>
> Thanks,
>
> Mike H.
>
>
>
>
>
> *From:* Holtz, Michael (IT Consultant)
> *Sent:* Tuesday, August 30, 2016 1:03 PM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* RE: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> I think I’ll switch to Elastic Search version 2.1, just to see it work
> first:
>
>
>
> *Plugin for Elastic Search (toolkit only)*
>
> Latest release (Apache ManifoldCF Plugin for Elastic Search, version 2.1,
> 2015 Jul 25)
>
>
>
>
>
>
>
>
>
>
>
> BTW I was able to see my first use case work. It turns out that I needed
> to explicitly include the custom meta data fields, “include all metadata”
> didn’t work.
>
>
>
>
>
> That leaves me with #4
>
>
>
> 1.       Index custom meta data fields from SharePoint into the
> ElasticSearch index *done*
>
> 2.       Full text indexing of SharePoint content in ElasticSearch *done*
>
> 3.       SharePoint security information indexed in ElasticSearch* done*
>
> 4.       ElasticSearch Plugin to enforce SharePoint security settings for
> search *todo*
>
>
>
> Thanks,
>
> Mike H.
>
>
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
> *Sent:* Tuesday, August 30, 2016 12:34 PM
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> We'll probably need to build against ES 2.x, and will need a whole new
> plugin.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 12:29 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Actually I see a classpath exception when starting elastic search so it
> doesn’t appear to be that simple.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 11:55 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Ok, you just included this in the jar and all was OK?  It should be
> straightforward to set that up if so...
>
>
>
> Thanks!
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Karl,
>
>
>
> Ok, I will see what I can do.
>
>
>
> I just created the attached file for the 1.5 version of the plugin and was
> able to install it. Let me know if you see any issues with it.
>
>
>
> I’m going to start with a clean version of the new 2.5 release for my use
> cases just using the example setup, let me know if you don’t think that is
> a good idea.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 11:17 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> The 2.0 spec has clearly changed from that needed for the 1.5-level
> plugins.
>
>
>
> The plugin code is likely to be pretty much unchanged but the packaging
> will need to change.  Now, I did not develop the 1.5-level plugin, and
> don't know that much about ES in the first place, so if you want to
> undertake the plugin transformation and contribute it, we'd be very
> grateful.
>
>
>
> Karl
>
>
>
>
>
>
>
> On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> It looks like a ‘plugin-descriptor.properties’ file needs to be added to
> the jar:
>
>
>
> https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/
> plugin-authors.html
>
>
>
>
>
> *From:* Holtz, Michael (IT Consultant)
> *Sent:* Tuesday, August 30, 2016 10:53 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* RE: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Karl,
>
>
>
>
>
> FYI, Now I’m seeing this:
>
>
>
>
>
> C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/
> workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/
> elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-
> 2.0.1-jar-with-dependencies.jar
>
> -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/
> elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar...
>
> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/
> elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ...
>
> Downloading ...........................DONE
>
> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/
> elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if
> available ...
>
> NOTE: Unable to verify checksum for downloaded plugin (unable to find
> .sha1 or .md5 file to verify)
>
> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
> plugin zip
>
>
>
> Mike H.
>
>
>
>
>
> *From:* Holtz, Michael (IT Consultant)
> *Sent:* Tuesday, August 30, 2016 10:48 AM
> *To:* user@manifoldcf.apache.org
> *Subject:* RE: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Karl,
>
>
>
> I got what your’re saying, I should use the 1.5 plugin. Got it.
>
>
>
> I will try that
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Holtz, Michael (IT Consultant)
> *Sent:* Tuesday, August 30, 2016 10:47 AM
> *To:* user@manifoldcf.apache.org
> *Subject:* RE: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hmm that’s confusing, here is the documentation:
>
>
>
> *Apache ManifoldCF ElasticSearch plugin compatibility*
>
> *ManifoldCF versions*
>
> *Plugin version*
>
> 0.1.x-1.4.x
>
> 0.x
>
> 1.5.x
>
> 1.x
>
> >=1.6.x
>
> 2.x
>
>
>
>
>
>
>
> I’m running version 2.4 of manifold so based on this I went with the 2.1
> plugin.
>
>
>
>
>
> Where can I get the 1.0 plugin? I don’t see it on the site or in the 2.4
> build.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com <daddywri@gmail.com>]
> *Sent:* Tuesday, August 30, 2016 10:38 AM
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> You need the ES-1.5 plugin.  That's the older (deprecated) 1.0 plugin.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Elasticsearch 2.3.3
>
> Plugin 2.1
>
>
>
> Here is the current error
>
>
>
> C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/
> workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/
> elasticsearch/elasticsearch-plugin-mcf-2.1.jar
>
> -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-
> plugin-mcf-2.1.jar...
>
> Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
> ...
>
> Downloading .DONE
>
> Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.
> 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
> checksums if available ...
>
> NOTE: Unable to verify checksum for downloaded plugin (unable to find
> .sha1 or .md5 file to verify)
>
> ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in
> plugin zip
>
>
>
>
>
>
>
>
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 10:22 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hi Mike,
>
>
>
> If you are just trying things out you can use the single-process example
> and it will make your life easier for experimentation.
>
>
>
> As far as the ES plugin, there are two: one for ES up to 1.5, and one for
> 1.5 onward.  Which did you try to install?
>
>
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Karl,
>
>
>
> You are correct.
>
>
>
> Before I setup zookeeper I would like to prove out a few use cases.
>
>
>
> 1.       Index custom meta data fields from SharePoint into the
> ElasticSearch index
>
> 2.       Full text indexing of SharePoint content in ElasticSearch
>
> 3.       SharePoint security information indexed in ElasticSearch
>
> 4.       ElasticSearch Plugin to enforce SharePoint security settings for
> search
>
>
>
> At this point I’ve only managed to see #2 and #3 work
>
>
>
> For #4 I was getting an error when installing the ElasticSearch plugin.
>
> I would like to see #1 work before I do anything else. Are there any
> example configurations that show how to do this?
>
>
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 9:49 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hi Mike,
>
>
>
> This sounds like you're having a lock problem.
>
>
>
> Let me guess: you are using the multiprocess example with file-based
> locking?  If so, you need to stop everything and do the "lock clean"
> procedure.  I would also strongly recommend that you switch to the
> zookeeper multiprocess example.
>
>
>
> Thanks,
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> I currently cannot view the job. The UI is hanging when I click the “view”
> link for the job. I’m wondering if that has anything to do with the
> “Metadata Adjuster” I added?
>
>
>
> I did not schedule the job.
>
>
>
> At this point the job has finished running and the Document Status and
> Queue status reports. I do not see any errors reported there.
>
>
>
> Also I still do not see my custom meta data fields from SharePoint in the
> ElasticSearch index.
>
>
>
> We are hoping to use this in production, but at this point it is not
> looking too good.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Tuesday, August 30, 2016 9:09 AM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> It is also important to know whether you have your job set up as
> "continuous".  If so, it will run forever.
>
>
>
> Karl
>
>
>
>
>
> On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <furkankamaci@gmail.com>
> wrote:
>
> Hi Mike,
>
>
>
> Could you tell us what "Document Status" and "Queue Status" says as Karl
> mentioned?
>
>
>
> Kind Regards,
>
> Furkan KAMACI
>
>
>
> On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> I’m looking at manifoldcf.log and I see no errors.
>
>
>
> I have only 13 files and they are all very tiny < 15kb .docx files.
>
>
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Monday, August 29, 2016 5:26 PM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> You should be looking at the ManifoldCF.log file, and the examples have
> logging for errors at least as being the default.
>
>
>
> The only other possibility is that you have a very large document that is
> taking a long time to transfer/index.
>
>
>
> Also, you can generally see what MCF is up to using the "document status"
> and "queue status" reports.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Karl,
>
>
>
> I see no errors in the log.
>
>
>
> I probably need to configure the logging I think.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Monday, August 29, 2016 5:20 PM
>
>
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hi Mike,
>
>
>
> Have a look in the log.  If there are errors there please send them to
> us.  Generally, "hangs" are due to MCF retrying documents that have errors.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Karl,
>
>
>
> Yes, it shows up. I include all meta-data.
>
>
>
> I’m no trying to use a MetaData adjuster like this
>
>
>
> Parameter
>
> field1                     “${Field 1}”
>
>
>
> And now I see “field1”=”” in Elastic Search.
>
>
>
> Also it seems each time I make a change the job will hang and I need to
> restart everything including ElasticSearch to get it to complete.
>
>
>
> Thanks,
>
> Mike H.
>
>
>
> *From:* Karl Wright [mailto:daddywri@gmail.com]
> *Sent:* Monday, August 29, 2016 5:13 PM
> *To:* user@manifoldcf.apache.org
> *Subject:* Re: Has anyone successfully populated Elastic Search
> properties from custom SharePoint Meta data fields?
>
>
>
> Hi Mike,
>
>
>
> If I were you I'd try to independently verify that the custom field shows
> up.  Do you see it appear in the pulldown for selecting metadata?  If not,
> it probably is not accessible through the inspection methods that give us
> access into SharePoint fields.
>
>
>
> If it appears there then we should look at what ES gets sent.  It's
> possible it appears but under a different field name.
>
>
>
> Karl
>
>
>
>
>
> On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <
> MHoltz@consultantemail.com> wrote:
>
> Hi,
>
>
>
> I have created a custom content type with a custom field “Field 1”.
>
>
>
> I have configured the job to include all meta data.
>
>
>
> I do not see “Field 1” being populated in the Elastic Search index.
>
>
>
>
>
> Does anyone have any experience with this?
>
>
>
> Thanks in advance,
>
> Mike H
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Mime
View raw message