manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Holtz, Michael (IT Consultant)" <MHo...@consultantemail.com>
Subject RE: Has anyone successfully populated Elastic Search properties from custom SharePoint Meta data fields?
Date Tue, 30 Aug 2016 15:27:37 GMT
Karl,

Ok, I will see what I can do.

I just created the attached file for the 1.5 version of the plugin and was able to install
it. Let me know if you see any issues with it.

I’m going to start with a clean version of the new 2.5 release for my use cases just using
the example setup, let me know if you don’t think that is a good idea.

Thanks,
Mike H.


From: Karl Wright [mailto:daddywri@gmail.com]
Sent: Tuesday, August 30, 2016 11:17 AM
To: user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

The 2.0 spec has clearly changed from that needed for the 1.5-level plugins.

The plugin code is likely to be pretty much unchanged but the packaging will need to change.
 Now, I did not develop the 1.5-level plugin, and don't know that much about ES in the first
place, so if you want to undertake the plugin transformation and contribute it, we'd be very
grateful.

Karl



On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
It looks like a ‘plugin-descriptor.properties’ file needs to be added to the jar:

https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugin-authors.html


From: Holtz, Michael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:53 AM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: RE: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Karl,


FYI, Now I’m seeing this:


C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar<file:///C:\Users\holtzm\workspace\apache-manifoldcf-2.4\dist\plugins\elasticsearch\elasticsearch-1.5\target\elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar>
-> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar...
Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
...
Downloading ...........................DONE
Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar
checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to
verify)
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip

Mike H.


From: Holtz, Michael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:48 AM
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: RE: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Karl,

I got what your’re saying, I should use the 1.5 plugin. Got it.

I will try that

Thanks,
Mike H.

From: Holtz, Michael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:47 AM
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: RE: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Hmm that’s confusing, here is the documentation:

Apache ManifoldCF ElasticSearch plugin compatibility

ManifoldCF versions

Plugin version

0.1.x-1.4.x

0.x

1.5.x

1.x

>=1.6.x

2.x




I’m running version 2.4 of manifold so based on this I went with the 2.1 plugin.


Where can I get the 1.0 plugin? I don’t see it on the site or in the 2.4 build.

Thanks,
Mike H.

From: Karl Wright [mailto:daddywri@gmail.com]
Sent: Tuesday, August 30, 2016 10:38 AM
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

You need the ES-1.5 plugin.  That's the older (deprecated) 1.0 plugin.

Karl


On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
Elasticsearch 2.3.3
Plugin 2.1

Here is the current error

C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar<file:///C:\Users\holtzm\workspace\apache-manifoldcf-2.4\dist\plugins\elasticsearch\elasticsearch\elasticsearch-plugin-mcf-2.1.jar>
-> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar...
Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
...
Downloading .DONE
Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar
checksums if available ...
NOTE: Unable to verify checksum for downloaded plugin (unable to find .sha1 or .md5 file to
verify)
ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in plugin zip





From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Tuesday, August 30, 2016 10:22 AM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Hi Mike,

If you are just trying things out you can use the single-process example and it will make
your life easier for experimentation.

As far as the ES plugin, there are two: one for ES up to 1.5, and one for 1.5 onward.  Which
did you try to install?

Karl


On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
Karl,

You are correct.

Before I setup zookeeper I would like to prove out a few use cases.


1.       Index custom meta data fields from SharePoint into the ElasticSearch index

2.       Full text indexing of SharePoint content in ElasticSearch

3.       SharePoint security information indexed in ElasticSearch

4.       ElasticSearch Plugin to enforce SharePoint security settings for search

At this point I’ve only managed to see #2 and #3 work

For #4 I was getting an error when installing the ElasticSearch plugin.
I would like to see #1 work before I do anything else. Are there any example configurations
that show how to do this?


Thanks,
Mike H.

From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Tuesday, August 30, 2016 9:49 AM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Hi Mike,

This sounds like you're having a lock problem.

Let me guess: you are using the multiprocess example with file-based locking?  If so, you
need to stop everything and do the "lock clean" procedure.  I would also strongly recommend
that you switch to the zookeeper multiprocess example.

Thanks,
Karl


On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
I currently cannot view the job. The UI is hanging when I click the “view” link for the
job. I’m wondering if that has anything to do with the “Metadata Adjuster” I added?

I did not schedule the job.

At this point the job has finished running and the Document Status and Queue status reports.
I do not see any errors reported there.

Also I still do not see my custom meta data fields from SharePoint in the ElasticSearch index.

We are hoping to use this in production, but at this point it is not looking too good.

Thanks,
Mike H.

From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Tuesday, August 30, 2016 9:09 AM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

It is also important to know whether you have your job set up as "continuous".  If so, it
will run forever.

Karl


On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <furkankamaci@gmail.com<mailto:furkankamaci@gmail.com>>
wrote:
Hi Mike,

Could you tell us what "Document Status" and "Queue Status" says as Karl mentioned?

Kind Regards,
Furkan KAMACI

On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
I’m looking at manifoldcf.log and I see no errors.

I have only 13 files and they are all very tiny < 15kb .docx files.

Mike H.

From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Monday, August 29, 2016 5:26 PM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

You should be looking at the ManifoldCF.log file, and the examples have logging for errors
at least as being the default.

The only other possibility is that you have a very large document that is taking a long time
to transfer/index.

Also, you can generally see what MCF is up to using the "document status" and "queue status"
reports.

Karl


On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
Karl,

I see no errors in the log.

I probably need to configure the logging I think.

Thanks,
Mike H.

From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Monday, August 29, 2016 5:20 PM

To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Hi Mike,

Have a look in the log.  If there are errors there please send them to us.  Generally, "hangs"
are due to MCF retrying documents that have errors.

Karl


On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
Karl,

Yes, it shows up. I include all meta-data.

I’m no trying to use a MetaData adjuster like this

Parameter
field1                     “${Field 1}”

And now I see “field1”=”” in Elastic Search.

Also it seems each time I make a change the job will hang and I need to restart everything
including ElasticSearch to get it to complete.

Thanks,
Mike H.

From: Karl Wright [mailto:daddywri@gmail.com<mailto:daddywri@gmail.com>]
Sent: Monday, August 29, 2016 5:13 PM
To: user@manifoldcf.apache.org<mailto:user@manifoldcf.apache.org>
Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint
Meta data fields?

Hi Mike,

If I were you I'd try to independently verify that the custom field shows up.  Do you see
it appear in the pulldown for selecting metadata?  If not, it probably is not accessible through
the inspection methods that give us access into SharePoint fields.

If it appears there then we should look at what ES gets sent.  It's possible it appears but
under a different field name.

Karl


On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) <MHoltz@consultantemail.com<mailto:MHoltz@consultantemail.com>>
wrote:
Hi,

I have created a custom content type with a custom field “Field 1”.

I have configured the job to include all meta data.

I do not see “Field 1” being populated in the Elastic Search index.


Does anyone have any experience with this?

Thanks in advance,
Mike H









Mime
View raw message