Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D4875200B76 for ; Tue, 30 Aug 2016 17:54:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D15DA160ABA; Tue, 30 Aug 2016 15:54:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F21CF160AAF for ; Tue, 30 Aug 2016 17:54:46 +0200 (CEST) Received: (qmail 13438 invoked by uid 500); 30 Aug 2016 15:54:46 -0000 Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@manifoldcf.apache.org Delivered-To: mailing list user@manifoldcf.apache.org Received: (qmail 13424 invoked by uid 99); 30 Aug 2016 15:54:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Aug 2016 15:54:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 8CB8D18649E for ; Tue, 30 Aug 2016 15:54:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.4 X-Spam-Level: ** X-Spam-Status: No, score=2.4 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_FONT_LOW_CONTRAST=0.001, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 8lzGZoJmJ7Qj for ; Tue, 30 Aug 2016 15:54:41 +0000 (UTC) Received: from mail-it0-f44.google.com (mail-it0-f44.google.com [209.85.214.44]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 154895FADC for ; Tue, 30 Aug 2016 15:54:40 +0000 (UTC) Received: by mail-it0-f44.google.com with SMTP id x131so172315516ite.0 for ; Tue, 30 Aug 2016 08:54:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=n9Pn3RmDeIKlm/W27HIKCLze6vWyV90GU2e5TOs1U7o=; b=wd3M/vSSEIkYNuOKhxVaVscAEKNEpl3sOwN6tZRroXz0441w6jzj2p1NqhbeWcLIxB 3NvzZlwzv9MBfW1xbqgpFqlxLPftNIAx4dHB+MA9ofTJIei0ZC0XNOF+8CHnmR+hnSvW ACwTPUeuTh9NSl2wGvk+iEl8lIC7UBGLMoHfy6eYNeCaJYmODGk57w/eEvTuHQR+9v+d sIk2hJ96zGSntU5uNVpClnj5z/gL9sEaGlR7adYDSMjbXlWRNvCR1o2L95Cml66JOduL Og9c0ykEN7eK+xf5NNzf9af1Gvdrc2vLy5EqrhuQeq/zcH85ysaxYMMk3QC3Ds3Mltqo WlPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=n9Pn3RmDeIKlm/W27HIKCLze6vWyV90GU2e5TOs1U7o=; b=NxsarYAQa9y0qjMh9vYMFqgeaphrAI2QoDIq2qMVNuWOdkwYIhk4Wfr59+P4Y5E7i8 E/clEQBD6frR613YU5yAq0ThkA/+Up7Ehl+KDcEXgTOLbrXW+axiRV3AKNvO4VNpwQBn u86G8fCK7iPJntBs7MEx/ySwVtD/loi1if8rMeLSCTVFvfFpAOTr/GynALxFl4T9LWke wOKLCfjrZ7XcHu0XMVs84BuyzPideO42Hl6H1QhUNIWVC4fXEIIDx7zjRV1gvpCtGjO+ aAFQl6A83V+1OspINEwTzh/y/6h/lHKATXh2d20nrrfZbL/0LFyXWL/bDBhBpGVwjXWo 9G1g== X-Gm-Message-State: AE9vXwPu9a2AL8xf4u70Ya3VM3soywIwKUgRlytPeZQNAOxmKQPEU2nq4WiYVdy+dqbhJcAm764n70A4Z80DIA== X-Received: by 10.36.3.76 with SMTP id e73mr25772399ite.42.1472572478751; Tue, 30 Aug 2016 08:54:38 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.18.31 with HTTP; Tue, 30 Aug 2016 08:54:37 -0700 (PDT) In-Reply-To: References: <02cb564630da43e28a46ae528c97364b@MDCAMSPRDMBX2.wellmanage.com> <03e787474597429783e165f57edfaba0@MDCAMSPRDMBX2.wellmanage.com> From: Karl Wright Date: Tue, 30 Aug 2016 11:54:37 -0400 Message-ID: Subject: Re: Has anyone successfully populated Elastic Search properties from custom SharePoint Meta data fields? To: "user@manifoldcf.apache.org" Content-Type: multipart/alternative; boundary=001a1141994080acd5053b4bffe3 archived-at: Tue, 30 Aug 2016 15:54:49 -0000 --001a1141994080acd5053b4bffe3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ok, you just included this in the jar and all was OK? It should be straightforward to set that up if so... Thanks! Karl On Tue, Aug 30, 2016 at 11:27 AM, Holtz, Michael (IT Consultant) < MHoltz@consultantemail.com> wrote: > Karl, > > > > Ok, I will see what I can do. > > > > I just created the attached file for the 1.5 version of the plugin and wa= s > able to install it. Let me know if you see any issues with it. > > > > I=E2=80=99m going to start with a clean version of the new 2.5 release fo= r my use > cases just using the example setup, let me know if you don=E2=80=99t thin= k that is > a good idea. > > > > Thanks, > > Mike H. > > > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Tuesday, August 30, 2016 11:17 AM > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > The 2.0 spec has clearly changed from that needed for the 1.5-level > plugins. > > > > The plugin code is likely to be pretty much unchanged but the packaging > will need to change. Now, I did not develop the 1.5-level plugin, and > don't know that much about ES in the first place, so if you want to > undertake the plugin transformation and contribute it, we'd be very > grateful. > > > > Karl > > > > > > > > On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > It looks like a =E2=80=98plugin-descriptor.properties=E2=80=99 file needs= to be added to > the jar: > > > > https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/ > plugin-authors.html > > > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:53 AM > > > *To:* user@manifoldcf.apache.org > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Karl, > > > > > > FYI, Now I=E2=80=99m seeing this: > > > > > > C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/ > workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/ > elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf- > 2.0.1-jar-with-dependencies.jar > > -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar... > > Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar ... > > Downloading ...........................DONE > > Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch-1.5/target/ > elasticsearch-1.5-plugin-mcf-2.0.1-jar-with-dependencies.jar checksums if > available ... > > NOTE: Unable to verify checksum for downloaded plugin (unable to find > .sha1 or .md5 file to verify) > > ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in > plugin zip > > > > Mike H. > > > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:48 AM > *To:* user@manifoldcf.apache.org > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Karl, > > > > I got what your=E2=80=99re saying, I should use the 1.5 plugin. Got it. > > > > I will try that > > > > Thanks, > > Mike H. > > > > *From:* Holtz, Michael (IT Consultant) > *Sent:* Tuesday, August 30, 2016 10:47 AM > *To:* user@manifoldcf.apache.org > *Subject:* RE: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hmm that=E2=80=99s confusing, here is the documentation: > > > > *Apache ManifoldCF ElasticSearch plugin compatibility* > > *ManifoldCF versions* > > *Plugin version* > > 0.1.x-1.4.x > > 0.x > > 1.5.x > > 1.x > > >=3D1.6.x > > 2.x > > > > > > > > I=E2=80=99m running version 2.4 of manifold so based on this I went with = the 2.1 > plugin. > > > > > > Where can I get the 1.0 plugin? I don=E2=80=99t see it on the site or in = the 2.4 > build. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com ] > *Sent:* Tuesday, August 30, 2016 10:38 AM > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > You need the ES-1.5 plugin. That's the older (deprecated) 1.0 plugin. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > Elasticsearch 2.3.3 > > Plugin 2.1 > > > > Here is the current error > > > > C:\APPS\elasticsearch-2.3.3>bin\plugin install file:///C:/Users/holtzm/ > workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/ > elasticsearch/elasticsearch-plugin-mcf-2.1.jar > > -> Installing from file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch- > plugin-mcf-2.1.jar... > > Trying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.j= ar > ... > > Downloading .DONE > > Verifying file:/C:/Users/holtzm/workspace/apache-manifoldcf-2. > 4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.j= ar > checksums if available ... > > NOTE: Unable to verify checksum for downloaded plugin (unable to find > .sha1 or .md5 file to verify) > > ERROR: Could not find plugin descriptor 'plugin-descriptor.properties' in > plugin zip > > > > > > > > > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Tuesday, August 30, 2016 10:22 AM > > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > If you are just trying things out you can use the single-process example > and it will make your life easier for experimentation. > > > > As far as the ES plugin, there are two: one for ES up to 1.5, and one for > 1.5 onward. Which did you try to install? > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > Karl, > > > > You are correct. > > > > Before I setup zookeeper I would like to prove out a few use cases. > > > > 1. Index custom meta data fields from SharePoint into the > ElasticSearch index > > 2. Full text indexing of SharePoint content in ElasticSearch > > 3. SharePoint security information indexed in ElasticSearch > > 4. ElasticSearch Plugin to enforce SharePoint security settings for > search > > > > At this point I=E2=80=99ve only managed to see #2 and #3 work > > > > For #4 I was getting an error when installing the ElasticSearch plugin. > > I would like to see #1 work before I do anything else. Are there any > example configurations that show how to do this? > > > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Tuesday, August 30, 2016 9:49 AM > > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > This sounds like you're having a lock problem. > > > > Let me guess: you are using the multiprocess example with file-based > locking? If so, you need to stop everything and do the "lock clean" > procedure. I would also strongly recommend that you switch to the > zookeeper multiprocess example. > > > > Thanks, > > Karl > > > > > > On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > I currently cannot view the job. The UI is hanging when I click the =E2= =80=9Cview=E2=80=9D > link for the job. I=E2=80=99m wondering if that has anything to do with t= he > =E2=80=9CMetadata Adjuster=E2=80=9D I added? > > > > I did not schedule the job. > > > > At this point the job has finished running and the Document Status and > Queue status reports. I do not see any errors reported there. > > > > Also I still do not see my custom meta data fields from SharePoint in the > ElasticSearch index. > > > > We are hoping to use this in production, but at this point it is not > looking too good. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Tuesday, August 30, 2016 9:09 AM > > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > It is also important to know whether you have your job set up as > "continuous". If so, it will run forever. > > > > Karl > > > > > > On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI > wrote: > > Hi Mike, > > > > Could you tell us what "Document Status" and "Queue Status" says as Karl > mentioned? > > > > Kind Regards, > > Furkan KAMACI > > > > On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > I=E2=80=99m looking at manifoldcf.log and I see no errors. > > > > I have only 13 files and they are all very tiny < 15kb .docx files. > > > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Monday, August 29, 2016 5:26 PM > > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > You should be looking at the ManifoldCF.log file, and the examples have > logging for errors at least as being the default. > > > > The only other possibility is that you have a very large document that is > taking a long time to transfer/index. > > > > Also, you can generally see what MCF is up to using the "document status" > and "queue status" reports. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > Karl, > > > > I see no errors in the log. > > > > I probably need to configure the logging I think. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Monday, August 29, 2016 5:20 PM > > > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > Have a look in the log. If there are errors there please send them to > us. Generally, "hangs" are due to MCF retrying documents that have error= s. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > Karl, > > > > Yes, it shows up. I include all meta-data. > > > > I=E2=80=99m no trying to use a MetaData adjuster like this > > > > Parameter > > field1 =E2=80=9C${Field 1}=E2=80=9D > > > > And now I see =E2=80=9Cfield1=E2=80=9D=3D=E2=80=9D=E2=80=9D in Elastic Se= arch. > > > > Also it seems each time I make a change the job will hang and I need to > restart everything including ElasticSearch to get it to complete. > > > > Thanks, > > Mike H. > > > > *From:* Karl Wright [mailto:daddywri@gmail.com] > *Sent:* Monday, August 29, 2016 5:13 PM > *To:* user@manifoldcf.apache.org > *Subject:* Re: Has anyone successfully populated Elastic Search > properties from custom SharePoint Meta data fields? > > > > Hi Mike, > > > > If I were you I'd try to independently verify that the custom field shows > up. Do you see it appear in the pulldown for selecting metadata? If not= , > it probably is not accessible through the inspection methods that give us > access into SharePoint fields. > > > > If it appears there then we should look at what ES gets sent. It's > possible it appears but under a different field name. > > > > Karl > > > > > > On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT Consultant) < > MHoltz@consultantemail.com> wrote: > > Hi, > > > > I have created a custom content type with a custom field =E2=80=9CField 1= =E2=80=9D. > > > > I have configured the job to include all meta data. > > > > I do not see =E2=80=9CField 1=E2=80=9D being populated in the Elastic Sea= rch index. > > > > > > Does anyone have any experience with this? > > > > Thanks in advance, > > Mike H > > > > > > > > > > > > > > > > > > > --001a1141994080acd5053b4bffe3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Ok, you just included this in the jar and all was OK?=C2= =A0 It should be straightforward to set that up if so...

Thanks!
Karl


On Tue, Aug 30, 2016 at 11:27 AM, Holtz, M= ichael (IT Consultant) <MHoltz@consultantemail.com>= wrote:

Karl,

=C2=A0

Ok, I will see what I can= do.

=C2=A0

I just created the attach= ed file for the 1.5 version of the plugin and was able to install it. Let m= e know if you see any issues with it.

=C2=A0

I=E2=80=99m going to star= t with a clean version of the new 2.5 release for my use cases just using t= he example setup, let me know if you don=E2=80=99t think that is a good idea.

=C2=A0

Thanks,

Mike H.

=C2=A0

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Tuesday, August 30, 2016 11:17 AM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

<= /p>

=C2=A0

The 2.0 spec has clearly changed from that needed fo= r the 1.5-level plugins.

=C2=A0

The plugin code is likely to be pretty much unchange= d but the packaging will need to change.=C2=A0 Now, I did not develop the 1= .5-level plugin, and don't know that much about ES in the first place, = so if you want to undertake the plugin transformation and contribute it, we'd be very grateful.

=C2=A0

Karl

=C2=A0

=C2=A0

=C2=A0

On Tue, Aug 30, 2016 at 11:09 AM, Holtz, Michael (IT= Consultant) <MHoltz@consultantemail.com> wrote:

It looks like a =E2=80=98= plugin-descriptor.properties=E2=80=99 file needs to be added to the jar:

=C2=A0

https://www.elastic.co/guide/en/elasticsearch/plugins/2.0/plugin-authors.html

=C2=A0

=C2=A0

From: Holtz, M= ichael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:53 AM


To: = user@manifoldcf.apache.org
Subject: RE: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Karl,

=C2=A0

=C2=A0

FYI, Now I=E2=80=99m seei= ng this:

=C2=A0

=C2=A0

C:\APPS\elasticsearch-2.3= .3>bin\plugin install file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plug= ins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plug= in-mcf-2.0.1-jar-with-dependencies.jar

-> Installing from fil= e:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/e= lasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mc= f-2.0.1-jar-with-dependencies.jar...

Trying file:/C:/Users/hol= tzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1-ja= r-with-dependencies.jar ...

Downloading .............= ..............DONE

Verifying file:/C:/Users/= holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch= /elasticsearch-1.5/target/elasticsearch-1.5-plugin-mcf-2.0.1= -jar-with-dependencies.jar checksums if available ...

NOTE: Unable to verify ch= ecksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)<= /span>

ERROR: Could not find plu= gin descriptor 'plugin-descriptor.properties' in plugin zip<= u>

=C2=A0

Mike H.<= /u>

=C2=A0

=C2=A0

From: Holtz, M= ichael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:48 AM
To: = user@manifoldcf.apache.org
Subject: RE: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Karl,

=C2=A0

I got what your=E2=80=99r= e saying, I should use the 1.5 plugin. Got it.

=C2=A0

I will try that=

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Holtz, M= ichael (IT Consultant)
Sent: Tuesday, August 30, 2016 10:47 AM
To: = user@manifoldcf.apache.org
Subject: RE: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Hmm that=E2=80=99s confus= ing, here is the documentation:

=C2=A0

Apache ManifoldCF ElasticSearch plugin compatibi= lity

ManifoldCF versions

Plugin version

2.x

=C2=A0

=C2=A0

=C2=A0

I=E2=80=99m running versi= on 2.4 of manifold so based on this I went with the 2.1 plugin.

=C2=A0

=C2=A0

Where can I get the 1.0 p= lugin? I don=E2=80=99t see it on the site or in the 2.4 build.

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Tuesday, August 30, 2016 10:38 AM
To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

You need the ES-1.5 plugin.=C2=A0 That's the old= er (deprecated) 1.0 plugin.

=C2=A0

Karl

=C2=A0

=C2=A0

On Tue, Aug 30, 2016 at 10:36 AM, Holtz, Michael (IT= Consultant) <MHoltz@consultantemail.com> wrote:

Elasticsearch 2.3.3

Plugin 2.1<= u>

=C2=A0

Here is the current error=

=C2=A0

C:\APPS\elasticsearch-2.3= .3>bin\plugin install file:///C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plug= ins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar<= /a>

-> Installing from fil= e:/C:/Users/holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/e= lasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar...

Trying file:/C:/Users/hol= tzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch/elasticsearch/elasticsearch-plugin-mcf-2.1.jar ...

Downloading .DONE<= u>

Verifying file:/C:/Users/= holtzm/workspace/apache-manifoldcf-2.4/dist/plugins/elasticsearch= /elasticsearch/elasticsearch-plugin-mcf-2.1.jar checksums if available ...

NOTE: Unable to verify ch= ecksum for downloaded plugin (unable to find .sha1 or .md5 file to verify)<= /span>

ERROR: Could not find plu= gin descriptor 'plugin-descriptor.properties' in plugin zip<= u>

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Tuesday, August 30, 2016 10:22 AM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Hi Mike,

=C2=A0

If you are just trying things out you can use the si= ngle-process example and it will make your life easier for experimentation.=

=C2=A0

As far as the ES plugin, there are two: one for ES u= p to 1.5, and one for 1.5 onward.=C2=A0 Which did you try to install?

=C2=A0

Karl

=C2=A0

=C2=A0

On Tue, Aug 30, 2016 at 9:56 AM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

Karl,

=C2=A0

You are correct.

=C2=A0

Before I setup zookeeper = I would like to prove out a few use cases.

=C2=A0

1.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Index custom meta data fields from ShareP= oint into the ElasticSearch index

2.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Full text indexing of SharePoint content = in ElasticSearch

3.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 SharePoint security information indexed i= n ElasticSearch

4.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ElasticSearch Plugin to enforce SharePoin= t security settings for search

=C2=A0

At this point I=E2=80=99v= e only managed to see #2 and #3 work

=C2=A0

For #4 I was getting an e= rror when installing the ElasticSearch plugin.

I would like to see #1 wo= rk before I do anything else. Are there any example configurations that show how to do this?

=C2=A0

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Tuesday, August 30, 2016 9:49 AM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Hi Mike,

=C2=A0

This sounds like you're having a lock problem.

=C2=A0

Let me guess: you are using the multiprocess example= with file-based locking?=C2=A0 If so, you need to stop everything and do t= he "lock clean" procedure.=C2=A0 I would also strongly recommend that you switch to the zookeeper multiprocess example.

=C2=A0

Thanks,

Karl

=C2=A0

=C2=A0

On Tue, Aug 30, 2016 at 9:44 AM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

I currently cannot view t= he job. The UI is hanging when I click the =E2=80=9Cview=E2=80=9D link for = the job. I=E2=80=99m wondering if that has anything to do with the =E2=80=9CMetadata Adjuster= =E2=80=9D I added?

=C2=A0

I did not schedule the jo= b.

=C2=A0

At this point the job has= finished running and the Document Status and Queue status reports. I do not see any errors reported there.

=C2=A0

Also I still do not see m= y custom meta data fields from SharePoint in the ElasticSearch index.

=C2=A0

We are hoping to use this= in production, but at this point it is not looking too good.=

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Tuesday, August 30, 2016 9:09 AM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

It is also important to know whether you have your j= ob set up as "continuous".=C2=A0 If so, it will run forever.

=C2=A0

Karl

=C2=A0

=C2=A0

On Tue, Aug 30, 2016 at 8:48 AM, Furkan KAMACI <<= a href=3D"mailto:furkankamaci@gmail.com" target=3D"_blank">furkankamaci@gma= il.com> wrote:

Hi Mike,

=C2=A0

Could you tell us what "Document Status" a= nd "Queue Status" says as Karl mentioned?

=C2=A0

Kind Regards,

Furkan KAMACI

=C2=A0

On Tue, Aug 30, 2016 at 3:39 PM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

I=E2=80=99m looking at ma= nifoldcf.log and I see no errors.

=C2=A0

I have only 13 files and = they are all very tiny < 15kb .docx files.

=C2=A0

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Monday, August 29, 2016 5:26 PM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

You should be looking at the ManifoldCF.log file, an= d the examples have logging for errors at least as being the default.

=C2=A0

The only other possibility is that you have a very l= arge document that is taking a long time to transfer/index.

=C2=A0

Also, you can generally see what MCF is up to using = the "document status" and "queue status" reports.

=C2=A0

Karl

=C2=A0

=C2=A0

On Mon, Aug 29, 2016 at 5:22 PM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

Karl,

=C2=A0

I see no errors in the lo= g.

=C2=A0

I probably need to config= ure the logging I think.

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Monday, August 29, 2016 5:20 PM


To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Hi Mike,

=C2=A0

Have a look in the log.=C2=A0 If there are errors th= ere please send them to us.=C2=A0 Generally, "hangs" are due to M= CF retrying documents that have errors.

=C2=A0

Karl

=C2=A0

=C2=A0

On Mon, Aug 29, 2016 at 5:18 PM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

Karl,

=C2=A0

Yes, it shows up. I inclu= de all meta-data.

=C2=A0

I=E2=80=99m no trying to = use a MetaData adjuster like this

=C2=A0

Parameter

field1=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 =E2=80=9C${Field 1}=E2=80=9D

=C2=A0

And now I see =E2=80=9Cfi= eld1=E2=80=9D=3D=E2=80=9D=E2=80=9D in Elastic Search.<= /p>

=C2=A0

Also it seems each time I= make a change the job will hang and I need to restart everything including ElasticSearch to get it to complete.

=C2=A0

Thanks,<= /u>

Mike H.<= /u>

=C2=A0

From: Karl Wri= ght [mailto:daddywr= i@gmail.com]
Sent: Monday, August 29, 2016 5:13 PM
To: = user@manifoldcf.apache.org
Subject: Re: Has anyone successfully populated Elastic Search proper= ties from custom SharePoint Meta data fields?

=C2=A0

Hi Mike,

=C2=A0

If I were you I'd try to independently verify th= at the custom field shows up.=C2=A0 Do you see it appear in the pulldown fo= r selecting metadata?=C2=A0 If not, it probably is not accessible through the inspection methods that give us access into SharePoint fields.=

=C2=A0

If it appears there then we should look at what ES g= ets sent.=C2=A0 It's possible it appears but under a different field na= me.

=C2=A0

Karl

=C2=A0

=C2=A0

On Mon, Aug 29, 2016 at 5:01 PM, Holtz, Michael (IT = Consultant) <MHoltz@consultantemail.com> wrote:

Hi,

=C2=A0

I have created a custom content type with a custom f= ield =E2=80=9CField 1=E2=80=9D.

=C2=A0

I have configured the job to include all meta data.<= u>

=C2=A0

I do not see =E2=80=9CField 1=E2=80=9D being populat= ed in the Elastic Search index.

=C2=A0

=C2=A0

Does anyone have any experience with this?=

=C2=A0

Thanks in advance,

Mike H

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0

=C2=A0


--001a1141994080acd5053b4bffe3--