manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damien Collis <Damien.Col...@aas.com.au>
Subject RE: Issue Extracting Authorities.
Date Thu, 21 Dec 2017 21:02:30 GMT
Thanks,

This has resolved my issue.

Regards
Damien Collis
Team Leader – Systems Integration
Link Group

•   Level 4, 1A Homebush Bay Drive, Rhodes NSW 2138
• Email: damien.collis@linkgroup.com<mailto:damien.collis@linkgroup.com>
• Ph: +61 2 8571 5616

From: Karl Wright [mailto:daddywri@gmail.com]
Sent: Thursday, 21 December 2017 9:13 PM
To: user@manifoldcf.apache.org
Subject: Re: Issue Extracting Authorities.

Right, we cannot distribute jcifs.jar for licensing reasons.  You can also build ManifoldCF
yourself from the distribution sources and libs and then run "ant make-deps" to download the
missing jars.  All of this is described in the "how-to-build-and-deploy" page.

Thanks,
Karl


On Wed, Dec 20, 2017 at 11:25 PM, Shinichiro Abe <shinichiro.abe.1@gmail.com<mailto:shinichiro.abe.1@gmail.com>>
wrote:
Hi,

> 6. Created Repository connection of Type: “File System” (There was no windows share
connector available in the drop down as stated in the documentation)
LocalFileConnector does not get access tokens of windows shared files.
To use SharedDriveConnector, you want to put the following for o.a.manifoldcf.connectorsconfigurationfile(i.e.
connectors.xml) :
  <repositoryconnector name="Windows shares" class="org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector"/>
Then you need to download jcifs.jar and put into libdir(i.e. connector-lib).

Regards,
Shinichiro Abe


2017-12-21 11:40 GMT+09:00 Damien Collis <Damien.Collis@aas.com.au<mailto:Damien.Collis@aas.com.au>>:
Hi User Group,

I am attempting to use Manifoldcf 2.8.1 and Solr 7.1.0 to index windows file system documents.
I am currently experiencing issues extracting the authority tokens, essentially no security
tokens are being propagated to Solr

I have implemented the following to no success.

1. Added new Authority Group “LinkGroup”
2. Created an authority connection to  my AD domain controller associated to the “LinkGroup”
Authority Group – Connection status: Connection Working
3. Tested the http://haystack:8345/mcf-combined-service-2.8.1/UserACLs?username=user@domain<https://urldefense.proofpoint.com/v2/url?u=http-3A__haystack-3A8345_mcf-2Dcombined-2Dservice-2D2.8.1_UserACLs-3Fusername-3Duser-40domain&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=ql15-SLpnEjw8V5aYAQMFLF71sfapsBYr42SBRxYH7Q&e=>
and received:
AUTHORIZED:LinkGroup
TOKEN:LinkGroup:S-1-5-21-1537756157-1994918190-4060197294-17387
TOKEN:LinkGroup:S-1-5-21-1537756157-1994918190-4060197294-1198
TOKEN:LinkGroup:S-1-5-21-1537756157-1994918190-4060197294-1190
….

4. Added fields to the Solr Schema xml file.
<field name="allow_token_document" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>
<field name="allow_token_parent" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>
<field name="allow_token_share" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>
<field name="deny_token_document" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>
<field name="deny_token_parent" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>
<field name="deny_token_share" type="string" indexed="true" stored="true" multiValued="true"
required="false" default="__nosecurity__"/>

5. Copied  apache-manifoldcf-solr-7.x-plugin-2.2.jar to  D:\ProgramFiles\solr-7.1.0a\solr-7.1.0-bin\contrib\extraction\lib
 (I wasn’t sure of the exact location to copy this lib)

6. Created Repository connection of Type: “File System” (There was no windows share connector
available in the drop down as stated in the documentation)

7. Created job to crawl LinkGroup file system.


I can see the following in my Solr logs, I was expecting to see the access tokens, but I’m
not sure how that information is passed to Solr or if it is presented in the logs:

2017-12-20 21:14:07.086 INFO  (qtp466002798-20) [   x:LinkGroup] o.a.s.u.p.LogUpdateProcessorFactory
[LinkGroup]  webapp=/solr path=/update/extract params={literal.uri=\\servername\HaystackTest\All.txt&resource.name<https://urldefense.proofpoint.com/v2/url?u=http-3A__resource.name&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=YvD885phPVeSCtY9uD--P348ec1-RNwoULJa3VMMBgY&e=>=All.txt&literal.id<https://urldefense.proofpoint.com/v2/url?u=http-3A__literal.id&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=tY5xQUH8PdP528EmqcgxDkypEp21sM3V4tjn8X61ccs&e=>=file:////servername/HaystackTest/All.txt&wt=xml&version=2.2}{add=[file:////servername/HaystackTest/All.txt
(1587339011890872320)]} 0 33
2017-12-20 21:14:07.102 INFO  (qtp466002798-19) [   x:LinkGroup] o.a.s.u.p.LogUpdateProcessorFactory
[LinkGroup]  webapp=/solr path=/update/extract params={literal.uri=\\servername\HaystackTest\secured.txt&resource.name<https://urldefense.proofpoint.com/v2/url?u=http-3A__resource.name&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=YvD885phPVeSCtY9uD--P348ec1-RNwoULJa3VMMBgY&e=>=secured.txt&literal.id<https://urldefense.proofpoint.com/v2/url?u=http-3A__literal.id&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=tY5xQUH8PdP528EmqcgxDkypEp21sM3V4tjn8X61ccs&e=>=file:////servername/HaystackTest/secured.txt&wt=xml&version=2.2}{add=[file:////servername/HaystackTest/secured.txt
(1587339011907649536)]} 0 46
2017-12-20 21:14:20.055 INFO  (qtp466002798-15) [   x:LinkGroup] o.a.s.u.DirectUpdateHandler2
start commit{_version_=1587339025506631680,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
2017-12-20 21:14:20.055 INFO  (qtp466002798-15) [   x:LinkGroup] o.a.s.u.SolrIndexWriter Calling
setCommitData with IW:org.apache.solr.update.SolrIndexWriter@68f515e5 commitCommandVersion:1587339025506631680
2017-12-20 21:14:20.070 INFO  (qtp466002798-15) [   x:LinkGroup] o.a.s.s.SolrIndexSearcher
Opening [Searcher@30e03581[LinkGroup] main]
2017-12-20 21:14:20.070 INFO  (searcherExecutor-7-thread-1-processing-x:LinkGroup) [   x:LinkGroup]
o.a.s.c.QuerySenderListener QuerySenderListener sending requests to Searcher@30e03581[LinkGroup]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_26(7.1.0):C2) Uninverting(_27(7.1.0):C2)))}
2017-12-20 21:14:20.070 INFO  (qtp466002798-15) [   x:LinkGroup] o.a.s.u.DirectUpdateHandler2
end_commit_flush
2017-12-20 21:14:20.070 INFO  (searcherExecutor-7-thread-1-processing-x:LinkGroup) [   x:LinkGroup]
o.a.s.c.QuerySenderListener QuerySenderListener done.
2017-12-20 21:14:20.070 INFO  (searcherExecutor-7-thread-1-processing-x:LinkGroup) [   x:LinkGroup]
o.a.s.c.SolrCore [LinkGroup] Registered new searcher Searcher@30e03581[LinkGroup] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_26(7.1.0):C2)
Uninverting(_27(7.1.0):C2)))}
2017-12-20 21:14:20.070 INFO  (qtp466002798-15) [   x:LinkGroup] o.a.s.u.p.LogUpdateProcessorFactory
[LinkGroup]  webapp=/solr path=/update/extract params={commit=true&wt=xml&version=2.2}{commit=}
0 25


Any assistance would be highly appreciated.




Regards
Damien Collis
Team Leader – Systems Integration
Link Group

•   Level 4, 1A Homebush Bay Drive, Rhodes NSW 2138<https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D1A-2BHomebush-2BBay-2BDrive-2C-2BRhodes-2BNSW-2B2138-26entry-3Dgmail-26source-3Dg&d=DwMFaQ&c=EyrAshB9xIzcegaT9SDe6g&r=Gn5yxeb6W9ERepUyEmssft7I4Tobgyxsu0tR69ePkS8&m=HWVrE2lsbMKYTjtInbg-zhB0pWkxCmzefQuBPN0bv-s&s=eynnCVdqaKiwFYcPfztVkwCnzJ-fIbENRLu8598kNxw&e=>
• Email: damien.collis@linkgroup.com<mailto:damien.collis@linkgroup.com>
• Ph: +61 2 8571 5616<tel:+61%202%208571%205616>



Mime
View raw message