manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: Problem in fetching the access tokens from Active Directory in Elastic Search index for FileSystem Connector
Date Mon, 05 Feb 2018 14:26:09 GMT
No, everything looks properly configured.

First, please verify that the document for which you are seeing
"__nosecurity__" tokens being indexed is NOT a public document in
Sharepoint.  If it is public, then the security tokens being indexed are
indeed correct.

Once you have verified that, then the next step is to turn on wire
debugging.  This will log all SOAP traffic back and forth between
ManifoldCF and the server.  We really need the logs for only one document
being indexed -- the goal is to see whether the response we're getting back
from the MCPermissions call actually returns any ACLs.  It seems like it is
silently failing to do so -- very bad, because that means that security is
not being enforced.  Turning on wire debugging involves making changes to
the logging.xml file; you will need to add appropriate loggers for
HttpComponents/HttpClient.  The instructions are here:

https://cwiki.apache.org/confluence/display/CONNECTORS/Debugging+Connections

Once you have redacted whatever sensitive information you feel necessary,
please post a link to a google doc that includes the chunk of log
corresponding to one document that is not public.

Thanks!

Karl




On Mon, Feb 5, 2018 at 9:09 AM, Nikita Ahuja <nikita@smartshore.nl> wrote:

> Hi Karl,
>
>
> This is Sharepoint 2013 version being used. and it uses the Claim Based
> Authentication technique only.
>  And the repository connection created is for Sharepoint 2013 version only
> [image: Inline image 1]
>
>
>
> [image: Inline image 2]
>
>
> [image: Inline image 3]
>
> Is there anything which is being missed to add in the connector??
>
>
>
> Thanks and Regards,
> Nikita
>
>
> On Mon, Feb 5, 2018 at 7:33 PM, Karl Wright <daddywri@gmail.com> wrote:
>
>> The "__nosecurity__" values, for Elastic Search as an output, come from
>> ManifoldCF.  So you are definitely getting the access tokens from
>> ManifoldCF to ElasticSearch.
>>
>> The configuration for the Security tab also looks correct.
>>
>> So at the moment I cannot explain why you are not getting security tokens
>> out of SharePoint.  About the only possibility left is whether the wrong
>> version of Sharepoint is specified.  Can you describe:
>> - What version of Sharepoint this is;
>> - Whether it is configured to use Claims Based authorization
>>
>> And please then take a screen shot when you view the repository
>> connection.  Thanks!
>>
>> Karl
>>
>>
>> On Mon, Feb 5, 2018 at 3:21 AM, Nikita Ahuja <nikita@smartshore.nl>
>> wrote:
>>
>>> Hi Karl,
>>>
>>> I have done the configuration of the job as per the *End User
>>> Documentation*. Still, I am getting "no security" value in the access
>>> tokens in the elastic search index (output of my job).
>>>
>>> [image: Inline image 2]
>>>
>>>
>>> This is the Output connector for Elastic Search. Is there anything needs
>>> to added here?
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>> Also, the security is also enabled.
>>>
>>> [image: Inline image 3]
>>>
>>>
>>> I am not able to understand where is the actual problem?
>>>
>>>
>>> Thanks and Regards,
>>> Nikita
>>>
>>>
>>>
>>>
>>> On Fri, Feb 2, 2018 at 7:53 PM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>>> Hi Nikita,
>>>>
>>>> If you are seeing *no* tokens indexed then I would check your output
>>>> connection and your index configuration.  Custom fields need to be added
to
>>>> Solr for these tokens to be indexed, if you are using Solr.  There are
>>>> instructions in the online documentation on what to add and how to
>>>> configure them.
>>>>
>>>> The other possibility is that you have disabled security for you
>>>> Sharepoint job in the "Security" tab.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Fri, Feb 2, 2018 at 8:48 AM, Nikita Ahuja <nikita@smartshore.nl>
>>>> wrote:
>>>>
>>>>> Hi Karl,
>>>>>
>>>>>
>>>>> Yes, you are right I am able to fetch the files  but the access
>>>>> tokens indexed with the files are not included what is required. And
I am
>>>>> using the ManifoldCF security model not any custom. Everything is fine
but
>>>>> only problem is with the token.
>>>>> And am not sure what is the reason behind that: is it because the
>>>>> Sharepoint site which is used is not with *HTTPS* protocol or any
>>>>> other reason?
>>>>>
>>>>>
>>>>> Thanks and Regards,
>>>>> Nikita
>>>>>
>>>>> On Fri, Feb 2, 2018 at 7:05 PM, Karl Wright <daddywri@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I am not sure I understand -- are you saying that you can fetch the
>>>>>> files but the access tokens indexed with the files do not include
what you
>>>>>> require? And that the reason you require it done a certain way is
because
>>>>>> you intend to apply security in a manner of your own design?
>>>>>>
>>>>>> The access tokens that are indexed with files in SharePoint are what
>>>>>> are needed to enforce security using the ManifoldCF Sharepoint authorities,
>>>>>> no more and no less.  The authorities map an individual user to the
access
>>>>>> tokens needed for that user, which include tokens for the groups
as well as
>>>>>> for the user name.  The tokens indexed with the files do NOT include
tokens
>>>>>> generated by mapping groups to individual users -- the only tokens
indexed
>>>>>> for a file are for the groups and for any individual users specified
>>>>>> explicitly.
>>>>>>
>>>>>> I strongly suggest you attempt to apply security fully using the
>>>>>> ManifoldCF model rather than trying to do something custom.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 2, 2018 at 8:04 AM, Nikita Ahuja <nikita@smartshore.nl>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Karl,
>>>>>>>
>>>>>>> Thanks a lot for your help.It is working fine now.. I am able
to
>>>>>>> fetch all the documents in the path but there is a requirement
to get the
>>>>>>> access tokens for the files. But,it shows the authority group
name not any
>>>>>>> details related to the users who all have access for the document
accessed.
>>>>>>>
>>>>>>> Please guide for that.
>>>>>>>
>>>>>>>
>>>>>>> [image: Inline image 1]
>>>>>>>
>>>>>>>
>>>>>>> Thanks and Regards,
>>>>>>> Nikita
>>>>>>>
>>>>>>> On Wed, Jan 31, 2018 at 5:55 PM, Karl Wright <daddywri@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Clearly you are unable to fetch document ACLs from SharePoint.
>>>>>>>> This has nothing to do with whether your Active Directory
is properly
>>>>>>>> configured, and has more to do with the MCPermissions plugin.
>>>>>>>>
>>>>>>>> When you view the SharePoint repository connection, does
it say
>>>>>>>> "Connection working"?  If not, you need the figure out why.
>>>>>>>>
>>>>>>>> Here are some possibilities:
>>>>>>>> (1) The plugin was properly installed but with the wrong
>>>>>>>> permissions.  You must be logged in as an Admin to install
it properly.  If
>>>>>>>> that's not done it will mostly work but not be able to retrieve
the
>>>>>>>> permissions.
>>>>>>>> (2) You may have selected the wrong version of Sharepoint
in the
>>>>>>>> repository connection.  Make sure you select the right one
or, once again,
>>>>>>>> you won't be able to retrieve permissions.
>>>>>>>>
>>>>>>>> The documentation says this: "Select your SharePoint server
>>>>>>>> version from the pulldown. If you do not select the correct
server version,
>>>>>>>> your documents may either be indexed with insufficient security
protection,
>>>>>>>> or you may not be able to index any documents. Check with
your SharePoint
>>>>>>>> system administrator if you are not sure what to select."
>>>>>>>>
>>>>>>>> Please let me know whether this helps.  Thanks!
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jan 31, 2018 at 6:51 AM, Nikita Ahuja <nikita@smartshore.nl
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi Karl,
>>>>>>>>>
>>>>>>>>> I have checked the log Files but there is issue with
the ACLs but
>>>>>>>>> there is Active Directory already connected with Sharepoint
site. And it is
>>>>>>>>> not possible to pass the Access tokens in the connector.
Please provide the
>>>>>>>>> suggestion for the problem.
>>>>>>>>>
>>>>>>>>> [image: Inline image 1]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks and Regards,
>>>>>>>>> Nikita
>>>>>>>>>
>>>>>>>>> On Mon, Jan 29, 2018 at 6:55 PM, Karl Wright <daddywri@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> I don't see any path rules there.  That will clearly
not work.
>>>>>>>>>>
>>>>>>>>>> If you want confirmation of this, you can turn on
connector
>>>>>>>>>> debugging (set properties.xml property org.apache.manifoldcf.connectors
>>>>>>>>>> to "DEBUG"), look at the log output.  Every time
the SharePoint connector
>>>>>>>>>> makes a decision to not include something, it logs
it.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>> On Mon, Jan 29, 2018 at 8:07 AM, Nikita Ahuja <
>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have checked the documentation and the path
configuration as
>>>>>>>>>>> well. But still not able to get any output.
>>>>>>>>>>>
>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and the metadata path given is:
>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>
>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>> Nikita
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:45 PM, Karl Wright
<daddywri@gmail.com
>>>>>>>>>>> > wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I think your path configuration is incorrect.
 See the end-user
>>>>>>>>>>>> documentation.  It's a little tricky because
you must specify a matching
>>>>>>>>>>>> rule for both files and paths.
>>>>>>>>>>>>
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Jan 29, 2018 at 4:42 AM, Nikita Ahuja
<
>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Hi Karl,
>>>>>>>>>>>>>
>>>>>>>>>>>>> There is another query related to Sharepoint
2013 Connector. I
>>>>>>>>>>>>> am trying to crawl the files of Sharepoint
On Premise site. But I am not
>>>>>>>>>>>>> able to get anything in the output either
in ElasticSearch or in FileShare.
>>>>>>>>>>>>> I have followed given steps:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Created an Authority group for the
Active Directory and
>>>>>>>>>>>>>    same is used in Sharepoint Repository
Connection.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 2]
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Installed the Sharepoint 2013 MCPermission
Plugin and is
>>>>>>>>>>>>>    accessible on the site.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [image: Inline image 3]
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - Still not able to access any document
from the selected
>>>>>>>>>>>>>    folder of Sharepoint Site.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please guide me accordingly.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 6:46 PM, Nikita
Ahuja <
>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is working fine now and Also I
am able to get the access
>>>>>>>>>>>>>> tokens in the elastic search index.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks a lot for your valuable suggestions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> With Regards,
>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 4:42 PM,
Karl Wright <
>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The __nosecurity__ token is generated
by the Elastic Search
>>>>>>>>>>>>>>> Connector.  It is produced (correctly)
when no acls reach the connector.
>>>>>>>>>>>>>>> This is why I asked you about
the repository connection, because that
>>>>>>>>>>>>>>> connection is not producing any
ACLs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you are crawling files in
a file system, you must use the
>>>>>>>>>>>>>>> JCIFS connector if you want ACLs
to be generated.  The FileSystem connector
>>>>>>>>>>>>>>> does not generate ACLs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 3:56
AM, Nikita Ahuja <
>>>>>>>>>>>>>>> nikita@smartshore.nl> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I have tried for ElasticSearch
version 1.3.1 and 5.6.1 but
>>>>>>>>>>>>>>>> still getting this " __nosecurity__"
token and for
>>>>>>>>>>>>>>>> Repository connection  File
Sytem is used and for Transformation  "Tika
>>>>>>>>>>>>>>>> Content Extractor" and Metadata
adjuster Connector is used.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [image: Inline image 1]
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is there any other thing
which I need to perform??
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018 at 2:13
PM, Karl Wright <
>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Nikita,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You are getting the __nosecurity__
token value transmitted
>>>>>>>>>>>>>>>>> and stored, which means
your Elastic Search setup is probably reasonable.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you give us details
about your pipeline?  What
>>>>>>>>>>>>>>>>> repository connector
is this?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu, Jan 25, 2018
at 1:49 AM, Nikita Ahuja <
>>>>>>>>>>>>>>>>> nikita@smartshore.nl>
wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am using ManifoldCF
2.8.1 and trying to access the
>>>>>>>>>>>>>>>>>> files from FileShare
configured with the Active Directory and get the data
>>>>>>>>>>>>>>>>>> in elastic search
index.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  For this steps followed
are:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - An Authority
Group is created under which Authority
>>>>>>>>>>>>>>>>>>    Connection and
Repository connection are created.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [image: Inline image
3]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - And the elastic
search is created as an Output
>>>>>>>>>>>>>>>>>>    Connector and
a job is created for the connection in which a transformation
>>>>>>>>>>>>>>>>>>    and metadata connector
is also linked in the
>>>>>>>>>>>>>>>>>>    - Which is giving
all the output except the access
>>>>>>>>>>>>>>>>>>    token values.
>>>>>>>>>>>>>>>>>>    - [image: Inline
image 1]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please guide to take
appropriate steps for it so that I
>>>>>>>>>>>>>>>>>> can get the access
tokens in the elastic search index. Hoping for the
>>>>>>>>>>>>>>>>>> solution asap.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks and Regards,
>>>>>>>>>>>>>>>>>> Nikita
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message