chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piergiorgio Lucidi <piergior...@apache.org>
Subject Re: [ManifoldCF] - Content migration - Issue on OpenCMIS Server
Date Tue, 12 Dec 2017 23:36:13 GMT
Hi Florian,

2017-12-12 13:20 GMT+01:00 Florian Müller <fmui@apache.org>:

> Hi Piergiorgio,
>
> OpenCMIS does not cache queries or query results - neither on the client
> nor on the server side.
> All queries are sent to the repository and all query results are
> straightly converted to Java objects.
> I cannot explain the behavior that you are seeing.
>
> Can you describe your setup? Binding, repository vendor, query, etc.
> Is there a (simple) way to reproduce this?
>

we are actually using OpenCMIS InMemory Repository inside the integration
test for both source and target repo, this implementation is included in
the branch CONNECTORS-1356-2.7.1 [1].

The test starts two separate instances of the OpenCMIS InMemory Repository,
it prepares a test area with some sample contents and then starts
ManifoldCF.
During the test we add, change and finally remove contents and for each of
these steps we restart the job.

The connector allow to configure both the bindings but during the
integration tests we are using by default the REST binding.
You can take a look at the integration test included in the CMIS Connectors
module [2].

This module includes three different CMIS connectors:
- CMIS Repository Connector [3] (Read contents and executes queries)
- CMIS Authority Connector (access tokens)
- the new CMIS Output Connector [4] (injection of contents)

After upgrading to the latest OpenCMIS library 1.1.0, the integration tests
included in both CMIS Repository [5] and CMIS Output Connector [6] return
an exception during the last step. This last step is related to the
removeDocument method included in the CMIS Output Connector.

We need to upgrade OpenCMIS because we want to use the existsPath and
deleteByPath introduced in the new version of OpenCMIS but I'm wondering if
these methods are fully supported by the OpenCMIS InMemory Repository.


> The best way to get information about deleted objects is getContentChanges
> (see CMIS spec 1.1, section 2.1.15 "Change Log").

Unfortunately, it is only supported by a few repositories or repositories
> have deactivated it by default. You will always need your query solution as
> a backup.


I definitely should try to take a look at this, but I'm wondering if exists
a way for getting changes related to a specific query and not to all the
repo, with ManifoldCF we are doing this. You can configure different jobs
for indexing or migrating contents using a standard CMIS Query, all the
changes will be tracked by the framework.

For reproducing this issue follow these steps:
1. Checkout of the branch [1]
2. Run the command from the root of the project: ant make-core-deps
3. Run the command from the root of the project: ant make-deps
4. Run the command from /connectors/cmis: mvn clean install

Any feedback will be welcome and thank you again for your support.

Cheers,
PJ

[1] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/

[2] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/

[3] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/cmis/CmisRepositoryConnector.java

[4] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/main/java/org/apache/manifoldcf/agents/output/cmisoutput/CmisOutputConnector.java

[5] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/test/java/org/apache/manifoldcf/crawler/connectors/cmis/tests/APISanityHSQLDBIT.java

[6] -
https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/test/java/org/apache/manifoldcf/agents/output/cmisoutput/tests/APISanityHSQLDBIT.java

[7] - http://chemistry.apache.org/opencmis-inmemory-repository.html



>
>
> - Florian
>
>
>
>
>
> Hi guys,
>>
>> we are adding a new feature in Apache ManifoldCF for migrating contents
>> from any repository supported by the framework (we have connectors for
>> CMIS, Alfresco, Documentum, Sharepoint, FileNet and so on...) to any
>> CMIS-compliant repo.
>>
>> We are finalizing this implementation but during the execution of
>> integration tests, we have found a strange behavior of the OpenCMIS Server
>> [1].
>>
>> After the upgrade of the OpenCMIS libraries to the latest version (we were
>> on 0.13.0), it seems that when we remove a content from the repo, queries
>> continue to return the content. Unfortunately in this way ManifoldCF can't
>> identify which contents should be removed in the target repo because the
>> query return the same results.
>>
>> Do you have changed the behavior for removal in the OpenCMIS Server?
>> Am I missing something?
>>
>> I'm wondering if there is a flag for identifying deleted content or I
>> should take care of the  caching strategy. I think that I need your
>> support
>> for solving this problem.
>>
>> Could you please support me?
>> Thank you.
>>
>> Cheers,
>> PJ
>>
>> [1] - https://issues.apache.org/jira/browse/CONNECTORS-1356
>>
>


-- 
Piergiorgio Lucidi
Open Source Evangelist and Enterprise Information Management Specialist
Mentor / PMC Member / Committer @ Apache Software Foundation
Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco
Author and Technical Reviewer @ Packt Publishing
Technical Advisory Group Member @ Microsoft
Top Community Contributor @ Crafter
Project Leader / Committer @ JBoss
https://www.open4dev.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message