chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Huebel, Jens" <j.hue...@sap.com>
Subject Re: [ManifoldCF] - Content migration - Issue on OpenCMIS Server
Date Wed, 13 Dec 2017 08:46:38 GMT
Hi Piergiorgi,

It is hard to find the root cause for your behaviour. Neither on the client side nor in the
InMemory server have been significant changes in the last release which could explain this.
Can this be a threading/parallelism issue somehow? Are you sure that the instance of the InMemory
server is not restarted (this will reset the whole state as there is no persistence)?Do you
use the same Java version?

I would recommend testing this with a different CMIS server. Can you run your tests against
the File Share repository or some other production ready server (e.g. Alfresco)?
Another option would be to set the log-level of the opencmis client package to DEBUG. In this
case we can trace the requests, which might help to investigate.

Best regards
Jens


On 13.12.17, 00:36, "Piergiorgio Lucidi" <piergiorgio@apache.org> wrote:

    Hi Florian,
    
    2017-12-12 13:20 GMT+01:00 Florian Müller <fmui@apache.org>:
    
    > Hi Piergiorgio,
    >
    > OpenCMIS does not cache queries or query results - neither on the client
    > nor on the server side.
    > All queries are sent to the repository and all query results are
    > straightly converted to Java objects.
    > I cannot explain the behavior that you are seeing.
    >
    > Can you describe your setup? Binding, repository vendor, query, etc.
    > Is there a (simple) way to reproduce this?
    >
    
    we are actually using OpenCMIS InMemory Repository inside the integration
    test for both source and target repo, this implementation is included in
    the branch CONNECTORS-1356-2.7.1 [1].
    
    The test starts two separate instances of the OpenCMIS InMemory Repository,
    it prepares a test area with some sample contents and then starts
    ManifoldCF.
    During the test we add, change and finally remove contents and for each of
    these steps we restart the job.
    
    The connector allow to configure both the bindings but during the
    integration tests we are using by default the REST binding.
    You can take a look at the integration test included in the CMIS Connectors
    module [2].
    
    This module includes three different CMIS connectors:
    - CMIS Repository Connector [3] (Read contents and executes queries)
    - CMIS Authority Connector (access tokens)
    - the new CMIS Output Connector [4] (injection of contents)
    
    After upgrading to the latest OpenCMIS library 1.1.0, the integration tests
    included in both CMIS Repository [5] and CMIS Output Connector [6] return
    an exception during the last step. This last step is related to the
    removeDocument method included in the CMIS Output Connector.
    
    We need to upgrade OpenCMIS because we want to use the existsPath and
    deleteByPath introduced in the new version of OpenCMIS but I'm wondering if
    these methods are fully supported by the OpenCMIS InMemory Repository.
    
    
    > The best way to get information about deleted objects is getContentChanges
    > (see CMIS spec 1.1, section 2.1.15 "Change Log").
    
    Unfortunately, it is only supported by a few repositories or repositories
    > have deactivated it by default. You will always need your query solution as
    > a backup.
    
    
    I definitely should try to take a look at this, but I'm wondering if exists
    a way for getting changes related to a specific query and not to all the
    repo, with ManifoldCF we are doing this. You can configure different jobs
    for indexing or migrating contents using a standard CMIS Query, all the
    changes will be tracked by the framework.
    
    For reproducing this issue follow these steps:
    1. Checkout of the branch [1]
    2. Run the command from the root of the project: ant make-core-deps
    3. Run the command from the root of the project: ant make-deps
    4. Run the command from /connectors/cmis: mvn clean install
    
    Any feedback will be welcome and thank you again for your support.
    
    Cheers,
    PJ
    
    [1] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/
    
    [2] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/
    
    [3] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/main/java/org/apache/manifoldcf/crawler/connectors/cmis/CmisRepositoryConnector.java
    
    [4] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/main/java/org/apache/manifoldcf/agents/output/cmisoutput/CmisOutputConnector.java
    
    [5] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/test/java/org/apache/manifoldcf/crawler/connectors/cmis/tests/APISanityHSQLDBIT.java
    
    [6] -
    https://svn.apache.org/repos/asf/manifoldcf/branches/CONNECTORS-1356-2.7.1/connectors/cmis/connector/src/test/java/org/apache/manifoldcf/agents/output/cmisoutput/tests/APISanityHSQLDBIT.java
    
    [7] - http://chemistry.apache.org/opencmis-inmemory-repository.html
    
    
    
    >
    >
    > - Florian
    >
    >
    >
    >
    >
    > Hi guys,
    >>
    >> we are adding a new feature in Apache ManifoldCF for migrating contents
    >> from any repository supported by the framework (we have connectors for
    >> CMIS, Alfresco, Documentum, Sharepoint, FileNet and so on...) to any
    >> CMIS-compliant repo.
    >>
    >> We are finalizing this implementation but during the execution of
    >> integration tests, we have found a strange behavior of the OpenCMIS Server
    >> [1].
    >>
    >> After the upgrade of the OpenCMIS libraries to the latest version (we were
    >> on 0.13.0), it seems that when we remove a content from the repo, queries
    >> continue to return the content. Unfortunately in this way ManifoldCF can't
    >> identify which contents should be removed in the target repo because the
    >> query return the same results.
    >>
    >> Do you have changed the behavior for removal in the OpenCMIS Server?
    >> Am I missing something?
    >>
    >> I'm wondering if there is a flag for identifying deleted content or I
    >> should take care of the  caching strategy. I think that I need your
    >> support
    >> for solving this problem.
    >>
    >> Could you please support me?
    >> Thank you.
    >>
    >> Cheers,
    >> PJ
    >>
    >> [1] - https://issues.apache.org/jira/browse/CONNECTORS-1356
    >>
    >
    
    
    -- 
    Piergiorgio Lucidi
    Open Source Evangelist and Enterprise Information Management Specialist
    Mentor / PMC Member / Committer @ Apache Software Foundation
    Community Star / Wiki Gardener / Global Forum Moderator @ Alfresco
    Author and Technical Reviewer @ Packt Publishing
    Technical Advisory Group Member @ Microsoft
    Top Community Contributor @ Crafter
    Project Leader / Committer @ JBoss
    https://www.open4dev.com
    

Mime
View raw message