From dev-return-10112-archive-asf-public=cust-asf.ponee.io@chemistry.apache.org Sun Jun 10 05:54:56 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E6D01180648 for ; Sun, 10 Jun 2018 05:54:54 +0200 (CEST) Received: (qmail 84380 invoked by uid 500); 10 Jun 2018 03:54:53 -0000 Mailing-List: contact dev-help@chemistry.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@chemistry.apache.org Delivered-To: mailing list dev@chemistry.apache.org Received: (qmail 84368 invoked by uid 99); 10 Jun 2018 03:54:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 10 Jun 2018 03:54:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 9FBE118045C for ; Sun, 10 Jun 2018 03:54:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 5.649 X-Spam-Level: ***** X-Spam-Status: No, score=5.649 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_REPLY=1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, KAM_EU=0.5, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id cZ6t2Ffs-Hp1 for ; Sun, 10 Jun 2018 03:54:46 +0000 (UTC) Received: from mail-wr0-f195.google.com (mail-wr0-f195.google.com [209.85.128.195]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id E4C935F205 for ; Sun, 10 Jun 2018 03:54:45 +0000 (UTC) Received: by mail-wr0-f195.google.com with SMTP id k16-v6so16967674wro.0 for ; Sat, 09 Jun 2018 20:54:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=e9/oOxWxFuSRxrW8XFKb/rh9OSlcRBuj6TmJ26G0Fhk=; b=ZmC5UL7qx1tdK7N8RDIrPGsb1BKh2F/mnFMqIYIj/o8ThLJs8lMIHUHMf/Z86MvOSt dJ5RN2VWrCHTYHa3/wg9pnHE1EDP4hzy9cBUkKFrG57I951FcO1D/2e12ASGLJl/rt6h e95FAy/baWrS5qmy3O6v121lO8IbVGThWtVpLV+nx4Vcu8y8CEOyFdmUsrlDJ0bDqDyx 8M2SKUFCRWFsk1tMs3tuI08Z7WvAifnvYsKilZCHp9Lvz5IQxAmVa5zPbcGA27WcOusK 7+E8z29LvZbdhL32agK0zhefdfgM+FpMtJ15OY2tAcnXPwKafAqxnQ7OjoUUkQoNixSa s8CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=e9/oOxWxFuSRxrW8XFKb/rh9OSlcRBuj6TmJ26G0Fhk=; b=g8A1e5bTdg3Ug7z5ppOSmONUiT71RaPvL9zC5OcyUYxLqx7DMk2fpMaDPKpGLnIJcw mL3F/wrrI1O55POYmXUA+We317Ee9ACT5HN7muXnFQm012R5w1QRNqj7AarM3lTtCI7+ 4nKf5qOqzU+5U1QMK2unD7I1n91CP2JSpAwrXOrFawE6LsHtSdXyjnzjZWf/MlzbVyh0 YB2n+k5dv9KPx0P7oi48DM/iy+aChHJLGqwacy65x51aoM8RKeU4xUivTQfJRs9AUeV3 xjMK2V08/2QZq4molv8S/NVgZChph9/dlO5/T/F4ZTRQWsDD3Ezi8BxmJzz4l7+lJIcI 8n7w== X-Gm-Message-State: APt69E3PXRIjQz4bxFVU5oT7NVWdcKeSqscm2pEEYiIKsNdd/JL90VYh LeadtFtrnvMcxzUH0m5NHs5ndFp/DB+1zPfow6A= X-Google-Smtp-Source: ADUXVKI2xP6dQq1Xmo00ltgazQIEASKPSPv8oBoC5j75J2Ao4LfnsFVrDP9fm0g/bC/dZ1ML9ZQ659R3ZBnqZWSRJvY= X-Received: by 2002:adf:f8c2:: with SMTP id f2-v6mr8675365wrq.237.1528602884576; Sat, 09 Jun 2018 20:54:44 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:adf:b07b:0:0:0:0:0 with HTTP; Sat, 9 Jun 2018 20:54:43 -0700 (PDT) In-Reply-To: References: From: Jeff Potts Date: Sat, 9 Jun 2018 22:54:43 -0500 Message-ID: Subject: Re: Does cmislib python support multi-threading? To: dev@chemistry.apache.org Content-Type: multipart/alternative; boundary="000000000000f01c71056e4197be" --000000000000f01c71056e4197be Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable What I am seeing using cmislib 0.6.0 and Python 2.7.12 against Alfresco 201707-GA... If you switch to the browser binding, and you comment out the newDoc.updateProperties(props) call, your code runs successfully. If you run with the browser binding and leave in the newDoc.updateProperties(props) call you get a different problem. If you run with the atompub binding you will get an update conflict. If you add a delay with time.sleep(1) just before the createDocument call the update conflict will go away. Side note: You do not need to re-init client or repo. Those calls are redundant. However, removing them makes no difference on the error condition. Jeff On Tue, May 29, 2018 at 6:25 AM, Mignon, Laurent wrote: > HI > > "from cmislib.browser import BrowserBinding" works in all my projects. > > > On Mon, May 28, 2018 at 8:25 PM, Mike IT Expert > wrote: > > > I tried > > > > client =3D cmislib.CmisClient(SERVER_URL, USER, PASSWD, > > binding=3Dcmislib.browser.BrowserBinding()) > > > > But apparently it couldn't be found : > > > > Exception in thread Thread-15: > > Traceback (most recent call last): > > File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner > > self.run() > > File "cmislib_multi_threading_test_.py", line 40, in run > > client =3D cmislib.CmisClient(SERVER_URL, USER, PASSWD, > > binding=3Dcmislib.browser.BrowserBinding()) > > AttributeError: module 'cmislib' has no attribute 'browser' > > > > Where is BrowserBinding? > > > > > > > > > > > > On Mon, May 28, 2018 at 5:04 PM, Mignon, Laurent < > laurent.mignon@acsone.eu > > > wrote: > > > >> By default the cmislib uses the 'atompub' binding. Can you try with th= e > >> 'browser' binding. > >> > >> from cmislib.browser.binding import BrowserBinding > >> client =3D CmisClient(.., .., binding=3DBrowserBinding()) > >> > >> > >> > >> > >> On Mon, May 28, 2018 at 2:26 PM, Mike IT Expert > > >> wrote: > >> > >> > Thanks you so much for quick resposne. > >> > > >> > Still the same no difference. (files from one thread goes to folder > for > >> > another thread) > >> > > >> > I checked out the code like below: > >> > > >> > git clone > >> > git fetch origin pull/12/head > >> > git checkout 1073eb533fc3478515f4af3a6f779b63c1b651b2 > >> > > >> > > >> > I used Python 3.6. > >> > > >> > I don't think it is related to Alfresco, I used the same logic for > >> > multi-threading using REST API it works just fine. > >> > > >> > Thanks, > >> > Mike > >> > > >> > > >> > > >> > > >> > > >> > On Mon, May 28, 2018 at 12:26 PM, Mignon, Laurent < > >> > laurent.mignon@acsone.eu> > >> > wrote: > >> > > >> > > Hi Mike, > >> > > > >> > > I'm sorry but I've no time to investigate if the problem is on the > >> > cmislib > >> > > side or on the alfresco side. Nevertheless, I recently completely > >> > reworked > >> > > the cmislib library to make it py2 py3 compatible and at the same > >> time I > >> > > replaced the dependency on httplib2 by requests. ( > >> > > https://github.com/apache/chemistry-cmislib/pull/12) > >> > > It could be interesting to make a try with this new version of the > >> > library > >> > > and see if the problem is also present into this new one. If the > >> problem > >> > > comes from cmislib, personally I would prefer to correct it in thi= s > >> new > >> > > version and speed up thus its release. > >> > > > >> > > Regards, > >> > > > >> > > Laurent Mignon > >> > > > >> > > > >> > > On Sat, May 26, 2018 at 9:42 PM, Mike IT Expert < > >> mikeitexpert@gmail.com> > >> > > wrote: > >> > > > >> > > > Hi Laurent, > >> > > > > >> > > > Please let me know if there has been update on this. I have adde= d > my > >> > > > employer just to see there is anything we can do on our side to > make > >> > the > >> > > > cmislib thread safe. > >> > > > > >> > > > Regards, > >> > > > M > >> > > > > >> > > > On Fri, May 25, 2018 at 7:12 PM, Mike IT Expert < > >> > mikeitexpert@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > > >> > > > > Apparently the exception doesn't occur if I use separate paren= t > >> > folder > >> > > > for > >> > > > > each thread, but the new documents SOMETIMES are created under > >> > > incorrect > >> > > > > folder (created by other threads). > >> > > > > > >> > > > > Attached is the source code but I cleaned up the code a little > and > >> > took > >> > > > > out unrelated stuff of mine. > >> > > > > > >> > > > > Please let me know the outcome .... > >> > > > > > >> > > > > mike > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Fri, May 25, 2018 at 5:59 PM, Mignon, Laurent < > >> > > > laurent.mignon@acsone.eu > >> > > > > > wrote: > >> > > > > > >> > > > >> Not sure if it's the root cause but new files are added at sa= me > >> time > >> > > > into > >> > > > >> the same parent folder. That means that the parent folder is > >> updated > >> > > at > >> > > > >> same time. > >> > > > >> Do you have the same problem if each thread create files in i= ts > >> own > >> > > > parent > >> > > > >> folder? > >> > > > >> > >> > > > >> lmignon > >> > > > >> > >> > > > >> On Fri, May 25, 2018 at 1:52 PM, Mike IT Expert < > >> > > mikeitexpert@gmail.com > >> > > > > > >> > > > >> wrote: > >> > > > >> > >> > > > >> > You are right but my script does't update the same > >> > resource/document > >> > > > >> > simultaneously because each thread uploads new files under > new > >> > > unique > >> > > > >> names > >> > > > >> > on the server, in that each thread > >> > > > >> > > >> > > > >> > *1) creates its own instance of the repository* > >> > > > >> > > >> > > > >> > * # 2) upload files and update properties* > >> > > > >> > * # connect to the server * > >> > > > >> > * client =3D cmislib.CmisClient(SERVER_URL, USER, PASSWD)* > >> > > > >> > > >> > > > >> > * # get default repository* > >> > > > >> > * repo =3D client.defaultRepository* > >> > > > >> > > >> > > > >> > *2) retrieves the target folder's node id* > >> > > > >> > > >> > > > >> > * # get the target folder to upload new documents into* > >> > > > >> > * results =3D repo.query("select * from cmis:folder where > >> cmis:name > >> > =3D > >> > > > >> '%s'"%( > >> > > > >> > TARGET_ALFRESCO_FOLDER, ))* > >> > > > >> > * targetFolder =3D results[0]* > >> > > > >> > * targetFolder =3D repo.getObject( targetFolder.id )* > >> > > > >> > > >> > > > >> > *3) for each file (eg. file_i ) in a loop python statement* > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > * newDoc =3D targetFolder.createDocument( file_i['docName']= , > >> > > > contentFile =3D > >> > > > >> > inFile ) * > >> > > > >> > > >> > > > >> > * props =3D {* > >> > > > >> > * u'cm:title': file_i[ HDR_TITLE ],* > >> > > > >> > * u'cm:description': file_i[ HDR_DESCR ],* > >> > > > >> > * }* > >> > > > >> > > >> > > > >> > > >> > > > >> > * newDoc.updateProperties(props)* > >> > > > >> > > >> > > > >> > > >> > > > >> > So what I am doing is that I access the same folder and > upload > >> new > >> > > > >> > documents using unique names. I am 100% sure the document > names > >> > (ie. > >> > > > >> > * file_i['docName']*) are unique under the same folder. > >> > > > >> > > >> > > > >> > It just doesn't seem to me to be any concurrency problem wi= th > >> > above > >> > > > >> code. > >> > > > >> > > >> > > > >> > > >> > > > >> > Many thanks anyways, > >> > > > >> > M > >> > > > >> > > >> > > > >> > > >> > > > >> > On Fri, May 25, 2018 at 3:57 PM, Mignon, Laurent < > >> > > > >> laurent.mignon@acsone.eu > >> > > > >> > > > >> > > > >> > wrote: > >> > > > >> > > >> > > > >> > > Hi Mike, > >> > > > >> > > > >> > > > >> > > I'm not aware of a design issue that could prevent to use > >> > cmislib > >> > > > >> python > >> > > > >> > > into multi threads. > >> > > > >> > > The problem you describe is a normal behaviour when two > >> actors > >> > > > (human > >> > > > >> or > >> > > > >> > > prog) update the same resource at same time into alfresco= . > >> > > > >> > > > >> > > > >> > > Regards, > >> > > > >> > > > >> > > > >> > > lmignon > >> > > > >> > > > >> > > > >> > > On Fri, May 25, 2018 at 12:19 PM, Mike IT Expert < > >> > > > >> mikeitexpert@gmail.com > >> > > > >> > > > >> > > > >> > > wrote: > >> > > > >> > > > >> > > > >> > >> Dear All, > >> > > > >> > >> > >> > > > >> > >> Apologies in advance for sending this message to dev > mailing > >> > list > >> > > > (I > >> > > > >> > wish > >> > > > >> > >> there was userlist also I wasn't sure it is bug or not.) > >> > > > >> > >> > >> > > > >> > >> I have started working with cmislib since past week usin= g > >> > > Alfresco > >> > > > as > >> > > > >> > the > >> > > > >> > >> local server. > >> > > > >> > >> > >> > > > >> > >> My goal is to upload documents provided by json rcord li= st > >> in a > >> > > > for > >> > > > >> > loop > >> > > > >> > >> as you see below *which is part of run-overiden method o= f > >> > > > >> > >> threading.Thread.run method for python 2.7*. > >> > > > >> > >> > >> > > > >> > >> The main code is attached just in case. > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> The above code works find using ONE thread only. If I us= e > >> more > >> > > than > >> > > > >> one > >> > > > >> > >> thread I get the below exception for some records which = is > >> odd. > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> File "module_proj_2_parts_1_and_2.py", line 82, in run > >> > > > >> > >> newDoc =3D targetFolder.createDocument( docName, > >> contentFile > >> > =3D > >> > > > >> inFile > >> > > > >> > ) > >> > > > >> > >> File "/home/mike/Desktop/hassan_ > proj2/.pyenv2/local/lib/ > >> > > python2. > >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 2788, > in > >> > > > >> > createDocument > >> > > > >> > >> contentEncoding) > >> > > > >> > >> File "/home/mike/Desktop/hassan_ > proj2/.pyenv2/local/lib/ > >> > > python2. > >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 1807, > in > >> > > > >> > createDocument > >> > > > >> > >> ATOM_XML_ENTRY_TYPE) > >> > > > >> > >> File "/home/mike/Desktop/hassan_ > proj2/.pyenv2/local/lib/ > >> > > python2. > >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 188, i= n > >> post > >> > > > >> > >> self._processCommonErrors(resp, url) > >> > > > >> > >> File "/home/mike/Desktop/hassan_ > proj2/.pyenv2/local/lib/ > >> > > python2. > >> > > > >> > >> 7/site-packages/cmislib/cmis_services.py", line 63, in > >> > > > >> > >> _processCommonErrors > >> > > > >> > >> raise UpdateConflictException(error[\'status\'], > >> > > > >> > >> url)\nUpdateConflictException: Error 409 at > >> > > > >> > >> http://127.0.0.1:8080/alfresco/api/-default-/public/cmis= / > >> > > > >> > >> versions/1.1/atom/children?id=3D4baddda1-87eb-45eb-8d0b- > >> > > > 9958b8942c4f\n > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> I doubt if it is bug ... but if it is please let me know > >> where > >> > I > >> > > > can > >> > > > >> > >> start to fix this. > >> > > > >> > >> > >> > > > >> > >> Also, my pip freeze package versions are as below: > >> > > > >> > >> > >> > > > >> > >> certifi=3D=3D2018.4.16 > >> > > > >> > >> chardet=3D=3D3.0.4 > >> > > > >> > >> cmislib=3D=3D0.6.0 > >> > > > >> > >> httplib2=3D=3D0.11.3 > >> > > > >> > >> idna=3D=3D2.6 > >> > > > >> > >> iso8601=3D=3D0.1.12 > >> > > > >> > >> requests=3D=3D2.18.4 > >> > > > >> > >> urllib3=3D=3D1.22 > >> > > > >> > >> > >> > > > >> > >> I appreciate any clue/comment. > >> > > > >> > >> > >> > > > >> > >> Best Regards, > >> > > > >> > >> Mike > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > >> > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > -- > >> > > > >> > > *Laurent Mignon* > >> > > > >> > > Senior software engineer > >> > > > >> > > > >> > > > >> > > Tel: +352 20 21 10 20 32 > >> > > > >> > > Fax: +352 20 21 10 21 > >> > > > >> > > Email: laurent.mignon@acsone.eu > >> > > > >> > > > >> > > > >> > > Acsone SA, Succursale de Luxembourg > >> > > > >> > > 22, Zone Industrielle > >> > > > >> > > =E2=80=8B =E2=80=8B > >> > > > >> > > =E2=80=8B| L-8287 Kehlen, Luxembourg > >> > > > >> > > TVA LU24733605 > >> > > > >> > > =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > >> > > > >> > > > >> > > > >> > > =E2=80=8B > >> > > > >> > > Acsone sa/nv > >> > > > >> > > Waterloo Atrium, Dr=C3=A8ve Richelle 167 > >> gmail&source=3Dg> > >> > > > >> > > gm > >> > > > >> ail&source=3Dg > >> > > > >> > > > >> > > > >> > > | B-1410 Waterloo > >> > > > >> > > RPM Bruxelles 0835.207.216 RPR Brussel > >> > > > >> > > > >> > > > >> > > www.acsone.eu > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > *Laurent Mignon* > >> > > Senior software engineer > >> > > > >> > > Tel: +352 20 21 10 20 32 > >> > > Fax: +352 20 21 10 21 > >> > > Email: laurent.mignon@acsone.eu > >> > > > >> > > Acsone SA, Succursale de Luxembourg > >> > > 22, Zone Industrielle > >> > > =E2=80=8B =E2=80=8B > >> > > =E2=80=8B| L-8287 Kehlen, Luxembourg > >> > > TVA LU24733605 > >> > > =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > >> > > > >> > > =E2=80=8B > >> > > Acsone sa/nv > >> > > Waterloo Atrium, Dr=C3=A8ve Richelle 167 > >> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=3Dgmail&source=3Dg> > >> > > | B-1410 Waterloo > >> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=3Dgmail&source=3Dg> > >> > > RPM Bruxelles 0835.207.216 RPR Brussel > >> > > > >> > > www.acsone.eu > >> > > > >> > > >> > >> > >> > >> -- > >> *Laurent Mignon* > >> Senior software engineer > >> > >> Tel: +352 20 21 10 20 32 > >> Fax: +352 20 21 10 21 > >> Email: laurent.mignon@acsone.eu > >> > >> Acsone SA, Succursale de Luxembourg > >> 22, Zone Industrielle > >> =E2=80=8B =E2=80=8B > >> =E2=80=8B| L-8287 Kehlen, Luxembourg > >> TVA LU24733605 > >> =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > >> > >> =E2=80=8B > >> Acsone sa/nv > >> Waterloo Atrium, Dr=C3=A8ve Richelle 167 > >> 0A%C2%A0%7C+B-1410+Waterloo&entry=3Dgmail&source=3Dg> > >> | B-1410 Waterloo > >> RPM Bruxelles 0835.207.216 RPR Brussel > >> > >> www.acsone.eu > >> > > > > > > > -- > *Laurent Mignon* > Senior software engineer > > Tel: +352 20 21 10 20 32 > Fax: +352 20 21 10 21 > Email: laurent.mignon@acsone.eu > > Acsone SA, Succursale de Luxembourg > 22, Zone Industrielle > =E2=80=8B =E2=80=8B > =E2=80=8B| L-8287 Kehlen, Luxembourg > TVA LU24733605 > =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > > =E2=80=8B > Acsone sa/nv > Waterloo Atrium, Dr=C3=A8ve Richelle 167 > | B-1410 Waterloo > RPM Bruxelles 0835.207.216 RPR Brussel > > www.acsone.eu > --000000000000f01c71056e4197be--