From dev-return-10103-archive-asf-public=cust-asf.ponee.io@chemistry.apache.org Mon May 28 14:27:10 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2A2FD180608 for ; Mon, 28 May 2018 14:27:08 +0200 (CEST) Received: (qmail 91257 invoked by uid 500); 28 May 2018 12:27:08 -0000 Mailing-List: contact dev-help@chemistry.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@chemistry.apache.org Delivered-To: mailing list dev@chemistry.apache.org Received: (qmail 91245 invoked by uid 99); 28 May 2018 12:27:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 May 2018 12:27:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 026F2C004C for ; Mon, 28 May 2018 12:27:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 4.38 X-Spam-Level: **** X-Spam-Status: No, score=4.38 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_BADIPHTTP=2, KAM_EU=0.5, KAM_SHORT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id XBPsdV1RSS5O for ; Mon, 28 May 2018 12:27:02 +0000 (UTC) Received: from mail-ot0-f179.google.com (mail-ot0-f179.google.com [74.125.82.179]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 883635F2C4 for ; Mon, 28 May 2018 12:27:02 +0000 (UTC) Received: by mail-ot0-f179.google.com with SMTP id n1-v6so13278026otf.7 for ; Mon, 28 May 2018 05:27:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=TzxN0Pj4jWETYaphyE/0pdgkC/y4f10B48sPdrVCrMc=; b=u0UqGWYmTgdJOG7ay2zqYhR+l3RoawBoaeGVR+tmETsqLMshFMVN80LZ9eU66yrfT/ T0oQoDG7y8YxSylG9/SPrDbQHTIQYmpMZ2mEw+l4l30++8CjKPdeyLjUL7ATVHVhs5d5 RpxoJX9E1t2kW+Fzk7av5Z8Atbu1F1SubfBUFFgNPyQEdnxnP3YW8/WtEd1Gy5MoV4o6 wS814PbhE79L3ZiT+D4tgteZckev7Ufw94NIxGuelPp/PkERsvSFTAlzV3TZaBxBkmEs KKRJP/pw4ZE5qXIz1pCNsBrCV5HroIGSAGh3r29jWn8myQiLwwAXxqcuXwYSOR/LZVER JKJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=TzxN0Pj4jWETYaphyE/0pdgkC/y4f10B48sPdrVCrMc=; b=Q+f+lRUI9k52oRHH1lde3dOgD/A0Nj81h8uyON9KJJfygFNgHCo9DnJ0KxndrcN+9p dp86NEXcHtRbK4X2/nphdLrt+Li5kQ+6Bo+kh+k+Oxmux6x3EsXPTbBPRPmJwSvjjhIA 0bfo4HBMCUvfArqrAHIdII/EIG6Yl6ORV/que7+yq3Uhc34A6zSl3I+l+GLYjkmKOHm9 E4seEQ0VRRjhtdmdm0RUkWyLwujQOQDqyznl6w5IY8QJ84zQHcasZ00XicQQbLRUMN3I y+aXU4b+ALfmvKwKZ7bWKsXVNvrbDol0nCO2hvWy6pvG7yYKigg4m1nIXOCeMztzSmAq 3QuA== X-Gm-Message-State: ALKqPwezLMZjHlv8kLZVXXg7u1PCB6FWYuwFGjShoClOWcPDNmTDI0WA hiT3yKdo2yvjuGjiwl+11C56MyaQsIYAFSie7cYSxA== X-Google-Smtp-Source: ADUXVKJ0tl+UOBhTjRSz9jye2gdz7/d1kdF9SqDaOXidIWifIq4WkhM184LtSsTsS0whwqaVybav9K5I9u00OQAR9J0= X-Received: by 2002:a9d:2c8:: with SMTP id 66-v6mr7776966otl.28.1527510416164; Mon, 28 May 2018 05:26:56 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a4a:d12c:0:0:0:0:0 with HTTP; Mon, 28 May 2018 05:26:55 -0700 (PDT) In-Reply-To: References: From: Mike IT Expert Date: Mon, 28 May 2018 16:56:55 +0430 Message-ID: Subject: Re: Does cmislib python support multi-threading? To: dev@chemistry.apache.org Content-Type: multipart/alternative; boundary="000000000000bf0db0056d433bad" --000000000000bf0db0056d433bad Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks you so much for quick resposne. Still the same no difference. (files from one thread goes to folder for another thread) I checked out the code like below: git clone git fetch origin pull/12/head git checkout 1073eb533fc3478515f4af3a6f779b63c1b651b2 I used Python 3.6. I don't think it is related to Alfresco, I used the same logic for multi-threading using REST API it works just fine. Thanks, Mike On Mon, May 28, 2018 at 12:26 PM, Mignon, Laurent wrote: > Hi Mike, > > I'm sorry but I've no time to investigate if the problem is on the cmisli= b > side or on the alfresco side. Nevertheless, I recently completely reworke= d > the cmislib library to make it py2 py3 compatible and at the same time I > replaced the dependency on httplib2 by requests. ( > https://github.com/apache/chemistry-cmislib/pull/12) > It could be interesting to make a try with this new version of the librar= y > and see if the problem is also present into this new one. If the problem > comes from cmislib, personally I would prefer to correct it in this new > version and speed up thus its release. > > Regards, > > Laurent Mignon > > > On Sat, May 26, 2018 at 9:42 PM, Mike IT Expert > wrote: > > > Hi Laurent, > > > > Please let me know if there has been update on this. I have added my > > employer just to see there is anything we can do on our side to make th= e > > cmislib thread safe. > > > > Regards, > > M > > > > On Fri, May 25, 2018 at 7:12 PM, Mike IT Expert > > wrote: > > > > > > > > Apparently the exception doesn't occur if I use separate parent folde= r > > for > > > each thread, but the new documents SOMETIMES are created under > incorrect > > > folder (created by other threads). > > > > > > Attached is the source code but I cleaned up the code a little and to= ok > > > out unrelated stuff of mine. > > > > > > Please let me know the outcome .... > > > > > > mike > > > > > > > > > > > > > > > > > > On Fri, May 25, 2018 at 5:59 PM, Mignon, Laurent < > > laurent.mignon@acsone.eu > > > > wrote: > > > > > >> Not sure if it's the root cause but new files are added at same time > > into > > >> the same parent folder. That means that the parent folder is updated > at > > >> same time. > > >> Do you have the same problem if each thread create files in its own > > parent > > >> folder? > > >> > > >> lmignon > > >> > > >> On Fri, May 25, 2018 at 1:52 PM, Mike IT Expert < > mikeitexpert@gmail.com > > > > > >> wrote: > > >> > > >> > You are right but my script does't update the same resource/docume= nt > > >> > simultaneously because each thread uploads new files under new > unique > > >> names > > >> > on the server, in that each thread > > >> > > > >> > *1) creates its own instance of the repository* > > >> > > > >> > * # 2) upload files and update properties* > > >> > * # connect to the server * > > >> > * client =3D cmislib.CmisClient(SERVER_URL, USER, PASSWD)* > > >> > > > >> > * # get default repository* > > >> > * repo =3D client.defaultRepository* > > >> > > > >> > *2) retrieves the target folder's node id* > > >> > > > >> > * # get the target folder to upload new documents into* > > >> > * results =3D repo.query("select * from cmis:folder where cmis:nam= e =3D > > >> '%s'"%( > > >> > TARGET_ALFRESCO_FOLDER, ))* > > >> > * targetFolder =3D results[0]* > > >> > * targetFolder =3D repo.getObject( targetFolder.id )* > > >> > > > >> > *3) for each file (eg. file_i ) in a loop python statement* > > >> > > > >> > > > >> > > > >> > * newDoc =3D targetFolder.createDocument( file_i['docName'], > > contentFile =3D > > >> > inFile ) * > > >> > > > >> > * props =3D {* > > >> > * u'cm:title': file_i[ HDR_TITLE ],* > > >> > * u'cm:description': file_i[ HDR_DESCR ],* > > >> > * }* > > >> > > > >> > > > >> > * newDoc.updateProperties(props)* > > >> > > > >> > > > >> > So what I am doing is that I access the same folder and upload new > > >> > documents using unique names. I am 100% sure the document names (i= e. > > >> > * file_i['docName']*) are unique under the same folder. > > >> > > > >> > It just doesn't seem to me to be any concurrency problem with abov= e > > >> code. > > >> > > > >> > > > >> > Many thanks anyways, > > >> > M > > >> > > > >> > > > >> > On Fri, May 25, 2018 at 3:57 PM, Mignon, Laurent < > > >> laurent.mignon@acsone.eu > > >> > > > > >> > wrote: > > >> > > > >> > > Hi Mike, > > >> > > > > >> > > I'm not aware of a design issue that could prevent to use cmisli= b > > >> python > > >> > > into multi threads. > > >> > > The problem you describe is a normal behaviour when two actors > > (human > > >> or > > >> > > prog) update the same resource at same time into alfresco. > > >> > > > > >> > > Regards, > > >> > > > > >> > > lmignon > > >> > > > > >> > > On Fri, May 25, 2018 at 12:19 PM, Mike IT Expert < > > >> mikeitexpert@gmail.com > > >> > > > > >> > > wrote: > > >> > > > > >> > >> Dear All, > > >> > >> > > >> > >> Apologies in advance for sending this message to dev mailing li= st > > (I > > >> > wish > > >> > >> there was userlist also I wasn't sure it is bug or not.) > > >> > >> > > >> > >> I have started working with cmislib since past week using > Alfresco > > as > > >> > the > > >> > >> local server. > > >> > >> > > >> > >> My goal is to upload documents provided by json rcord list in a > > for > > >> > loop > > >> > >> as you see below *which is part of run-overiden method of > > >> > >> threading.Thread.run method for python 2.7*. > > >> > >> > > >> > >> The main code is attached just in case. > > >> > >> > > >> > >> > > >> > >> > > >> > >> The above code works find using ONE thread only. If I use more > than > > >> one > > >> > >> thread I get the below exception for some records which is odd. > > >> > >> > > >> > >> > > >> > >> File "module_proj_2_parts_1_and_2.py", line 82, in run > > >> > >> newDoc =3D targetFolder.createDocument( docName, contentFil= e =3D > > >> inFile > > >> > ) > > >> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/ > python2. > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 2788, in > > >> > createDocument > > >> > >> contentEncoding) > > >> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/ > python2. > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 1807, in > > >> > createDocument > > >> > >> ATOM_XML_ENTRY_TYPE) > > >> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/ > python2. > > >> > >> 7/site-packages/cmislib/atompub/binding.py", line 188, in post > > >> > >> self._processCommonErrors(resp, url) > > >> > >> File "/home/mike/Desktop/hassan_proj2/.pyenv2/local/lib/ > python2. > > >> > >> 7/site-packages/cmislib/cmis_services.py", line 63, in > > >> > >> _processCommonErrors > > >> > >> raise UpdateConflictException(error[\'status\'], > > >> > >> url)\nUpdateConflictException: Error 409 at > > >> > >> http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/ > > >> > >> versions/1.1/atom/children?id=3D4baddda1-87eb-45eb-8d0b- > > 9958b8942c4f\n > > >> > >> > > >> > >> > > >> > >> I doubt if it is bug ... but if it is please let me know where = I > > can > > >> > >> start to fix this. > > >> > >> > > >> > >> Also, my pip freeze package versions are as below: > > >> > >> > > >> > >> certifi=3D=3D2018.4.16 > > >> > >> chardet=3D=3D3.0.4 > > >> > >> cmislib=3D=3D0.6.0 > > >> > >> httplib2=3D=3D0.11.3 > > >> > >> idna=3D=3D2.6 > > >> > >> iso8601=3D=3D0.1.12 > > >> > >> requests=3D=3D2.18.4 > > >> > >> urllib3=3D=3D1.22 > > >> > >> > > >> > >> I appreciate any clue/comment. > > >> > >> > > >> > >> Best Regards, > > >> > >> Mike > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > >> > > >> > > > > >> > > > > >> > > -- > > >> > > *Laurent Mignon* > > >> > > Senior software engineer > > >> > > > > >> > > Tel: +352 20 21 10 20 32 > > >> > > Fax: +352 20 21 10 21 > > >> > > Email: laurent.mignon@acsone.eu > > >> > > > > >> > > Acsone SA, Succursale de Luxembourg > > >> > > 22, Zone Industrielle > > >> > > =E2=80=8B =E2=80=8B > > >> > > =E2=80=8B| L-8287 Kehlen, Luxembourg > > >> > > TVA LU24733605 > > >> > > =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > > >> > > > > >> > > =E2=80=8B > > >> > > Acsone sa/nv > > >> > > Waterloo Atrium, Dr=C3=A8ve Richelle 167 > > >> > > > >> ail&source=3Dg > > >> > > > > >> > > | B-1410 Waterloo > > >> > > RPM Bruxelles 0835.207.216 RPR Brussel > > >> > > > > >> > > www.acsone.eu > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > > > > > > > > > -- > *Laurent Mignon* > Senior software engineer > > Tel: +352 20 21 10 20 32 > Fax: +352 20 21 10 21 > Email: laurent.mignon@acsone.eu > > Acsone SA, Succursale de Luxembourg > 22, Zone Industrielle > =E2=80=8B =E2=80=8B > =E2=80=8B| L-8287 Kehlen, Luxembourg > TVA LU24733605 > =E2=80=8B | =E2=80=8BRCS B160400=E2=80=8B > > =E2=80=8B > Acsone sa/nv > Waterloo Atrium, Dr=C3=A8ve Richelle 167 > | B-1410 Waterloo > RPM Bruxelles 0835.207.216 RPR Brussel > > www.acsone.eu > --000000000000bf0db0056d433bad--