chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Potts <jeffpott...@gmail.com>
Subject Re: Does cmislib python support multi-threading?
Date Sun, 10 Jun 2018 03:54:43 GMT
What I am seeing using cmislib 0.6.0 and Python 2.7.12 against Alfresco
201707-GA...

If you switch to the browser binding, and you comment out the
newDoc.updateProperties(props) call, your code runs successfully.

If you run with the browser binding and leave in the
newDoc.updateProperties(props) call you get a different problem.

If you run with the atompub binding you will get an update conflict.

If you add a delay with time.sleep(1) just before the createDocument call
the update conflict will go away.

Side note: You do not need to re-init client or repo. Those calls are
redundant. However, removing them makes no difference on the error
condition.

Jeff

On Tue, May 29, 2018 at 6:25 AM, Mignon, Laurent <laurent.mignon@acsone.eu>
wrote:

> HI
>
> "from cmislib.browser import BrowserBinding" works in all my projects.
>
>
> On Mon, May 28, 2018 at 8:25 PM, Mike IT Expert <mikeitexpert@gmail.com>
> wrote:
>
> > I tried
> >
> > client =  cmislib.CmisClient(SERVER_URL, USER, PASSWD,
> > binding=cmislib.browser.BrowserBinding())
> >
> > But apparently it couldn't be found :
> >
> > Exception in thread Thread-15:
> > Traceback (most recent call last):
> >   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
> >     self.run()
> >   File "cmislib_multi_threading_test_.py", line 40, in run
> >     client =  cmislib.CmisClient(SERVER_URL, USER, PASSWD,
> > binding=cmislib.browser.BrowserBinding())
> > AttributeError: module 'cmislib' has no attribute 'browser'
> >
> > Where is BrowserBinding?
> >
> >
> >
> >
> >
> > On Mon, May 28, 2018 at 5:04 PM, Mignon, Laurent <
> laurent.mignon@acsone.eu
> > > wrote:
> >
> >> By default the cmislib uses the 'atompub' binding. Can you try with the
> >> 'browser' binding.
> >>
> >> from cmislib.browser.binding import BrowserBinding
> >> client  = CmisClient(.., .., binding=BrowserBinding())
> >>
> >>
> >>
> >>
> >> On Mon, May 28, 2018 at 2:26 PM, Mike IT Expert <mikeitexpert@gmail.com
> >
> >> wrote:
> >>
> >> > Thanks you so much for quick resposne.
> >> >
> >> > Still the same no difference. (files from one thread goes to folder
> for
> >> > another thread)
> >> >
> >> > I checked out the code like below:
> >> >
> >> > git clone
> >> > git fetch origin pull/12/head
> >> > git checkout 1073eb533fc3478515f4af3a6f779b63c1b651b2
> >> >
> >> >
> >> > I used Python 3.6.
> >> >
> >> > I don't think it is related to Alfresco, I used the same logic for
> >> > multi-threading using REST API it works just fine.
> >> >
> >> > Thanks,
> >> > Mike
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Mon, May 28, 2018 at 12:26 PM, Mignon, Laurent <
> >> > laurent.mignon@acsone.eu>
> >> > wrote:
> >> >
> >> > > Hi Mike,
> >> > >
> >> > > I'm sorry but I've no time to investigate if the problem is on the
> >> > cmislib
> >> > > side or on the alfresco side. Nevertheless, I recently completely
> >> > reworked
> >> > > the cmislib library to make it py2 py3 compatible and at the same
> >> time I
> >> > > replaced the dependency on httplib2 by requests. (
> >> > > https://github.com/apache/chemistry-cmislib/pull/12)
> >> > > It could be interesting to make a try with this new version of the
> >> > library
> >> > > and see if the problem is also present into this new one. If the
> >> problem
> >> > > comes from cmislib, personally I would prefer to correct it in this
> >> new
> >> > > version and speed up thus its release.
> >> > >
> >> > > Regards,
> >> > >
> >> > > Laurent Mignon
> >> > >
> >> > >
> >> > > On Sat, May 26, 2018 at 9:42 PM, Mike IT Expert <
> >> mikeitexpert@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Laurent,
> >> > > >
> >> > > > Please let me know if there has been update on this. I have added
> my
> >> > > > employer just to see there is anything we can do on our side
to
> make
> >> > the
> >> > > > cmislib thread safe.
> >> > > >
> >> > > > Regards,
> >> > > > M
> >> > > >
> >> > > > On Fri, May 25, 2018 at 7:12 PM, Mike IT Expert <
> >> > mikeitexpert@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > >
> >> > > > > Apparently the exception doesn't occur if I use separate
parent
> >> > folder
> >> > > > for
> >> > > > > each thread, but the new documents SOMETIMES are created
under
> >> > > incorrect
> >> > > > > folder (created by other threads).
> >> > > > >
> >> > > > > Attached is the source code but I cleaned up the code a
little
> and
> >> > took
> >> > > > > out unrelated stuff of mine.
> >> > > > >
> >> > > > > Please let me know the outcome ....
> >> > > > >
> >> > > > > mike
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Fri, May 25, 2018 at 5:59 PM, Mignon, Laurent <
> >> > > > laurent.mignon@acsone.eu
> >> > > > > > wrote:
> >> > > > >
> >> > > > >> Not sure if it's the root cause but new files are added
at same
> >> time
> >> > > > into
> >> > > > >> the same parent folder. That means that the parent folder
is
> >> updated
> >> > > at
> >> > > > >> same time.
> >> > > > >> Do you have the same problem if each thread create files
in its
> >> own
> >> > > > parent
> >> > > > >> folder?
> >> > > > >>
> >> > > > >> lmignon
> >> > > > >>
> >> > > > >> On Fri, May 25, 2018 at 1:52 PM, Mike IT Expert <
> >> > > mikeitexpert@gmail.com
> >> > > > >
> >> > > > >> wrote:
> >> > > > >>
> >> > > > >> > You are right but my script does't update the same
> >> > resource/document
> >> > > > >> > simultaneously because each thread uploads new
files under
> new
> >> > > unique
> >> > > > >> names
> >> > > > >> > on the server, in that each thread
> >> > > > >> >
> >> > > > >> > *1) creates its own instance of the repository*
> >> > > > >> >
> >> > > > >> > * # 2) upload files and update properties*
> >> > > > >> > * # connect to the server *
> >> > > > >> > * client =  cmislib.CmisClient(SERVER_URL, USER,
PASSWD)*
> >> > > > >> >
> >> > > > >> > * # get default repository*
> >> > > > >> > * repo = client.defaultRepository*
> >> > > > >> >
> >> > > > >> > *2) retrieves the target folder's node id*
> >> > > > >> >
> >> > > > >> > * # get the target folder to upload new documents
into*
> >> > > > >> > * results = repo.query("select * from cmis:folder
where
> >> cmis:name
> >> > =
> >> > > > >> '%s'"%(
> >> > > > >> > TARGET_ALFRESCO_FOLDER, ))*
> >> > > > >> > * targetFolder = results[0]*
> >> > > > >> > * targetFolder = repo.getObject( targetFolder.id
)*
> >> > > > >> >
> >> > > > >> > *3) for each file (eg. file_i ) in a loop python
statement*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > * newDoc = targetFolder.createDocument( file_i['docName'],
> >> > > > contentFile =
> >> > > > >> > inFile ) *
> >> > > > >> >
> >> > > > >> > * props = {*
> >> > > > >> > * u'cm:title': file_i[ HDR_TITLE ],*
> >> > > > >> > * u'cm:description': file_i[ HDR_DESCR ],*
> >> > > > >> > * }*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > * newDoc.updateProperties(props)*
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > So what I am doing is that I access the same folder
and
> upload
> >> new
> >> > > > >> > documents using unique names. I am 100% sure the
document
> names
> >> > (ie.
> >> > > > >> > * file_i['docName']*) are unique under the same
folder.
> >> > > > >> >
> >> > > > >> > It just doesn't seem to me to be any concurrency
problem with
> >> > above
> >> > > > >> code.
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Many thanks anyways,
> >> > > > >> > M
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > On Fri, May 25, 2018 at 3:57 PM, Mignon, Laurent
<
> >> > > > >> laurent.mignon@acsone.eu
> >> > > > >> > >
> >> > > > >> > wrote:
> >> > > > >> >
> >> > > > >> > > Hi Mike,
> >> > > > >> > >
> >> > > > >> > > I'm not aware of a design issue that could
prevent to use
> >> > cmislib
> >> > > > >> python
> >> > > > >> > > into multi threads.
> >> > > > >> > > The problem you describe is a normal behaviour
when two
> >> actors
> >> > > > (human
> >> > > > >> or
> >> > > > >> > > prog) update the same resource at same time
into alfresco.
> >> > > > >> > >
> >> > > > >> > > Regards,
> >> > > > >> > >
> >> > > > >> > > lmignon
> >> > > > >> > >
> >> > > > >> > > On Fri, May 25, 2018 at 12:19 PM, Mike IT
Expert <
> >> > > > >> mikeitexpert@gmail.com
> >> > > > >> > >
> >> > > > >> > > wrote:
> >> > > > >> > >
> >> > > > >> > >> Dear All,
> >> > > > >> > >>
> >> > > > >> > >> Apologies in advance for sending this
message to dev
> mailing
> >> > list
> >> > > > (I
> >> > > > >> > wish
> >> > > > >> > >> there was userlist also I wasn't sure
it is bug or not.)
> >> > > > >> > >>
> >> > > > >> > >> I have started working with cmislib since
past week using
> >> > > Alfresco
> >> > > > as
> >> > > > >> > the
> >> > > > >> > >> local server.
> >> > > > >> > >>
> >> > > > >> > >> My goal is to upload documents provided
by json rcord list
> >> in a
> >> > > > for
> >> > > > >> > loop
> >> > > > >> > >> as you see below *which is part of run-overiden
method of
> >> > > > >> > >> threading.Thread.run method for python
2.7*.
> >> > > > >> > >>
> >> > > > >> > >> The main code is attached just in case.
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> The above code works find using ONE thread
only. If I use
> >> more
> >> > > than
> >> > > > >> one
> >> > > > >> > >> thread I get the below exception for some
records which is
> >> odd.
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> File "module_proj_2_parts_1_and_2.py",
line 82, in run
> >> > > > >> > >>     newDoc = targetFolder.createDocument(
docName,
> >> contentFile
> >> > =
> >> > > > >> inFile
> >> > > > >> > )
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py",
line 2788,
> in
> >> > > > >> > createDocument
> >> > > > >> > >>     contentEncoding)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py",
line 1807,
> in
> >> > > > >> > createDocument
> >> > > > >> > >>     ATOM_XML_ENTRY_TYPE)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/atompub/binding.py",
line 188, in
> >> post
> >> > > > >> > >>     self._processCommonErrors(resp, url)
> >> > > > >> > >>   File "/home/mike/Desktop/hassan_
> proj2/.pyenv2/local/lib/
> >> > > python2.
> >> > > > >> > >> 7/site-packages/cmislib/cmis_services.py",
line 63, in
> >> > > > >> > >> _processCommonErrors
> >> > > > >> > >>     raise UpdateConflictException(error[\'status\'],
> >> > > > >> > >> url)\nUpdateConflictException: Error 409
at
> >> > > > >> > >> http://127.0.0.1:8080/alfresco/api/-default-/public/cmis/
> >> > > > >> > >> versions/1.1/atom/children?id=4baddda1-87eb-45eb-8d0b-
> >> > > > 9958b8942c4f\n
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> I doubt if it is bug ... but if it is
please let me know
> >> where
> >> > I
> >> > > > can
> >> > > > >> > >> start to fix this.
> >> > > > >> > >>
> >> > > > >> > >> Also, my pip freeze package versions are
as below:
> >> > > > >> > >>
> >> > > > >> > >> certifi==2018.4.16
> >> > > > >> > >> chardet==3.0.4
> >> > > > >> > >> cmislib==0.6.0
> >> > > > >> > >> httplib2==0.11.3
> >> > > > >> > >> idna==2.6
> >> > > > >> > >> iso8601==0.1.12
> >> > > > >> > >> requests==2.18.4
> >> > > > >> > >> urllib3==1.22
> >> > > > >> > >>
> >> > > > >> > >> I appreciate any clue/comment.
> >> > > > >> > >>
> >> > > > >> > >> Best Regards,
> >> > > > >> > >> Mike
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > > --
> >> > > > >> > > *Laurent Mignon*
> >> > > > >> > > Senior software engineer
> >> > > > >> > >
> >> > > > >> > > Tel: +352 20 21 10 20 32
> >> > > > >> > > Fax: +352 20 21 10 21
> >> > > > >> > > Email: laurent.mignon@acsone.eu
> >> > > > >> > >
> >> > > > >> > > Acsone SA, Succursale de Luxembourg
> >> > > > >> > > 22, Zone Industrielle
> >> > > > >> > > ​ ​
> >> > > > >> > > ​| L-8287 Kehlen, Luxembourg
> >> > > > >> > > TVA LU24733605
> >> > > > >> > > ​ | ​RCS B160400​
> >> > > > >> > >
> >> > > > >> > > ​
> >> > > > >> > > Acsone sa/nv
> >> > > > >> > > Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167&entry=
> gmail&source=g>
> >> > > > >> > > <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167&entry=
> gm
> >> > > > >> ail&source=g
> >> > > > >> > >
> >> > > > >> > >  | B-1410 Waterloo
> >> > > > >> > > RPM Bruxelles 0835.207.216 RPR Brussel
> >> > > > >> > >
> >> > > > >> > > www.acsone.eu
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > *Laurent Mignon*
> >> > > Senior software engineer
> >> > >
> >> > > Tel: +352 20 21 10 20 32
> >> > > Fax: +352 20 21 10 21
> >> > > Email: laurent.mignon@acsone.eu
> >> > >
> >> > > Acsone SA, Succursale de Luxembourg
> >> > > 22, Zone Industrielle
> >> > > ​ ​
> >> > > ​| L-8287 Kehlen, Luxembourg
> >> > > TVA LU24733605
> >> > > ​ | ​RCS B160400​
> >> > >
> >> > > ​
> >> > > Acsone sa/nv
> >> > > Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=gmail&source=g>
> >> > >  | B-1410 Waterloo
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A+%C2%A0+%7C+B-1410+Waterloo&entry=gmail&source=g>
> >> > > RPM Bruxelles 0835.207.216 RPR Brussel
> >> > >
> >> > > www.acsone.eu
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> *Laurent Mignon*
> >> Senior software engineer
> >>
> >> Tel: +352 20 21 10 20 32
> >> Fax: +352 20 21 10 21
> >> Email: laurent.mignon@acsone.eu
> >>
> >> Acsone SA, Succursale de Luxembourg
> >> 22, Zone Industrielle
> >> ​ ​
> >> ​| L-8287 Kehlen, Luxembourg
> >> TVA LU24733605
> >> ​ | ​RCS B160400​
> >>
> >> ​
> >> Acsone sa/nv
> >> Waterloo Atrium, Drève Richelle 167
> >> <https://maps.google.com/?q=Dr%C3%A8ve+Richelle+167+%0D%
> 0A%C2%A0%7C+B-1410+Waterloo&entry=gmail&source=g>
> >>  | B-1410 Waterloo
> >> RPM Bruxelles 0835.207.216 RPR Brussel
> >>
> >> www.acsone.eu
> >>
> >
> >
>
>
> --
> *Laurent Mignon*
> Senior software engineer
>
> Tel: +352 20 21 10 20 32
> Fax: +352 20 21 10 21
> Email: laurent.mignon@acsone.eu
>
> Acsone SA, Succursale de Luxembourg
> 22, Zone Industrielle
> ​ ​
> ​| L-8287 Kehlen, Luxembourg
> TVA LU24733605
> ​ | ​RCS B160400​
>
> ​
> Acsone sa/nv
> Waterloo Atrium, Drève Richelle 167
>  | B-1410 Waterloo
> RPM Bruxelles 0835.207.216 RPR Brussel
>
> www.acsone.eu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message