chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Inouye, Brian" <Brian.Ino...@xerox.com>
Subject RE: create service with a large content stream causes a "Limit exceeded!" exception
Date Thu, 03 Apr 2014 16:55:27 GMT
Hi Florian,

Yes, that sounds reasonable.  The Client generating the data is an iOS Mobile App going over
WiFi so it's much slower than going over Ethernet.  I typically get the "Limit exceeded" exception
about three minutes after the Client starts sending the content stream.

... Brian ...

-----Original Message-----
From: Florian Müller [mailto:fmui@apache.org] 
Sent: Thursday, April 03, 2014 9:50 AM
To: Inouye, Brian
Cc: dev@chemistry.apache.org
Subject: RE: create service with a large content stream causes a "Limit exceeded!" exception

 Hi Brian,

 I might have found the cause of this issue. I cannot exactly reproduce  it, but I can get
the XML parser into the same state that your stack  trace shows.
 It looks like your application doesn't send the content fast enough. I  have added a delay
of 15 seconds after the first base64 bytes have been  written. When the parser retrieves the
next bytes, it tries to get as  many bytes as it can. Unfortunately, it doesn't return the
control to  the OpenCMIS code before the XML limit is reached and deduced from the  CappedInputStream
counter.

 Does this sound reasonable? How long does it take from starting the  upload to the error
message in the application?


 - Florian



> Hi Florian,
>
> I suspect the reason the Workbench is able to upload without
> triggering the "Limit exceeded!" exception is that it is using 
> chunked
> transfer encoding, whereas the Mobile App is not.  Can the Workbench
> be configured to not do chunked encoding so we can see if that is the
> culprit?
>
> Here is the request coming from the Workbench:
>
> POST
> 
> /docushare/ds_mobile_connector/atom/docushare/children?id=Collection-14&versioningState=major
> HTTP/1.1
> User-Agent: Apache Chemistry OpenCMIS/0.8.0
> Content-Type: application/atom+xml;type=entry
> Authorization: Basic <confidential>
> Accept-Encoding: gzip,deflate
> Cache-Control: no-cache
> Pragma: no-cache
> Host: <confidential>
> Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
> Connection: keep-alive
> Transfer-Encoding: chunked
>
> fff7
> <?xml version='1.0' encoding='UTF-8'?><atom:entry
> xmlns:atom="http://www.w3.org/2005/Atom"
> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/"
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/"
> 
> xmlns:chemistry="http://chemistry.apache.org/"><atom:id>urn:uuid:00000000-0000-0000-0000-00000000000</atom:id><atom:title>global_.zip</atom:title><atom:updated>2014-04-03T00:21:10Z</atom:updated><cmisra:content><cmisra:mediatype>application/zip</cmisra:mediatype><chemistry:filename>global_.zip</chemistry:filename><cmisra:base64>UEsDB...
>
> And here is the request coming from the Mobile App:
>
> POST
> 
> /docushare/ds_mobile_connector/atom/docushare/children?id=Collection-14&includeAllowableActions=true
> HTTP/1.1
> Host: <confidential>
> Authorization: Basic <confidential>
> Accept-Encoding: gzip
> Content-Type: application/atom+xml;type=entry
> Accept-Language: en
> Content-Length: 2044539388
> Cookie-DS: <confidential>
> Connection: close
> User-Agent: <confidential>
>
> <?xml version="1.0" encoding="UTF-8" ?><entry
> xmlns="http://www.w3.org/2005/Atom"
> xmlns:app="http://www.w3.org/2007/app"
> 
> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/"><cmisra:content><cmisra:mediatype>application/zip</cmisra:mediatype><cmisra:base64>...
>
> ... Brian ...
>
> -----Original Message-----
> From: Florian Müller [mailto:fmui@apache.org]
> Sent: Wednesday, April 02, 2014 1:42 AM
> To: Inouye, Brian
> Cc: dev@chemistry.apache.org
> Subject: RE: create service with a large content stream causes a
> "Limit exceeded!" exception
>
>  Hi Brian,
>
>  A trace would be great.
>  It's good to know that the Workbench upload works. That focuses the
> investigation on the incoming XML.
>
>
>  - Florian
>
>
>
>> Hi Florian,
>>
>> This will be rather difficult to reproduce.  The "Limit exceeded!"
>> server exception occurs when the file is being uploaded from a
>> particular Mobile App going through WiFi to my server.  However, the
>> exception does not occur if the same file is uploaded from the CMIS
>> Workbench to my server.  I hope this isn't a timing-sensitive
>> problem.
>> I am going to look at a packet trace to see if I can see any
>> differences between how the Mobile App is transferring the file vs.
>> the CMIS Workbench.
>>
>> ... Brian ...
>>
>> -----Original Message-----
>> From: Florian Müller [mailto:fmui@apache.org]
>> Sent: Monday, March 31, 2014 12:31 PM
>> To: Inouye, Brian
>> Cc: dev@chemistry.apache.org
>> Subject: Re: create service with a large content stream causes a
>> "Limit exceeded!" exception
>>
>> Hi Brian,
>>
>> I tried a lot but I cannot reproduce it.
>> Is it possible to reproduce it with a smaller content and send me 
>> the
>> request? I would like to debug this.
>>
>>
>> - Florian
>>
>>
>>> Hi Florian,
>>>
>>> No, we did not change anything in the OpenCMIS code.
>>>
>>> AtomEntryParser uses XMLUtils to create the parser:
>>>
>>> 	XMLStreamReader parser = XMLUtils.createParser(cappedStream);
>>>
>>> XMLUtils explicitly sets isCoalescing to FALSE.
>>>
>>>         
>>> XML_INPUT_FACTORY.setProperty(XMLInputFactory.IS_COALESCING,
>>> Boolean.FALSE);
>>>
>>> ... Brian ...
>>>
>>> -----Original Message-----
>>> From: Florian Müller [mailto:fmui@apache.org]
>>> Sent: Friday, March 28, 2014 1:38 PM
>>> To: Inouye, Brian
>>> Cc: dev@chemistry.apache.org
>>> Subject: Re: create service with a large content stream causes a
>>> "Limit exceeded!" exception
>>>
>>> Hi Brian,
>>>
>>> I've revalidated that uploading huge documents works as expected.
>>>
>>> What you are seeing is only possible if the property
>>> "javax.xml.stream.isCoalescing" at the XMLInputFactory has been set 
>>> to
>>> TRUE somehow. Did you change anything in the OpenCMIS code?
>>>
>>>
>>> - Florian
>>>
>>>
>>>> Hi Brian,
>>>>
>>>> The parser should process the the content of base64 tag in small
>>>> chunks.
>>>> For some reasons it tries to read much more than that in your
>>>> setup.
>>>> I'll look into it.
>>>>
>>>> - Florian
>>>>
>>>>
>>>>> Hi Florian,
>>>>>
>>>>> Here is the exception stack.  It looks like it's using the
>>>>> Woodstox
>>>>> parser (I assume that's what com.ctc.wstx is).  The exception is
>>>>> happening when AtomEntryParser is calling readBase64, so I think
>>>>> it
>>>>> doesn't like the amount of the base64 encoded content stream.  
>>>>> The
>>>>> rest of the XML, excluding the content stream, is very small.
>>>>>
>>>>> We don't have control over the Client so our options are limited.
>>>>>
>>>>> ... Brian ...
>>>>>
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.commons.exceptions.CmisInvalidArgumentException:
>>>>>
>>>>> Limit exceeded!
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.shared.CappedInputStream.checkL
>>>>> e
>>>>> ngth(CappedInputStream.java:71)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.shared.CappedInputStream.read(C
>>>>> a
>>>>> ppedInputStream.java:107)
>>>>>
>>>>>     at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155)
>>>>>     at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:368)
>>>>>     at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:111)
>>>>>     at 
>>>>> com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:87)
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.
>>>>> java:57)
>>>>>
>>>>>     at
>>>>> com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:991)
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReade
>>>>> r
>>>>> .java:4647)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:
>>>>> 3721)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.
>>>>> j
>>>>> ava:3675)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> com.ctc.wstx.sr.BasicStreamReader.getTextLength(BasicStreamReader.ja
>>>>> v
>>>>> a:907)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.re
>>>>> a
>>>>> dBase64(AtomEntryParser.java:461)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.pa
>>>>> r
>>>>> seCmisContent(AtomEntryParser.java:375)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.pa
>>>>> r
>>>>> seEntry(AtomEntryParser.java:251)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.AtomEntryParser.pa
>>>>> r
>>>>> se(AtomEntryParser.java:214)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.ObjectService$Crea
>>>>> t
>>>>> e.serve(ObjectService.java:97)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.shared.Dispatcher.dispatch(Disp
>>>>> a
>>>>> tcher.java:88)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.
>>>>> dispatch(CmisAtomPubServlet.java:292)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.chemistry.opencmis.server.impl.atompub.CmisAtomPubServlet.
>>>>> service(CmisAtomPubServlet.java:228)
>>>>>
>>>>>     at
>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(App
>>>>> l
>>>>> icationFilterChain.java:290)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Application
>>>>> F
>>>>> ilterChain.java:206)
>>>>>
>>>>>     at
>>>>>
>>>>> 
>>>>> com.xerox.docushare.amber.util.UTF8Filter.doFilter(UTF8Filter.java:28)
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(App
>>>>> l
>>>>> icationFilterChain.java:235)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(Application
>>>>> F
>>>>> ilterChain.java:206)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapper
>>>>> V
>>>>> alve.java:233)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContext
>>>>> V
>>>>> alve.java:191)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.
>>>>> j
>>>>> ava:127)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.
>>>>> j
>>>>> ava:102)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVa
>>>>> l
>>>>> ve.java:109)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.ja
>>>>> v
>>>>> a:293)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.jav
>>>>> a
>>>>> :859)
>>>>>
>>>>>     at
>>>>>
>>>>>
>>>>> 
>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.proc
>>>>> e
>>>>> ss(Http11Protocol.java:602)
>>>>>
>>>>>     at
>>>>>
>>>>> 
>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
>>>>>     at java.lang.Thread.run(Thread.java:662)
>>>>>
>>>>> -----Original Message-----
>>>>> From: Florian Müller [mailto:fmui@apache.org]
>>>>> Sent: Thursday, March 27, 2014 4:46 PM
>>>>> To: Inouye, Brian
>>>>> Cc: dev@chemistry.apache.org
>>>>> Subject: Re: create service with a large content stream causes a
>>>>> "Limit exceeded!" exception
>>>>>
>>>>> One additional remark:
>>>>> To avoid the Base64 encoding create an empty document and then 
>>>>> set
>>>>> the content. It's much more efficient when you have to handle
>>>>> documents of that size.
>>>>> Or use the Browser binding.
>>>>>
>>>>>
>>>>> - Florian
>>>>>
>>>>>
>>>>>> Hi Florian,
>>>>>>
>>>>>> The Client is sending the content stream embedded in the
>>>>>> cmisra:content element, so the resulting XML is huge.
>>>>>>
>>>>>> <?xml version="1.0" encoding="UTF-8" ?> <entry
>>>>>> xmlns="http://www.w3.org/2005/Atom"
>>>>>> xmlns:app="http://www.w3.org/2007/app"
>>>>>>
>>>>>> 
>>>>>> xmlns:cmisra="http://docs.oasis-open.org/ns/cmis/restatom/200908/">
>>>>>> <cmisra:content>
>>>>>> <cmisra:mediatype>video/quicktime</cmisra:mediatype>
>>>>>> <cmisra:base64>[base64 encoding of
>>>>>> file]</cmisra:base64></cmisra:content>
>>>>>> <cmisra:object
>>>>>> xmlns:cmis="http://docs.oasis-open.org/ns/cmis/core/200908/">
>>>>>> <cmis:properties>...</cmis:properties>
>>>>>> </cmisra:object>
>>>>>> <title>Demo.mov</title>
>>>>>> </entry>
>>>>>>
>>>>>> ... Brian ...
>>>>>> Brian Inouye, Xerox Corporation
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Florian Müller [mailto:fmui@apache.org]
>>>>>> Sent: Thursday, March 27, 2014 3:03 PM
>>>>>> To: Inouye, Brian
>>>>>> Cc: dev@chemistry.apache.org
>>>>>> Subject: Re: create service with a large content stream causes a
>>>>>> "Limit exceeded!" exception
>>>>>>
>>>>>> Hi Brian,
>>>>>>
>>>>>> The CappedInputStream does not limit the content stream, but the
>>>>>> "envelope" around it. In case of the AtomPub binding it is
>>>>>> limiting
>>>>>> the size of the XML, but doesn't count the embedded document
>>>>>> content.
>>>>>> Could you check how the request is created and if it contains a
>>>>>> big
>>>>>> XML portion?
>>>>>>
>>>>>>
>>>>>> - Florian
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I've run into a limitation in my CMIS Provider which uses
>>>>>>> OpenCMIS
>>>>>>> 0.10.0.  When a Client sends a create request to my CMIS
>>>>>>> Provider
>>>>>>> and the accompanying content stream is large, say 1.4 GB,
>>>>>>> CappedInputStream raises an exception
>>>>>>> CmisInvalidArgumentException("Limit exceeded!").
>>>>>>> AtomEntryParser.java creates the CappedInputStream object,
>>>>>>> passing
>>>>>>> in a constant MAX_STREAM_LENGTH which is set to 10 * 1024 *
>>>>>>> 1024.
>>>>>>>
>>>>>>> public class CappedInputStream extends InputStream {
>>>>>>>     ...
>>>>>>>     private void checkLength() throws IOException {
>>>>>>>         if (counter > max) {
>>>>>>>             throw new CmisInvalidArgumentException("Limit
>>>>>>> exceeded!");
>>>>>>>         }
>>>>>>>     }
>>>>>>>     ...
>>>>>>> }
>>>>>>>
>>>>>>> public class AtomEntryParser {
>>>>>>>     ...
>>>>>>>     private static final long MAX_STREAM_LENGTH = 10 * 1024 *
>>>>>>> 1024;
>>>>>>>     ...
>>>>>>>     public void parse(InputStream stream) throws
>>>>>>> XMLStreamException, IOException {
>>>>>>>         ...
>>>>>>>         cappedStream = new CappedInputStream(stream,
>>>>>>> MAX_STREAM_LENGTH);
>>>>>>>         ...
>>>>>>>     }
>>>>>>> }
>>>>>>>
>>>>>>> What can I do to prevent this exception from occurring?  I 
>>>>>>> tried
>>>>>>> doubling the value, but the exception still occurs.
>>>>>>>
>>>>>>> ... Brian ...
>>>>>>> Brian Inouye, Xerox Corporation
>>>>>>>
>>>>>>>
>>>>

Mime
View raw message