chemistry-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Müller <f...@apache.org>
Subject Re: CMIS paging query result sample code
Date Fri, 17 Nov 2017 09:58:12 GMT
Hi Wentao,

OpenCMIS does pretty much what you have done, but behind the scenes. It 
sets the maxItems and skipCount parameters when it executes a query. 
Once a batch is processed, it fires the same query again with another 
skipCount.

Re 1) Do you have a cluster of servers? If so, the other calls might 
have been processed by another node.

Re 2) The server needed 34 seconds of the 52 seconds to compile the 
response. That leaves us with 18 seconds, which is still a lot of time. 
OpenCMIS is usually pretty fast. Do you have an idea how big the 
responses are?

Re 3) Depending on how paging is implemented on the server side, the 
server might have to internally recalculate the first page again before 
it can serve the second page. That’s the case for many repositories. 
Every page gets slower than the page before. That’s why you should make 
the pages as big as possible if you want process all query results.


- Florian



> Thanks Florian. It works.
> 
> Could you help to provide a little more details on how the  paging
> works between OpenCMIS client with CMIS server on the backend? Here
> are some logs I get from FileNet CMIS for FileNet access logs.
> 
> 2017-11-16      14:03:09        0.02    GET
> /fncmis/resources/Service?repositoryId=os1  200
> 2017-11-16      14:03:39        0.013   GET
> /fncmis/resources/Service?repositoryId= os1  200
> 2017-11-16      14:03:39        0.003   GET
> /fncmis/resources/Service?repositoryId= os1  200
> 2017-11-16      14:04:13        34.107  POST    
> /fncmis/resources/os1/Query 201
> 2017-11-16      14:05:05        0.028   GET
> /fncmis/resources/Service?repositoryId= os1  200
> 2017-11-16      14:05:05        0.004   GET
> /fncmis/resources/Service?repositoryId= os1  200
> 2017-11-16      14:05:55        50.104  POST    
> /fncmis/resources/os1/Query 201
> 
> There were total 6661 items back and the page size was 1500, from the
> log I noticed a few things that I am not quite understand:
> 
> 1. There were only 2 queries been triggered, I thought it should at
> least had 5 (6661/1500)?
> 2. There were 52 seconds (14:05:05 - 14:04:13) between the first query
> call and the subsequent calls, I guess it took a while for OpenCMIS
> client to process something?  52 seconds seems like a bit longer than
> expect.
> 3. The second query took longer than first query, but I guess it is a
> question for IBM CMIS expert to answer.
> 
> Wentao
> 
> -----Original Message-----
> From: Florian Müller [mailto:fmui@apache.org]
> Sent: Thursday, November 16, 2017 1:32 PM
> To: dev@chemistry.apache.org; Lu, Wentao
> Subject: Re: CMIS paging query result sample code
> 
> Hi Wentao,
> 
> If you are processing all query results, it's actually much simpler
> than you think.
> 
> Please see:
> https://chemistry.apache.org/docs/cmis-samples/samples/operation-context/index.html#list-batch-size
> 
> Use an Operation Context, set the maxItemsPerPage and just iterate
> over the query results:
> 
> 
> OperationContext oc = 
> OperationContextUtils.createMinimumOperationContext();
> oc.setMaxItemsPerPage(500);
> 
> ItemIterable<QueryResult>  results = session.query(queryString, false, 
> oc);
> 
> for (QueryResult result : results) {
>   System.out.println(result.getPropertyValueByQueryName("cmis:name"));
> }
> 
> 
> 
> - Florian
> 
> 
>> Hi,
>> 
>> Is there a sample code for OpenCMIS to paging query result?  We have a
>> query that will return large result set, we want to use paging query
>> to avoid timeout if return all dataset in one query from backend
>> FileNet repository, is there a sample code we can reference?  I
>> googled but get confused sample code especially after the first query
>> call,
>> 
>>      while (results.getHasMoreItems()){
>>          PAGE_NUM ++;
>> //Should we do another session.query() call or just use the 
>> results.skipTo() from the first query result?
>>          ItemIterable<QueryResult>  tmp_results = 
>> session.query(queryString, false).skipTo(PAGE_NUM * 
>> PAGE_SIZE).getPage(PAGE_SIZE);   //this line or next line?
>>          ItemIterable<QueryResult>  tmp_results =
>> results.skipTo(PAGE_NUM * PAGE_SIZE).getPage(PAGE_SIZE);
>> 
>>         for (QueryResult result : tmp_results) {
>>           
>> System.out.println(result.getPropertyValueByQueryName("cmis:name"));
>>           resultCount++;
>>         }
>>     }
>> 
>> Here is the complete code snip
>>      String queryString = "SELECT cmis:name FROM InformationSystem";
>> 
>>      int PAGE_NUM = 0;
>>      int PAGE_SIZE = 500;
>> 
>>      ItemIterable<QueryResult> results = session.query(queryString, 
>> false).skipTo(PAGE_NUM * PAGE_SIZE).getPage(PAGE_SIZE);
>>      int resultCount = 0;
>>      for (QueryResult result : results) {
>>           
>> System.out.println(result.getPropertyValueByQueryName("cmis:name"));
>>           resultCount ++;
>>      }
>>      System.out.println("resultCount="+resultCount);
>> 
>>      while (results.getHasMoreItems()){
>>          PAGE_NUM ++;
>> //Should we do another session.query() call or just use the 
>> results.skipTo() from the first query result?
>>          ItemIterable<QueryResult>  tmp_results = 
>> session.query(queryString, false).skipTo(PAGE_NUM * 
>> PAGE_SIZE).getPage(PAGE_SIZE);
>>          ItemIterable<QueryResult>  tmp_results =
>> results.skipTo(PAGE_NUM * PAGE_SIZE).getPage(PAGE_SIZE);
>> 
>>         for (QueryResult result : tmp_results) {
>>           
>> System.out.println(result.getPropertyValueByQueryName("cmis:name"));
>>           resultCount++;
>>         }
>>         System.out.println("resultCount="+resultCount);
>>     }
>> 
>> 
>> 
>> Thanks
>> Wentao
>> ________________________________
>> This email and its attachments are intended solely for the personal 
>> use of the individual or entity named above. Any use of this 
>> communication by an unintended recipient is strictly prohibited. If 
>> you have received this email in error, any publication, use, 
>> reproduction, disclosure or dissemination of its contents is strictly 
>> prohibited. Please immediately delete this message and its attachments 
>> from your computer and servers. We would also appreciate if you would 
>> contact us by a collect call or return email to notify us of this 
>> error. Thank you for your cooperation.
>> -BCHydroDisclaimerID5.2.8.1541
>> 

Mime
View raw message