Return-Path: X-Original-To: apmail-manifoldcf-user-archive@www.apache.org Delivered-To: apmail-manifoldcf-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B4FF111867 for ; Fri, 18 Jul 2014 17:34:30 +0000 (UTC) Received: (qmail 43224 invoked by uid 500); 18 Jul 2014 17:34:30 -0000 Delivered-To: apmail-manifoldcf-user-archive@manifoldcf.apache.org Received: (qmail 43174 invoked by uid 500); 18 Jul 2014 17:34:30 -0000 Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@manifoldcf.apache.org Delivered-To: mailing list user@manifoldcf.apache.org Received: (qmail 43164 invoked by uid 99); 18 Jul 2014 17:34:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 17:34:30 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of daddywri@gmail.com designates 209.85.213.48 as permitted sender) Received: from [209.85.213.48] (HELO mail-yh0-f48.google.com) (209.85.213.48) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 17:34:26 +0000 Received: by mail-yh0-f48.google.com with SMTP id i57so2437499yha.35 for ; Fri, 18 Jul 2014 10:34:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=mPEP2IyWsHaN7rEy1Rks3V18br9k/8lfloS16Zxombk=; b=yz6Ewnrjp9Wz0pcPpgeIVf6xMoxADAtjEHwwiXMMG3OXvXSQmjT/EuUBkWpqen46SV YsUc9g89pAzjnsJ5qUhsP6fagtL4HsyPb+xA9HOd8s81walRTcUVoAotAo9bE92qF/l+ KdrMp0dAeLpdGgHcgV1vII2N5RwLgOcyysT3ub2c6ibe5HwIDzSH+YmEZFEvU7rfxySp aVfL3tl9SzZsMkXF4SJRU0ik4LKgRSrFAnx6jwyp28Wkdpl8vSDecVt5XjifrQHTTu5e m1FmBag3d/BuX0MiLYqPaWTrL4VW1mQJ0kM3X40c79w1Jss650T+KLClibuy4RvLga5P ytmA== MIME-Version: 1.0 X-Received: by 10.236.111.2 with SMTP id v2mr9313526yhg.39.1405704841062; Fri, 18 Jul 2014 10:34:01 -0700 (PDT) Received: by 10.170.118.139 with HTTP; Fri, 18 Jul 2014 10:34:00 -0700 (PDT) In-Reply-To: References: <-8665195481502634622@unknownmsgid> Date: Fri, 18 Jul 2014 13:34:00 -0400 Message-ID: Subject: Re: Performance issues From: Karl Wright To: Ameya Aware Cc: "user@manifoldcf.apache.org" Content-Type: multipart/alternative; boundary=001a11334698b5ed6f04fe7b29fc X-Virus-Checked: Checked by ClamAV on apache.org --001a11334698b5ed6f04fe7b29fc Content-Type: text/plain; charset=UTF-8 Hi Ameya, If you are still using Derby, which apparently you are according to the stack trace, then a pause of 420 seconds is likely because Derby got itself stuck. Derby is like that which is why we don't recommend it for production. Karl On Fri, Jul 18, 2014 at 1:31 PM, Ameya Aware wrote: > No Karl, > > I did not do VACUUM here. > > Why would queries stopped after running for about 420 sec? is it because > of the errors coming in? > > > On Fri, Jul 18, 2014 at 12:32 PM, Karl Wright wrote: > >> Hi Ameya, >> >> For future reference, when you see stuff like this in the log: >> >> >>>>>> >> WARN 2014-07-18 11:19:36,505 (Worker thread '39') - Found a long-running >> query (458934 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '4') - Found a long-running >> query (420965 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '39') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,505 (Worker thread '19') - Found a long-running >> query (421120 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '10') - Found a long-running >> query (420985 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '11') - Found a long-running >> query (421173 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '4') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,505 (Worker thread '11') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,505 (Worker thread '10') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,505 (Worker thread '39') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,505 (Worker thread '19') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,505 (Worker thread '39') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,505 (Worker thread '10') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,505 (Worker thread '22') - Found a long-running >> query (421052 ms): [UPDATE hopcount SET deathmark=?,distance=? WHERE id >> IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=? AND >> t0.childidhash=? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE >> t1.jobid=t0.jobid AND t1.linktype=t0.linktype AND >> t1.parentidhash=t0.parentidhash AND t1.childidhash=t0.childidhash AND >> t1.isnew=?))] >> WARN 2014-07-18 11:19:36,505 (Worker thread '11') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,505 (Worker thread '4') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,506 (Worker thread '11') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,506 (Worker thread '22') - Parameter 0: 'D' >> WARN 2014-07-18 11:19:36,506 (Worker thread '10') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,505 (Worker thread '39') - Parameter 3: >> '9ABFEB709B646CD0C84B4B7B6300E2C9BD5E3477' >> WARN 2014-07-18 11:19:36,505 (Worker thread '19') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,506 (Worker thread '39') - Parameter 4: 'B' >> WARN 2014-07-18 11:19:36,506 (Worker thread '10') - Parameter 3: >> 'A932EC77CEF156EA26A4239F12BAB365E6B4F58D' >> WARN 2014-07-18 11:19:36,506 (Worker thread '22') - Parameter 1: '-1' >> WARN 2014-07-18 11:19:36,506 (Worker thread '11') - Parameter 3: >> '9DFF75EBE13D0AAE8AFF025E992C68AB203ED1CB' >> WARN 2014-07-18 11:19:36,506 (Worker thread '4') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,506 (Worker thread '11') - Parameter 4: 'B' >> WARN 2014-07-18 11:19:36,506 (Worker thread '22') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,506 (Worker thread '22') - Parameter 3: >> '023FDBD3638711F4E55A918B862A064161B0892A' >> WARN 2014-07-18 11:19:36,506 (Worker thread '22') - Parameter 4: 'B' >> WARN 2014-07-18 11:19:36,506 (Worker thread '10') - Parameter 4: 'B' >> WARN 2014-07-18 11:19:36,506 (Worker thread '19') - Parameter 2: >> '1405692432586' >> WARN 2014-07-18 11:19:36,506 (Worker thread '4') - Parameter 3: >> '0158B8EDFEE3DDB10113B6D6E378D5FBF165E1FD' >> WARN 2014-07-18 11:19:36,506 (Worker thread '19') - Parameter 3: >> 'FD9641C67D0C1EC22B5F05671513D4DD71B4582C' >> WARN 2014-07-18 11:19:36,506 (Worker thread '4') - Parameter 4: 'B' >> WARN 2014-07-18 11:19:36,506 (Worker thread '19') - Parameter 4: 'B' >> <<<<<< >> >> ... it means that MANY queries basically stopped running for about 420 >> seconds. I bet you did a VACUUM then, right? >> >> Karl >> >> >> >> On Fri, Jul 18, 2014 at 12:30 PM, Karl Wright wrote: >> >>> Hi Ameya, >>> >>> The log file is full of errors of all sorts. For example: >>> >>> >>>>> >>> WARN 2014-07-17 17:32:38,709 (Worker thread '41') - IO exception during >>> indexing >>> file:/C:/Program%20Files/eclipse/configuration/org.eclipse.osgi/.manager/.tmp2043698995563843992.instance: >>> The process cannot access the file because another process has locked a >>> portion of the file >>> java.io.IOException: The process cannot access the file because another >>> process has locked a portion of the file >>> at java.io.FileInputStream.readBytes(Native Method) >>> at java.io.FileInputStream.read(Unknown Source) >>> at >>> org.apache.http.entity.mime.content.InputStreamBody.writeTo(InputStreamBody.java:91) >>> at >>> org.apache.manifoldcf.agents.output.solr.ModifiedHttpMultipart.doWriteTo(ModifiedHttpMultipart.java:211) >>> at >>> org.apache.manifoldcf.agents.output.solr.ModifiedHttpMultipart.writeTo(ModifiedHttpMultipart.java:229) >>> at >>> org.apache.manifoldcf.agents.output.solr.ModifiedMultipartEntity.writeTo(ModifiedMultipartEntity.java:187) >>> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>> at java.lang.reflect.Method.invoke(Unknown Source) >>> at >>> org.apache.http.impl.execchain.RequestEntityExecHandler.invoke(RequestEntityExecHandler.java:77) >>> at com.sun.proxy.$Proxy0.writeTo(Unknown Source) >>> at >>> org.apache.http.impl.DefaultBHttpClientConnection.sendRequestEntity(DefaultBHttpClientConnection.java:155) >>> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >>> at java.lang.reflect.Method.invoke(Unknown Source) >>> at org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProxy.java:138) >>> at com.sun.proxy.$Proxy1.sendRequestEntity(Unknown Source) >>> at >>> org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:236) >>> at >>> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:121) >>> at >>> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:254) >>> at >>> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) >>> at >>> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) >>> at >>> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) >>> at >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) >>> at >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106) >>> at >>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) >>> at >>> org.apache.manifoldcf.agents.output.solr.ModifiedHttpSolrServer.request(ModifiedHttpSolrServer.java:292) >>> at >>> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) >>> at >>> org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) >>> at >>> org.apache.manifoldcf.agents.output.solr.HttpPoster$IngestThread.run(HttpPoster.java:951) >>> <<<<< >>> >>> This error occurs because you are trying to index a file on Windows that >>> is open by an application. If you do this kind of thing, ManifoldCF will >>> requeue the document and will try it again later -- say, in 5 minutes, and >>> keep retrying it for many hours before it gives up. >>> >>> I suspect that you are not seeing "hangs", but rather situations where >>> MCF is simply waiting for a problem to resolve. >>> >>> Karl >>> >>> >>> >>> On Fri, Jul 18, 2014 at 11:27 AM, Ameya Aware >>> wrote: >>> >>>> Attaching log file >>>> >>>> >>>> On Fri, Jul 18, 2014 at 11:15 AM, Karl Wright >>>> wrote: >>>> >>>>> Also, please send the file logs/manifoldcf.log as well -- as a text >>>>> file. >>>>> >>>>> Karl >>>>> >>>>> >>>>> On Fri, Jul 18, 2014 at 11:12 AM, Karl Wright >>>>> wrote: >>>>> >>>>>> Could you please get a thread dump and send that to me? Please send >>>>>> as a text file not a screen shot. >>>>>> >>>>>> To get a thread dump, get the process ID of the agents process, and >>>>>> use the jdk's jstack utility to obtain the dump. >>>>>> >>>>>> Thanks, >>>>>> Karl >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Jul 18, 2014 at 11:08 AM, Ameya Aware >>>>>> wrote: >>>>>> >>>>>>> yeah.. i thought so that it should not effect in 4000 documents. >>>>>>> >>>>>>> I am using filesystem connector to crawl all of my C drive and >>>>>>> output connection is null. >>>>>>> >>>>>>> There are no error logs in MCF. MCF is standstill at same screen >>>>>>> since half an hour. >>>>>>> >>>>>>> Attaching some snapshots for your reference. >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> Ameya >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jul 18, 2014 at 11:02 AM, Karl Wright >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Ameya, >>>>>>>> >>>>>>>> 4000 documents is nothing at all. We have load tests which I run >>>>>>>> on every release that include more than 100000 documents on a crawl. >>>>>>>> >>>>>>>> Can you be more specific about the case that you say "hung up"? >>>>>>>> Specifically: >>>>>>>> >>>>>>>> (1) What kind of crawl is this? SharePoint? Web? >>>>>>>> (2) Are there any errors in the manifoldcf log? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Karl >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jul 18, 2014 at 10:59 AM, Ameya Aware < >>>>>>>> ameya.aware@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hi Karl, >>>>>>>>> >>>>>>>>> I spent some time going through PostgreSQL 9.3 manual. >>>>>>>>> I configured PostgreSQL for MCF and saw the significant change in >>>>>>>>> performance time. >>>>>>>>> >>>>>>>>> I ran it yesterday for some 4000 documents. When i started running >>>>>>>>> again today, the performance was very poor and after 200 documents, it hung >>>>>>>>> up. >>>>>>>>> >>>>>>>>> Is it because of periodic maintenance it needs? Also, i would >>>>>>>>> want to know where and how exactly VACUUM FULL command needs to >>>>>>>>> be used? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ameya >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jul 17, 2014 at 2:13 PM, Karl Wright >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> It is fine; I am running Postgresql 9.3 here. >>>>>>>>>> >>>>>>>>>> Karl >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jul 17, 2014 at 2:08 PM, Ameya Aware < >>>>>>>>>> ameya.aware@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> is PostgreySQL 9.3 version good because i already have it in my >>>>>>>>>>> machine.. Though documentation says "ManifoldCF has been tested >>>>>>>>>>> against version 8.3.7, 8.4.5 and 9.1 of PostgreSQL. " >>>>>>>>>>> >>>>>>>>>>> Ameya >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Jul 17, 2014 at 1:09 PM, Karl Wright >>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> If you haven't configured MCF to use PostgreSQL, then you are >>>>>>>>>>>> using Derby, which is not recommended for production use. >>>>>>>>>>>> >>>>>>>>>>>> Instructions on how to set up MCF to use PostgreSQL are >>>>>>>>>>>> available on the MCF site on the how-to-build-and-deploy page. Configuring >>>>>>>>>>>> PostgreSQL for millions or tens of millions of documents will require >>>>>>>>>>>> someone to learn about PostgreSQL and how to administer it. The >>>>>>>>>>>> how-to-build-and-deploy page provides some (old) guidelines and hints, but >>>>>>>>>>>> if I were you I'd read the postgresql manual for the version you install. >>>>>>>>>>>> >>>>>>>>>>>> Karl >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jul 17, 2014 at 1:04 PM, Ameya Aware < >>>>>>>>>>>> ameya.aware@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ooh ok. >>>>>>>>>>>>> >>>>>>>>>>>>> Actually i have never configured PostgreySQL yet. i am simply >>>>>>>>>>>>> using binary distribution of MCF to configure file system connectors to >>>>>>>>>>>>> connect to Solr. >>>>>>>>>>>>> >>>>>>>>>>>>> Do i need to configure PostgreySQL?? How can i proceed from >>>>>>>>>>>>> here to check performance measurements? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Ameya >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jul 17, 2014 at 12:10 PM, Karl Wright < >>>>>>>>>>>>> daddywri@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Yes. Also have a look at the how-to-build-and-deploy page >>>>>>>>>>>>>> for hints on how to configure PostgreSQL for maximum performance. >>>>>>>>>>>>>> >>>>>>>>>>>>>> ManifoldCF's performance is almost entirely based on the >>>>>>>>>>>>>> database. If you are using PostgreSQL, which is the fastest ManifoldCF >>>>>>>>>>>>>> choice, you should be able to see in the logs when queries take a long >>>>>>>>>>>>>> time, or when indexes are automatically rebuilt. Could you provide any >>>>>>>>>>>>>> information as to what your overall system setup looks like? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Karl >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jul 17, 2014 at 11:32 AM, Ameya Aware < >>>>>>>>>>>>>> ameya.aware@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> http://manifoldcf.apache.org/release/trunk/en_US/performance-tuning.html >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This page? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ameya >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jul 17, 2014 at 11:28 AM, Karl Wright < >>>>>>>>>>>>>>> daddywri@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ameya, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Have you read the performance page? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Karl >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent from my Windows Phone >>>>>>>>>>>>>>>> ------------------------------ >>>>>>>>>>>>>>>> From: Ameya Aware >>>>>>>>>>>>>>>> Sent: 7/17/2014 11:27 AM >>>>>>>>>>>>>>>> To: user@manifoldcf.apache.org >>>>>>>>>>>>>>>> Subject: Performance issues >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have millions of documents to crawl and send them to Solr. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> But when i run it for thousands documents, it takes too >>>>>>>>>>>>>>>> much time for it or sometimes it even hangs up. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So what could be the way to reduce the performance time? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also, i do not need content of the documents, i just need >>>>>>>>>>>>>>>> metadata, so can i skip content part from reading and fetching and will >>>>>>>>>>>>>>>> that improve performance time? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Ameya >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > --001a11334698b5ed6f04fe7b29fc Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Ameya,

If you are still using Derby, which appar= ently you are according to the stack trace, then a pause of 420 seconds is = likely because Derby got itself stuck.=C2=A0 Derby is like that which is wh= y we don't recommend it for production.

Karl



On Fri, Jul 18, 2014 at 1:31 PM, Ameya Aware <= ;ameya.aware@gma= il.com> wrote:
No Karl,

I did not do VACUUM here.

Why would queries stopp= ed after running for about 420 sec? is it because of the errors coming in?<= /div>


On Fri, Jul 18, 2014 at 12:32 PM, Karl W= right <daddywri@gmail.com> wrote:
Hi Ameya,

For future reference, when you = see stuff like this in the log:

>>>>>>
=C2=A0WA= RN 2014-07-18 11:19:36,505 (Worker thread '39') - Found a long-runn= ing query (458934 ms): [UPDATE hopcount SET deathmark=3D?,distance=3D? WHER= E id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=3D? AND t0.chil= didhash=3D? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WHERE t1.jo= bid=3Dt0.jobid AND t1.linktype=3Dt0.linktype AND t1.parentidhash=3Dt0.paren= tidhash AND t1.childidhash=3Dt0.childidhash AND t1.isnew=3D?))]
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '4') - Found a lo= ng-running query (420965 ms): [UPDATE hopcount SET deathmark=3D?,distance= =3D? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=3D? AN= D t0.childidhash=3D? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WH= ERE t1.jobid=3Dt0.jobid AND t1.linktype=3Dt0.linktype AND t1.parentidhash= =3Dt0.parentidhash AND t1.childidhash=3Dt0.childidhash AND t1.isnew=3D?))]<= br> =C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '39') -=C2=A0=C2= =A0 Parameter 0: 'D'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker = thread '19') - Found a long-running query (421120 ms): [UPDATE hopc= ount SET deathmark=3D?,distance=3D? WHERE id IN(SELECT ownerid FROM hopdele= tedeps t0 WHERE t0.jobid=3D? AND t0.childidhash=3D? AND EXISTS(SELECT '= x' FROM intrinsiclink t1 WHERE t1.jobid=3Dt0.jobid AND t1.linktype=3Dt0= .linktype AND t1.parentidhash=3Dt0.parentidhash AND t1.childidhash=3Dt0.chi= ldidhash AND t1.isnew=3D?))]
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '10') - Found a l= ong-running query (420985 ms): [UPDATE hopcount SET deathmark=3D?,distance= =3D? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=3D? AN= D t0.childidhash=3D? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WH= ERE t1.jobid=3Dt0.jobid AND t1.linktype=3Dt0.linktype AND t1.parentidhash= =3Dt0.parentidhash AND t1.childidhash=3Dt0.childidhash AND t1.isnew=3D?))]<= br> =C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '11') - Found a l= ong-running query (421173 ms): [UPDATE hopcount SET deathmark=3D?,distance= =3D? WHERE id IN(SELECT ownerid FROM hopdeletedeps t0 WHERE t0.jobid=3D? AN= D t0.childidhash=3D? AND EXISTS(SELECT 'x' FROM intrinsiclink t1 WH= ERE t1.jobid=3Dt0.jobid AND t1.linktype=3Dt0.linktype AND t1.parentidhash= =3Dt0.parentidhash AND t1.childidhash=3Dt0.childidhash AND t1.isnew=3D?))]<= br> =C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '4') -=C2=A0=C2= =A0 Parameter 0: 'D'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker = thread '11') -=C2=A0=C2=A0 Parameter 0: 'D'
=C2=A0WARN 2= 014-07-18 11:19:36,505 (Worker thread '10') -=C2=A0=C2=A0 Parameter= 0: 'D'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '39') -=C2=A0=C2= =A0 Parameter 1: '-1'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker= thread '19') -=C2=A0=C2=A0 Parameter 0: 'D'
=C2=A0WARN = 2014-07-18 11:19:36,505 (Worker thread '39') -=C2=A0=C2=A0 Paramete= r 2: '1405692432586'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '10') -=C2=A0=C2= =A0 Parameter 1: '-1'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker= thread '22') - Found a long-running query (421052 ms): [UPDATE hop= count SET deathmark=3D?,distance=3D? WHERE id IN(SELECT ownerid FROM hopdel= etedeps t0 WHERE t0.jobid=3D? AND t0.childidhash=3D? AND EXISTS(SELECT '= ;x' FROM intrinsiclink t1 WHERE t1.jobid=3Dt0.jobid AND t1.linktype=3Dt= 0.linktype AND t1.parentidhash=3Dt0.parentidhash AND t1.childidhash=3Dt0.ch= ildidhash AND t1.isnew=3D?))]
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '11') -=C2=A0=C2= =A0 Parameter 1: '-1'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker= thread '4') -=C2=A0=C2=A0 Parameter 1: '-1'
=C2=A0WARN = 2014-07-18 11:19:36,506 (Worker thread '11') -=C2=A0=C2=A0 Paramete= r 2: '1405692432586'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '22') -=C2=A0=C2= =A0 Parameter 0: 'D'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker = thread '10') -=C2=A0=C2=A0 Parameter 2: '1405692432586'
= =C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '39') -=C2=A0=C2= =A0 Parameter 3: '9ABFEB709B646CD0C84B4B7B6300E2C9BD5E3477'
=C2=A0WARN 2014-07-18 11:19:36,505 (Worker thread '19') -=C2=A0=C2= =A0 Parameter 1: '-1'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker= thread '39') -=C2=A0=C2=A0 Parameter 4: 'B'
=C2=A0WARN = 2014-07-18 11:19:36,506 (Worker thread '10') -=C2=A0=C2=A0 Paramete= r 3: 'A932EC77CEF156EA26A4239F12BAB365E6B4F58D'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '22') -=C2=A0=C2= =A0 Parameter 1: '-1'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker= thread '11') -=C2=A0=C2=A0 Parameter 3: '9DFF75EBE13D0AAE8AFF0= 25E992C68AB203ED1CB'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '4') -=C2=A0=C2= =A0 Parameter 2: '1405692432586'
=C2=A0WARN 2014-07-18 11:19:36,= 506 (Worker thread '11') -=C2=A0=C2=A0 Parameter 4: 'B'
= =C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '22') -=C2=A0=C2= =A0 Parameter 2: '1405692432586'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '22') -=C2=A0=C2= =A0 Parameter 3: '023FDBD3638711F4E55A918B862A064161B0892A'
=C2= =A0WARN 2014-07-18 11:19:36,506 (Worker thread '22') -=C2=A0=C2=A0 = Parameter 4: 'B'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '10') -=C2=A0=C2= =A0 Parameter 4: 'B'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker = thread '19') -=C2=A0=C2=A0 Parameter 2: '1405692432586'
= =C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '4') -=C2=A0=C2= =A0 Parameter 3: '0158B8EDFEE3DDB10113B6D6E378D5FBF165E1FD'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '19') -=C2=A0=C2= =A0 Parameter 3: 'FD9641C67D0C1EC22B5F05671513D4DD71B4582C'
=C2= =A0WARN 2014-07-18 11:19:36,506 (Worker thread '4') -=C2=A0=C2=A0 P= arameter 4: 'B'
=C2=A0WARN 2014-07-18 11:19:36,506 (Worker thread '19') -=C2=A0=C2= =A0 Parameter 4: 'B'
<<<<<<

... it means that MANY queries basica= lly stopped running for about 420 seconds.=C2=A0 I bet you did a VACUUM the= n, right?

Karl



On Fri, Jul 18, 2014 at 12:30 PM, Karl Wright <daddywri@gmail.com>= wrote:
Hi Ameya,

The log file is full of errors = of all sorts.=C2=A0 For example:

>>>>>
=C2=A0WARN = 2014-07-17 17:32:38,709 (Worker thread '41') - IO exception during = indexing file:/C:/Program%20Files/eclipse/configuration/org.eclipse.osgi/.m= anager/.tmp2043698995563843992.instance: The process cannot access the file= because another process has locked a portion of the file
java.io.IOException: The process cannot access the file because another pro= cess has locked a portion of the file
=C2=A0=C2=A0=C2=A0 at java.io.File= InputStream.readBytes(Native Method)
=C2=A0=C2=A0=C2=A0 at java.io.FileI= nputStream.read(Unknown Source)
=C2=A0=C2=A0=C2=A0 at org.apache.http.entity.mime.content.InputStreamBody.w= riteTo(InputStreamBody.java:91)
=C2=A0=C2=A0=C2=A0 at org.apache.manifol= dcf.agents.output.solr.ModifiedHttpMultipart.doWriteTo(ModifiedHttpMultipar= t.java:211)
=C2=A0=C2=A0=C2=A0 at org.apache.manifoldcf.agents.output.so= lr.ModifiedHttpMultipart.writeTo(ModifiedHttpMultipart.java:229)
=C2=A0=C2=A0=C2=A0 at org.apache.manifoldcf.agents.output.solr.ModifiedMult= ipartEntity.writeTo(ModifiedMultipartEntity.java:187)
=C2=A0=C2=A0=C2=A0= at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
=C2=A0= =C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Sou= rce)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Unknown Source)
= =C2=A0=C2=A0=C2=A0 at org.apache.http.impl.execchain.RequestEntityExecHandl= er.invoke(RequestEntityExecHandler.java:77)
=C2=A0=C2=A0=C2=A0 at com.su= n.proxy.$Proxy0.writeTo(Unknown Source)
=C2=A0=C2=A0=C2=A0 at org.apache.http.impl.DefaultBHttpClientConnection.sen= dRequestEntity(DefaultBHttpClientConnection.java:155)
=C2=A0=C2=A0=C2=A0= at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
=C2=A0= =C2=A0=C2=A0 at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Sou= rce)
=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Unknown Source)
= =C2=A0=C2=A0=C2=A0 at org.apache.http.impl.conn.CPoolProxy.invoke(CPoolProx= y.java:138)
=C2=A0=C2=A0=C2=A0 at com.sun.proxy.$Proxy1.sendRequestEntit= y(Unknown Source)
=C2=A0=C2=A0=C2=A0 at org.apache.http.protocol.HttpReq= uestExecutor.doSendRequest(HttpRequestExecutor.java:236)
=C2=A0=C2=A0=C2=A0 at org.apache.http.protocol.HttpRequestExecutor.execute(= HttpRequestExecutor.java:121)
=C2=A0=C2=A0=C2=A0 at org.apache.http.impl= .execchain.MainClientExec.execute(MainClientExec.java:254)
=C2=A0=C2=A0= =C2=A0 at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.= java:195)
=C2=A0=C2=A0=C2=A0 at org.apache.http.impl.execchain.RedirectExec.execute(R= edirectExec.java:108)
=C2=A0=C2=A0=C2=A0 at org.apache.http.impl.client.= InternalHttpClient.doExecute(InternalHttpClient.java:186)
=C2=A0=C2=A0= =C2=A0 at org.apache.http.impl.client.CloseableHttpClient.execute(Closeable= HttpClient.java:82)
=C2=A0=C2=A0=C2=A0 at org.apache.http.impl.client.CloseableHttpClient.execu= te(CloseableHttpClient.java:106)
=C2=A0=C2=A0=C2=A0 at org.apache.http.i= mpl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
=C2= =A0=C2=A0=C2=A0 at org.apache.manifoldcf.agents.output.solr.ModifiedHttpSol= rServer.request(ModifiedHttpSolrServer.java:292)
=C2=A0=C2=A0=C2=A0 at org.apache.solr.client.solrj.impl.HttpSolrServer.requ= est(HttpSolrServer.java:199)
=C2=A0=C2=A0=C2=A0 at org.apache.solr.clien= t.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:11= 7)
=C2=A0=C2=A0=C2=A0 at org.apache.manifoldcf.agents.output.solr.HttpPo= ster$IngestThread.run(HttpPoster.java:951)
<<<<<

This error occurs because you are trying = to index a file on Windows that is open by an application.=C2=A0 If you do = this kind of thing, ManifoldCF will requeue the document and will try it ag= ain later -- say, in 5 minutes, and keep retrying it for many hours before = it gives up.

I suspect that you are not seeing "hangs", but rather situati= ons where MCF is simply waiting for a problem to resolve.

Karl=



On Fri, Jul 18, 2014 at 11:27 AM, Ameya Aware <ameya.aware@gmail.com> wrote:
Attaching log file


On Fri, Jul 18, 2014 at 11:15 AM, Kar= l Wright <daddywri@gmail.com> wrote:
Also, please send the = file logs/manifoldcf.log as well -- as a text file.

Karl


On Fri, Jul 18, 2014 at 11:12 AM, Karl Wright <daddywri@gmail.com= > wrote:
Could you please get a= thread dump and send that to me?=C2=A0 Please send as a text file not a sc= reen shot.

To get a thread dump, get the process ID of the agents process, a= nd use the jdk's jstack utility to obtain the dump.

Thanks,
Karl


<= br>
On Fri, Jul 18, 2014 at 11:08 AM, Ameya Aware= <ameya.aware@gmail.com> wrote:
yeah.. i thought so that it= should not effect in 4000 documents.

I am using filesys= tem connector to crawl all of my C drive and output connection is null.

There are no error logs in MCF. MCF is standstill at sa= me screen since half an hour.

Attaching some snapshots for your reference.
=

Thanks,
Ameya




On Fri, Jul 18, 2014 at 11:02 AM, Karl Wright <daddywri@gmail.com>= wrote:
Hi Ameya,

4000 documents = is nothing at all.=C2=A0 We have load tests which I run on every release th= at include more than 100000 documents on a crawl.

Can you be m= ore specific about the case that you say "hung up"?=C2=A0 Specifi= cally:

(1) What kind of crawl is this?=C2=A0 SharePoint?=C2=A0 Web?
<= /div>(2) Are there any errors in the manifoldcf log?

Thanks,
Karl=





On Fri, Jul 18, 2014 at 10:59 AM, Ameya Awar= e <ameya.aware@gmail.com> wrote:
Hi Karl,

I spent some time going throug= h PostgreSQL 9.3 manual.
I configured PostgreSQL for MCF and saw = the significant change in performance time.

I ran it yesterday for some 4000 documents. When i star= ted running again today, the performance was very poor and after 200 docume= nts, it hung up.

Is it because of periodic mainten= ance it needs? =C2=A0Also, i would want to know where and how exactly=C2=A0= VACUUM FULL command needs to= be used?

<= span style=3D"color:rgb(0,0,0);font-family:Verdana,Helvetica,sans-serif;fon= t-size:13px;line-height:15.359999656677246px">Thanks,
Ameya


On Thu, Jul 17, 2014 at 2:13 PM, Karl Wright <daddywri@gmail.com>= wrote:
It is fine; I am running Po= stgresql 9.3 here.

Karl


On Thu, Jul 17, 2014 at 2:08 PM, Ameya Aware <ameya.a= ware@gmail.com> wrote:
is PostgreySQL 9.3 version = good because i already have it in my machine.. Though documentation says &q= uot;ManifoldCF has been test= ed against version 8.3.7, 8.4.5 and 9.1 of PostgreSQL.=C2=A0"

Ameya


On Thu, Jul 17, 2014 at 1:09 PM, Karl Wright <daddywri@g= mail.com> wrote:
If you haven'= t configured MCF to use PostgreSQL, then you are using Derby, which is not = recommended for production use.

Instructions on how to set up MCF to use PostgreSQL are available= on the MCF site on the how-to-build-and-deploy page.=C2=A0 Configuring Pos= tgreSQL for millions or tens of millions of documents will require someone = to learn about PostgreSQL and how to administer it.=C2=A0 The how-to-build-= and-deploy page provides some (old) guidelines and hints, but if I were you= I'd read the postgresql manual for the version you install.

Karl



On Thu, Jul 17, 2014 at 1:04 PM, Ameya Aware <= ameya.aware@gmai= l.com> wrote:
Ooh ok.

= Actually i have never configured PostgreySQL yet. i am simply using binary = distribution of MCF to configure file system connectors to connect to Solr.=

Do i need to configure PostgreySQL?? How can i proceed = from here to check performance measurements?

Thanks,
Ameya


On Thu, Jul 17, 2014 at= 12:10 PM, Karl Wright <daddywri@gmail.com> wrote:
Yes.=C2=A0 Also have a= look at the how-to-build-and-deploy page for hints on how to configure Pos= tgreSQL for maximum performance.

ManifoldCF's performance is almost entirely based on the data= base.=C2=A0 If you are using PostgreSQL, which is the fastest ManifoldCF ch= oice, you should be able to see in the logs when queries take a long time, = or when indexes are automatically rebuilt.=C2=A0 Could you provide any info= rmation as to what your overall system setup looks like?

Karl


On Thu, Jul 17, 2014 at 11:32 AM, Ameya Aw= are <ameya.aware@gmail.com> wrote:


On Thu, Jul 17, 2014 at 11:28 AM, Ka= rl Wright <daddywri@gmail.com> wrote:
Hi Ameya,

Have you read the performance p= age?

Karl

Sent from my Windows Phone
From: Ameya Aware
Sent: 7/17/2014 11:27 AM
To: = user@manifo= ldcf.apache.org
Subject: Performance issues

Hi=C2=A0

I have millions of documents to crawl and send them to Solr.=

But when i run it for thousands documents, it tak= es too much time for it or sometimes it even hangs up.

So what could be the way to reduce the performance time= ?=C2=A0

Also, i do not need content of the documen= ts, i just need metadata, so can i skip content part from reading and fetch= ing and will that improve performance time?


Thanks,
Ameya
















--001a11334698b5ed6f04fe7b29fc--