Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@manifoldcf.apache.org
Received-SPF: pass (nike.apache.org: domain of ameya.aware@gmail.com
 designates 209.85.214.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CALUFAGBt-57Qj17VJvk__bg9MK_ETQCi-hBv=9npnM+OwaBazw@mail.gmail.com>
References: 
 <CABrC6qjSzbcx+MX0PLZEOob7xzhNe-L64iFfDqpYeEe95Aoszw@mail.gmail.com>
	<CALUFAGBt-57Qj17VJvk__bg9MK_ETQCi-hBv=9npnM+OwaBazw@mail.gmail.com>
Date: Thu, 31 Jul 2014 14:44:21 -0400
Message-ID: 
 <CABrC6qhHw-62+oQseAnDbngGkOhSw5CTCf6Fg5qQfSwXpPmi9Q@mail.gmail.com>
Subject: Re: Crawling and indexing very slow
From: Ameya Aware <ameya.aware@gmail.com>
To: "user@manifoldcf.apache.org" <user@manifoldcf.apache.org>
Content-Type: multipart/alternative; boundary=001a11c2d5ce38eef504ff81a90a

--001a11c2d5ce38eef504ff81a90a
Content-Type: text/plain; charset=UTF-8

So the thing here is i am not looking for any data or content of any of
files. I am just interested in metadata of file.

So i thought it should be possible to not read any file and just get
metadata of file and give to Solr.

This should save lots of time.

Is it possible to do this?

Thanks,
Ameya


On Thu, Jul 31, 2014 at 2:13 PM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Ameya,
>
> (1) Please look at the Simple History report.  Note what kinds of
> documents are being fetched, what kinds are being indexed, and how long it
> is taking.  I have noted from your previous posts that you seem to be
> indexing a lot of very large EXE files.  This is useless and you should be
> excluding them.
>
> (2) Please look in the manifoldcf.log file for evidence that fetches
> and/or Solr indexing requests are being retried due to errors.  It doesn't
> take many documents being chronically retried before forward progress drops
> to near zero.
>
> (3) If you look into (1) & (2) and everything seems fine, it may be a
> misalignment between availability of several kinds of resources that is the
> problem.  Please get a thread dump of the agents process while it is
> crawling, using jstack.  Post that thread dump and we can tell you what to
> look at next.
>
> Karl
>
>
>
> On Thu, Jul 31, 2014 at 2:07 PM, Ameya Aware <ameya.aware@gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> I am using filesystem connector to index my entire C drive using Solr as
>> output connector.
>>
>> Initial 100000 documents were crawled and indexed successfully in couple
>> of hours but after that indexing slowed down badly (around 15-20 documents
>> per min).
>>
>>
>> I am not able to figure out whether there is issue with MCF or Solr.
>>
>>
>> Can you advice me how to proceed with this?
>>
>>
>> Thanks,
>> Ameya
>>
>
>

--001a11c2d5ce38eef504ff81a90a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">So the thing here is i am not looking for any data or cont=
ent of any of files. I am just interested in metadata of file.<div><br></di=
v><div>So i thought it should be possible to not read any file and just get=
 metadata of file and give to Solr.</div>
<div><br></div><div>This should save lots of time.</div><div><br></div><div=
>Is it possible to do this?</div><div><br></div><div>Thanks,<br>Ameya</div>=
<div><br></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gmail=
_quote">
On Thu, Jul 31, 2014 at 2:13 PM, Karl Wright <span dir=3D"ltr">&lt;<a href=
=3D"mailto:daddywri@gmail.com" target=3D"_blank">daddywri@gmail.com</a>&gt;=
</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .=
8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr"><div><div><div>Hi Ameya,<br><br>(1) Please look at the Sim=
ple History report.=C2=A0 Note what kinds of documents are being fetched, w=
hat kinds are being indexed, and how long it is taking.=C2=A0 I have noted =
from your previous posts that you seem to be indexing a lot of very large E=
XE files.=C2=A0 This is useless and you should be excluding them.<br>

<br></div>(2) Please look in the manifoldcf.log file for evidence that fetc=
hes and/or Solr indexing requests are being retried due to errors.=C2=A0 It=
 doesn&#39;t take many documents being chronically retried before forward p=
rogress drops to near zero.<br>

<br></div>(3) If you look into (1) &amp; (2) and everything seems fine, it =
may be a misalignment between availability of several kinds of resources th=
at is the problem.=C2=A0 Please get a thread dump of the agents process whi=
le it is crawling, using jstack.=C2=A0 Post that thread dump and we can tel=
l you what to look at next.<span class=3D"HOEnZb"><font color=3D"#888888"><=
br>

<br></font></span></div><span class=3D"HOEnZb"><font color=3D"#888888">Karl=
<br><br></font></span></div><div class=3D"HOEnZb"><div class=3D"h5"><div cl=
ass=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Thu, Jul 31, 2014=
 at 2:07 PM, Ameya Aware <span dir=3D"ltr">&lt;<a href=3D"mailto:ameya.awar=
e@gmail.com" target=3D"_blank">ameya.aware@gmail.com</a>&gt;</span> wrote:<=
br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi,<div><br></div><div><br>=
</div><div>I am using filesystem connector to index my entire C drive using=
 Solr as output connector.</div>

<div><br></div><div>Initial 100000 documents were crawled and indexed succe=
ssfully in couple of hours but after that indexing slowed down badly (aroun=
d 15-20 documents per min).</div>
<div><br></div><div><br></div><div>I am not able to figure out whether ther=
e is issue with MCF or Solr.</div><div><br></div><div><br></div><div>Can yo=
u advice me how to proceed with this?</div><div><br></div><div><br></div>


<div>Thanks,</div><div>Ameya</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>

--001a11c2d5ce38eef504ff81a90a--