manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Karl Wright <daddy...@gmail.com>
Subject Re: SharePoint crawler ArrayIndexOutOfBoundException in log
Date Fri, 17 Mar 2017 18:27:15 GMT
Hi Cihad,

There are NTLMEngineImpl tests that exercise precisely the case that is
failing.  I'm therefore becoming convinced that there is something very odd
about your installation.  Are you using a non-standard JVM, for instance?

Karl


On Fri, Mar 17, 2017 at 10:28 AM, Karl Wright <daddywri@gmail.com> wrote:

> Hi Cihad,
>
> Could you also check out and build the latest 4.5.x httpclient, from this
> branch?
>
> https://svn.apache.org/repos/asf/httpcomponents/httpclient/
> branches/pull-66
>
> You will need maven for this but otherwise you can build it any way you
> like.  Replace the "httpclient-4.5.1.jar" in the lib directory with the jar
> you build, and then you can rebuild MCF.  See if you still get the error.
> If you do, it should be possible to chase it down more readily.
>
> Thanks,
> Karl
>
>
> On Fri, Mar 17, 2017 at 9:57 AM, Cihad Guzel <cguzelg@gmail.com> wrote:
>
>> No. I don't use any custom library.
>>
>> I try with manifoldcf trunk on my notebook. I install sharepoint 2013 on
>> ms server 2012 for testing with default configuration.
>>
>> 17 Mar 2017 16:05 tarihinde "Karl Wright" <daddywri@gmail.com> yazdı:
>>
>> Hmm, I can see no way this can happen.  Are you by any chance using a
>>> modified version of the HttpClient library?
>>> Karl
>>>
>>>
>>> On Fri, Mar 17, 2017 at 8:09 AM, Karl Wright <daddywri@gmail.com> wrote:
>>>
>>>> Hi Cihad,
>>>>
>>>> This is very interesting because the problem is coming from
>>>> Httpclient's NTLM engine.  The allocated packet size for the Type 1 message
>>>> is being exceeded, which I didn't think was even possible.
>>>>
>>>> This may be a result of credentials that you have supplied being
>>>> strange in some way.  Let me look at the Httpclient code and get back to
>>>> you.
>>>>
>>>> Karl
>>>>
>>>>
>>>> On Fri, Mar 17, 2017 at 7:57 AM, Cihad Guzel <cguzelg@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I try sharepoint connector with Active Directory in debug mode. I saw
>>>>> ArrayIndexOutOfBoundException in manifoldcf.log file. Any bugs?
>>>>>
>>>>> DEBUG 2017-03-17 14:30:48,386 (Worker thread '0') - SharePoint:
>>>>> Getting version of '/Documents2//Step by step Installation of SharePoint
>>>>> 2013 on Windows Server 2012 R2 part 1 - SharePoint Community.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,466 (Worker thread '0') - SharePoint:
>>>>> Checking whether to include document '/Documents2/Step by step Installation
>>>>> of SharePoint 2013 on Windows Server 2012 R2 part 1 - SharePoint
>>>>> Community.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,466 (Worker thread '0') - SharePoint: File
>>>>> '/Documents2/Step by step Installation of SharePoint 2013 on Windows
Server
>>>>> 2012 R2 part 1 - SharePoint Community.pdf' exactly matched rule path
>>>>> '/Documents2/*'
>>>>> DEBUG 2017-03-17 14:30:48,467 (Worker thread '0') - SharePoint:
>>>>> Including file '/Documents2/Step by step Installation of SharePoint 2013
on
>>>>> Windows Server 2012 R2 part 1 - SharePoint Community.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,468 (Worker thread '0') - SharePoint:
>>>>> Finding metadata to include for document/item '/Documents2/Step by step
>>>>> Installation of SharePoint 2013 on Windows Server 2012 R2 part 1 -
>>>>> SharePoint Community.pdf'.
>>>>> DEBUG 2017-03-17 14:30:48,510 (Worker thread '0') - SharePoint: In
>>>>> getFieldValues; fieldNames=[Ljava.lang.String;@69f1a61a, site='',
>>>>> docLibrary='{1B694C45-DF1F-44E7-9814-F5096E85A126}',
>>>>> docId='/Documents2/Step by step Installation of SharePoint 2013 on Windows
>>>>> Server 2012 R2 part 1 - SharePoint Community.pdf', dspStsWorks=false
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '5') - SharePoint:
>>>>> Getting version of '/Documents2//'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '4') - SharePoint:
>>>>> Getting version of '/Documents2//CXFCA3100080010.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '4') - SharePoint:
>>>>> Checking whether to include document '/Documents2/CXFCA3100080010.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '4') - SharePoint: File
>>>>> '/Documents2/CXFCA3100080010.pdf' exactly matched rule path
>>>>> '/Documents2/*'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '4') - SharePoint:
>>>>> Including file '/Documents2/CXFCA3100080010.pdf'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '5') - SharePoint:
>>>>> Checking whether to include library '/Documents2'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '4') - SharePoint:
>>>>> Finding metadata to include for document/item '/Documents2/CXFCA3100080010.p
>>>>> df'.
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '5') - SharePoint:
>>>>> Library '/Documents2' partially matched file rule path '/Documents2/*'
-
>>>>> including
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '5') - SharePoint:
>>>>> Document identifier is a library: '/Documents2'
>>>>> DEBUG 2017-03-17 14:30:48,539 (Worker thread '5') - SharePoint: In
>>>>> getDocLibID; parentSite='', parentSiteDecoded='', docLibrary='Documents2'
>>>>> DEBUG 2017-03-17 14:30:48,540 (Worker thread '2') - SharePoint:
>>>>> Getting version of '/'
>>>>> DEBUG 2017-03-17 14:30:48,540 (Worker thread '2') - SharePoint:
>>>>> Checking whether to include site '/'
>>>>> DEBUG 2017-03-17 14:30:48,540 (Worker thread '2') - SharePoint: Site
>>>>> '/' partially matched file rule path '/Documents2/*' - including
>>>>> DEBUG 2017-03-17 14:30:48,548 (Worker thread '4') - SharePoint: In
>>>>> getFieldValues; fieldNames=[Ljava.lang.String;@6f447d2e, site='',
>>>>> docLibrary='{1B694C45-DF1F-44E7-9814-F5096E85A126}',
>>>>> docId='/Documents2/CXFCA3100080010.pdf', dspStsWorks=false
>>>>> DEBUG 2017-03-17 14:30:48,560 (Worker thread '2') - SharePoint:
>>>>> Document identifier is a site: ''
>>>>> DEBUG 2017-03-17 14:30:48,560 (Worker thread '2') - SharePoint: In
>>>>> getSites; parentSite=''
>>>>> DEBUG 2017-03-17 14:30:50,398 (Worker thread '4') - SharePoint: Got a
>>>>> remote exception getting field values for site  library
>>>>> {1B694C45-DF1F-44E7-9814-F5096E85A126} document
>>>>> [/Documents2/CXFCA3100080010.pdf] - retrying
>>>>> AxisFault
>>>>>  faultCode: {http://schemas.xmlsoap.org/so
>>>>> ap/envelope/}Server.userException
>>>>>  faultSubcode:
>>>>>  faultString: java.lang.ArrayIndexOutOfBoundsException: 41
>>>>>  faultActor:
>>>>>  faultNode:
>>>>>  faultDetail:
>>>>> {http://xml.apache.org/axis/}stackTrace:java.lang.ArrayIndex
>>>>> OutOfBoundsException: 41
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte
>>>>> (NTLMEngineImpl.java:911)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addULon
>>>>> g(NTLMEngineImpl.java:941)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getRes
>>>>> ponse(NTLMEngineImpl.java:1043)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTL
>>>>> MEngineImpl.java:148)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NT
>>>>> LMEngineImpl.java:1628)
>>>>> at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme
>>>>> .java:139)
>>>>> at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSc
>>>>> hemeBase.java:138)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthe
>>>>> nticator.java:239)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResp
>>>>> onse(HttpAuthenticator.java:202)
>>>>> at org.apache.http.impl.execchain.MainClientExec.execute(MainCl
>>>>> ientExec.java:262)
>>>>> at org.apache.http.impl.execchain.ProtocolExec.execute(Protocol
>>>>> Exec.java:184)
>>>>> at org.apache.http.impl.execchain.RedirectExec.execute(Redirect
>>>>> Exec.java:110)
>>>>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>>>>> ernalHttpClient.java:184)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:82)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:107)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:55)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er$BackgroundHTTPThread.run(CommonsHTTPSender.java:654)
>>>>>
>>>>> {http://xml.apache.org/axis/}hostname:cg-DELL
>>>>>
>>>>> java.lang.ArrayIndexOutOfBoundsException: 41
>>>>> at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er.invoke(CommonsHTTPSender.java:263)
>>>>> at org.apache.axis.strategies.InvocationStrategy.visit(Invocati
>>>>> onStrategy.java:32)
>>>>> at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
>>>>> at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
>>>>> at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
>>>>> at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2767)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2443)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2366)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:1812)
>>>>> at com.microsoft.schemas.sharepoint.soap.ListsSoapStub.getListI
>>>>> tems(ListsSoapStub.java:1841)
>>>>> at org.apache.manifoldcf.crawler.connectors.sharepoint.SPSProxy
>>>>> Helper.getFieldValues(SPSProxyHelper.java:2134)
>>>>> at org.apache.manifoldcf.crawler.connectors.sharepoint.SharePoi
>>>>> ntRepository.processDocuments(SharePointRepository.java:1433)
>>>>> at org.apache.manifoldcf.crawler.system.WorkerThread.run(Worker
>>>>> Thread.java:399)
>>>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 41
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte
>>>>> (NTLMEngineImpl.java:911)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addULon
>>>>> g(NTLMEngineImpl.java:941)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getRes
>>>>> ponse(NTLMEngineImpl.java:1043)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTL
>>>>> MEngineImpl.java:148)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NT
>>>>> LMEngineImpl.java:1628)
>>>>> at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme
>>>>> .java:139)
>>>>> at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSc
>>>>> hemeBase.java:138)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthe
>>>>> nticator.java:239)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResp
>>>>> onse(HttpAuthenticator.java:202)
>>>>> at org.apache.http.impl.execchain.MainClientExec.execute(MainCl
>>>>> ientExec.java:262)
>>>>> at org.apache.http.impl.execchain.ProtocolExec.execute(Protocol
>>>>> Exec.java:184)
>>>>> at org.apache.http.impl.execchain.RedirectExec.execute(Redirect
>>>>> Exec.java:110)
>>>>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>>>>> ernalHttpClient.java:184)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:82)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:107)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:55)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er$BackgroundHTTPThread.run(CommonsHTTPSender.java:654)
>>>>> DEBUG 2017-03-17 14:30:50,398 (Worker thread '0') - SharePoint: Got a
>>>>> remote exception getting field values for site  library
>>>>> {1B694C45-DF1F-44E7-9814-F5096E85A126} document [/Documents2/Step by
>>>>> step Installation of SharePoint 2013 on Windows Server 2012 R2 part 1
-
>>>>> SharePoint Community.pdf] - retrying
>>>>> AxisFault
>>>>>  faultCode: {http://schemas.xmlsoap.org/so
>>>>> ap/envelope/}Server.userException
>>>>>  faultSubcode:
>>>>>  faultString: java.lang.ArrayIndexOutOfBoundsException: 41
>>>>>  faultActor:
>>>>>  faultNode:
>>>>>  faultDetail:
>>>>> {http://xml.apache.org/axis/}stackTrace:java.lang.ArrayIndex
>>>>> OutOfBoundsException: 41
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte
>>>>> (NTLMEngineImpl.java:911)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addUSho
>>>>> rt(NTLMEngineImpl.java:933)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getRes
>>>>> ponse(NTLMEngineImpl.java:1040)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTL
>>>>> MEngineImpl.java:148)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NT
>>>>> LMEngineImpl.java:1628)
>>>>> at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme
>>>>> .java:139)
>>>>> at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSc
>>>>> hemeBase.java:138)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthe
>>>>> nticator.java:239)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResp
>>>>> onse(HttpAuthenticator.java:202)
>>>>> at org.apache.http.impl.execchain.MainClientExec.execute(MainCl
>>>>> ientExec.java:262)
>>>>> at org.apache.http.impl.execchain.ProtocolExec.execute(Protocol
>>>>> Exec.java:184)
>>>>> at org.apache.http.impl.execchain.RedirectExec.execute(Redirect
>>>>> Exec.java:110)
>>>>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>>>>> ernalHttpClient.java:184)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:82)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:107)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:55)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er$BackgroundHTTPThread.run(CommonsHTTPSender.java:654)
>>>>>
>>>>> {http://xml.apache.org/axis/}hostname:cg-DELL
>>>>>
>>>>> java.lang.ArrayIndexOutOfBoundsException: 41
>>>>> at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er.invoke(CommonsHTTPSender.java:263)
>>>>> at org.apache.axis.strategies.InvocationStrategy.visit(Invocati
>>>>> onStrategy.java:32)
>>>>> at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
>>>>> at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
>>>>> at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)
>>>>> at org.apache.axis.client.Call.invokeEngine(Call.java:2784)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2767)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2443)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:2366)
>>>>> at org.apache.axis.client.Call.invoke(Call.java:1812)
>>>>> at com.microsoft.schemas.sharepoint.soap.ListsSoapStub.getListI
>>>>> tems(ListsSoapStub.java:1841)
>>>>> at org.apache.manifoldcf.crawler.connectors.sharepoint.SPSProxy
>>>>> Helper.getFieldValues(SPSProxyHelper.java:2134)
>>>>> at org.apache.manifoldcf.crawler.connectors.sharepoint.SharePoi
>>>>> ntRepository.processDocuments(SharePointRepository.java:1433)
>>>>> at org.apache.manifoldcf.crawler.system.WorkerThread.run(Worker
>>>>> Thread.java:399)
>>>>> Caused by: java.lang.ArrayIndexOutOfBoundsException: 41
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addByte
>>>>> (NTLMEngineImpl.java:911)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$NTLMMessage.addUSho
>>>>> rt(NTLMEngineImpl.java:933)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl$Type1Message.getRes
>>>>> ponse(NTLMEngineImpl.java:1040)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.getType1Message(NTL
>>>>> MEngineImpl.java:148)
>>>>> at org.apache.http.impl.auth.NTLMEngineImpl.generateType1Msg(NT
>>>>> LMEngineImpl.java:1628)
>>>>> at org.apache.http.impl.auth.NTLMScheme.authenticate(NTLMScheme
>>>>> .java:139)
>>>>> at org.apache.http.impl.auth.AuthSchemeBase.authenticate(AuthSc
>>>>> hemeBase.java:138)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.doAuth(HttpAuthe
>>>>> nticator.java:239)
>>>>> at org.apache.http.impl.auth.HttpAuthenticator.generateAuthResp
>>>>> onse(HttpAuthenticator.java:202)
>>>>> at org.apache.http.impl.execchain.MainClientExec.execute(MainCl
>>>>> ientExec.java:262)
>>>>> at org.apache.http.impl.execchain.ProtocolExec.execute(Protocol
>>>>> Exec.java:184)
>>>>> at org.apache.http.impl.execchain.RedirectExec.execute(Redirect
>>>>> Exec.java:110)
>>>>> at org.apache.http.impl.client.InternalHttpClient.doExecute(Int
>>>>> ernalHttpClient.java:184)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:82)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:107)
>>>>> at org.apache.http.impl.client.CloseableHttpClient.execute(Clos
>>>>> eableHttpClient.java:55)
>>>>> at org.apache.manifoldcf.connectorcommon.common.CommonsHTTPSend
>>>>> er$BackgroundHTTPThread.run(CommonsHTTPSender.java:654)
>>>>>  WARN 2017-03-17 14:30:50,404 (Worker thread '4') - Service
>>>>> interruption reported for job 1489681799610 connection 'sp2013 crawl':
>>>>> Remote procedure exception: ; nested exception is:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 41
>>>>>  WARN 2017-03-17 14:30:50,405 (Worker thread '0') - Service
>>>>> interruption reported for job 1489681799610 connection 'sp2013 crawl':
>>>>> Remote procedure exception: ; nested exception is:
>>>>> java.lang.ArrayIndexOutOfBoundsException: 41
>>>>> DEBUG 2017-03-17 14:31:09,041 (Worker thread '2') - SharePoint:
>>>>> getSites xml response: <ns1:Webs xmlns:ns1="http://schemas.micr
>>>>> osoft.com/sharepoint/soap/"/>
>>>>> DEBUG 2017-03-17 14:31:09,043 (Worker thread '2') - SharePoint: In
>>>>> getDocumentLibraries; parentSite='', parentSiteDecoded=''
>>>>>
>>>>> --
>>>>> Teşekkürler
>>>>> Cihad Güzel
>>>>>
>>>>
>>>>
>>>
>

Mime
View raw message