Hello,
We are running Tomcat 6.0.26 on Windows Server 2008 R2 Enterprise 64 Bit us=
ing IIS and isapi redirector version 1.2.30 (just upgraded to 1.2.31). The=
6 months or so we have been using Tomcat have been plagued with intermitt=
ent periods crashes and instability. Specifically, tomcat requests are res=
ulting in crashes and the application log shows numerous (over 1200) instan=
ces of error 1000:
Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0=
x4a5bd0eb
Faulting module name: ntdll.dll, version: 6.1.7600.16385, time stamp: 0x4a5=
be02b
Exception code: 0xc0000005
Fault offset: 0x000000000004d174
Faulting process id: 0x1254
Faulting application start time: 0x01cbc19aceedc9cd
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll
Report Id: 0d413cc2-2d8e-11e0-92d0-005056340288
We installed debug diag on the machine and opened a case with Microsoft sin=
ce it seemed like it was a MS problem. After opening a case and sending MS=
several dump files for analysis we were told:
We are failing during a call into isapi_redirect64. It appears to be passi=
ng a bad value for a critical section to NTDLL (0x00000000`00000003) which =
in turn is causing the Access Violation to occur.
Here is the complete email response from MS with some of the data from the =
dumps:
All of the dumps show the same issue:
Child-SP RetAddr Call Site
00000000`019bd040 00000000`7734d0a2 ntdll!RtlpWaitOnCriticalSection+0xb4
00000000`019bd0f0 00000000`6a6d1c41 ntdll!RtlEnterCriticalSection+0xab
00000000`019bd120 00000000`6a6bd40c isapi_redirect64!jk_md5+0x1c1 00000000`=
019bd150 00000000`6a6ebbb7 isapi_redirect64!ajp_service+0xb4c
00000000`019bd1e0 00000000`6a6c679a isapi_redirect64!uri_worker_map_open+0x=
2c7
00000000`019bd240 000007fe`f31bc540 isapi_redirect64!read_registry_init_dat=
a+0x89a
00000000`019be520 000007fe`f31c4007 isapi!ProcessIsapiRequest+0x13c 0000000=
0`019be580 000007fe`f31bb421 isapi!W3_ISAPI_HANDLER::DoWork+0x4fb
00000000`019be690 000007fe`f31bb964 isapi!RequestDoWork+0x40d 00000000`019b=
e720 000007fe`f3db2fa7 isapi!CIISHttpModule::OnExecuteRequestHandler+0x1c
00000000`019be760 000007fe`f3db464e iiscore!NOTIFICATION_CONTEXT::RequestDo=
Work+0x233
00000000`019be7a0 000007fe`f3db4465 iiscore!NOTIFICATION_CONTEXT::CallModul=
esInternal+0x17f
00000000`019be890 000007fe`f3db5e56 iiscore!NOTIFICATION_CONTEXT::CallModul=
es+0x25
00000000`019be8e0 000007fe`f3de51bd iiscore!W3_CONTEXT::DoWork+0x3a6 000000=
00`019bec00 000007fe`f3de55df iiscore!W3_MAIN_CONTEXT::StartNotificationLoo=
p+0x5d
00000000`019bec30 000007fe`f319551e iiscore!W3_CONTEXT::ExecuteRequest+0x2b=
f
00000000`019becb0 000007fe`f3191060 filter!RequestDoWork+0x40be 00000000`01=
9bf360 000007fe`f3db6ac1 filter!CIISHttpModule::OnPostAuthenticateRequest+0=
x20
00000000`019bf3a0 000007fe`f3db464e iiscore!NOTIFICATION_CONTEXT::RequestDo=
Work+0x20a
00000000`019bf3e0 000007fe`f3db4465 iiscore!NOTIFICATION_CONTEXT::CallModul=
esInternal+0x17f
00000000`019bf4d0 000007fe`f3db593d iiscore!NOTIFICATION_CONTEXT::CallModul=
es+0x25
00000000`019bf520 000007fe`f3db8ede iiscore!W3_CONTEXT::DoWork+0x326 000000=
00`019bf840 000007fe`f3da10d2 iiscore!W3_MAIN_CONTEXT::OnNewRequest+0x1ea
00000000`019bf870 000007fe`f3da109c w3dt!UL_NATIVE_REQUEST::DoWork+0x126
00000000`019bf8d0 000007fe`f4581fba w3dt!OverlappedCompletionRoutine+0x1c
00000000`019bf900 000007fe`f4582024 w3tp!THREAD_POOL_DATA::ThreadPoolThread=
+0x7a
00000000`019bf950 000007fe`f45820a1 w3tp!THREAD_POOL_DATA::ThreadPoolThread=
+0x34
00000000`019bf980 00000000`770ff56d w3tp!THREAD_MANAGER::ThreadManagerThrea=
d+0x61
00000000`019bf9b0 00000000`77333281 kernel32!BaseThreadInitThunk+0xd 000000=
00`019bf9e0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
We are failing during a call into isapi_redirect64. It appears to be passi=
ng a bad value for a critical section to NTDLL (0x00000000`00000003) which =
in turn is causing the Access Violation to occur.
This is a TomCat redirector from the Apache Foundation:
0:004> lmvm isapi_redirect64
start end module name
00000000`6a6b0000 00000000`6a714000 isapi_redirect64 (export symbols) =
isapi_redirect64.dll
Loaded symbol image file: isapi_redirect64.dll
Image path: D:\isapiRedirect\isapi_redirect64.dll
Image name: isapi_redirect64.dll
Timestamp: Thu Sep 09 18:04:53 2010 (4C895A05)
CheckSum: 00062701
ImageSize: 00064000
File version: 1.2.30.0
Product version: 1.2.30.0
File flags: 0 (Mask 3F)
File OS: 40004 NT Win32
File type: 1.0 App
File date: 00000000.00000000
Translations: 0409.04b0
CompanyName: Apache Software Foundation
ProductName: Apache Tomcat Connectors project
InternalName: isapi_redirect-1.2.30
OriginalFilename: isapi_redirect-1.2.30.dll
ProductVersion: 1.2.30
FileVersion: 1.2.30
FileDescription: Apache Tomcat IIS Redirector
You will need to follow up with the vendor of that product for further deta=
ils on what is causing the error.
----end MS response-----------
We have not been able to figure out what is causing this problem. It is in=
termittent - we have gone as long as a month between incidents and had it h=
appen 5 or 6 times in a day. Searching the list archives we found severa=
l instances of people with similar problems, but none with solutions. We a=
lso found an one issue in ASF Bugzilla which appears similar:
https://issues.apache.org/bugzilla/show_bug.cgi?id=3D45063
However, we are using the same identity ("ApplicationPoolIdentity") to run =
all of our app pools, so that one doesn't quite match. We are also on diff=
erent version of windows/IIS.
We have just upgraded the redirector to version 1.2.31. Based on the chang=
e log for that version we don't expect that it will fix the problem but we =
will monitor and report if it does.
Any help would be appreciated.
|