directory-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Daniel.vanMa...@bosch-si.com>
Subject ApacheDS crashes on Startup after Backup-Restoration
Date Wed, 17 Oct 2012 11:40:47 GMT
Greetings,

We are using ApacheDS as LDAP provider for one of our Solutions (for user authentication).
Recently one of our ApacheDS Servers experienced a database corruption - we are not sure why.
The ApacheDS server booted up correctly, but when some client tried to retrieve a list of
all users (for instance the Apache Directory Studio), the Server throws a  exception and closes
the socket to the client:

[05:40:05] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected exception
forcing session to close: sending disconnect notice to client.
java.lang.Error: ERR_546 CRITICAL: page header magic for block 64 not OK 0
        at jdbm.recman.PageHeader.<init>(PageHeader.java:95)
        at jdbm.recman.PageHeader.getView(PageHeader.java:124)
        at jdbm.recman.PageManager.getNext(PageManager.java:234)
        at jdbm.recman.PageCursor.next(PageCursor.java:104)
        at jdbm.recman.PhysicalRowIdManager.fetch(PhysicalRowIdManager.java:158)
        at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:324)
        at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:262)
        at jdbm.btree.BPage.loadBPage(BPage.java:899)
        at jdbm.btree.BPage.childBPage(BPage.java:890)
        at jdbm.btree.BPage.find(BPage.java:284)
        at jdbm.btree.BTree.find(BTree.java:408)
        at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:395)
        at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmMasterTable.get(JdbmMasterTable.java:155)
        at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmStore.lookup(JdbmStore.java:1332)
        at org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmStore.lookup(JdbmStore.java:70)
        at org.apache.directory.server.xdbm.AbstractXdbmPartition.lookup(AbstractXdbmPartition.java:327)
        at org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:139)
        at org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:39)
        at org.apache.directory.server.core.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:503)
        at org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:314)
        at org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:749)
        at org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:978)
        at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1054)
        at org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:78)
        at org.apache.directory.server.ldap.handlers.ReferralAwareRequestHandler.handle(ReferralAwareRequestHandler.java:94)
        at org.apache.directory.server.ldap.handlers.ReferralAwareRequestHandler.handle(ReferralAwareRequestHandler.java:57)
        at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:208)
        at org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:58)
        at org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
        at org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:193)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:713)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
        at org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:71)
        at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
        at org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.runTask(UnorderedThreadPoolExecutor.java:480)
        at org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.run(UnorderedThreadPoolExecutor.java:434)
        at java.lang.Thread.run(Thread.java:662)
[05:40:05] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] - Unexpected exception
forcing session to close: sending disconnect notice to client.
org.apache.mina.core.write.WriteToClosedSessionException
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.clearWriteRequestQueue(AbstractPollingIoProcessor.java:573)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.removeNow(AbstractPollingIoProcessor.java:525)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.removeSessions(AbstractPollingIoProcessor.java:497)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:61)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:974)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

We do not think that there is a way to repair this corruption, therefore we tried to restore
a backup (we are using Tivoli for backup of our SAN storage).
We were able to determine the Point in Time when the corruption took place, as we use a oracle
database job which fails in case the LDAP query fails.
After trying to restore to the point in time when the Database was still working we experienced
a new Problem:
The ApacheDS Server fails to boot up - it crashes with the following Stack trace:

[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version information
: assuming version: 1
[11:27:59] ERROR [org.apache.directory.daemon.Bootstrapper] - Failed on null.init(InstallationLayout,
String[])
java.lang.NullPointerException
        at org.apache.directory.server.core.entry.ClonedServerEntry.<init>(ClonedServerEntry.java:67)
        at org.apache.directory.server.xdbm.AbstractXdbmPartition.lookup(AbstractXdbmPartition.java:327)
        at org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:139)
        at org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:39)
        at org.apache.directory.server.core.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:503)
        at org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:147)
        at org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:111)
        at org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:202)
        at org.apache.directory.server.core.interceptor.InterceptorChain.register0(InterceptorChain.java:442)
        at org.apache.directory.server.core.interceptor.InterceptorChain.register(InterceptorChain.java:398)
        at org.apache.directory.server.core.interceptor.InterceptorChain.init(InterceptorChain.java:258)
        at org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1447)
        at org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:907)
        at org.apache.directory.server.configuration.ApacheDS.startup(ApacheDS.java:163)
        at org.apache.directory.server.Service.initLdap(Service.java:136)
        at org.apache.directory.server.Service.init(Service.java:77)
        at org.apache.directory.daemon.Bootstrapper.callInit(Bootstrapper.java:154)
        at org.apache.directory.daemon.TanukiBootstrapper.start(TanukiBootstrapper.java:54)
        at org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)

I failed to archive any information in the internet about this szenario.
Does someone know what the problem is?
May the Tivoli backup restore some inconsistent state?

Is there any chance to rescue the data of this LDAP server?
Good thing it was no Productive environment this time - only QA, but it would be great to
solve this in order to know how to solve it the next time.. ;-) (if there is a way to solve
it)

Some additional information about our Environment: We are using Red Head Enterprise Linux
release 5.8 (x86_64) on the impacted server.
ApacheDS is running on the embedded Java Virtual Machine


Mit freundlichen Grüßen / Best regards

Daniel van Maren

Bosch Software Innovations GmbH
Stuttgarter Str. 130,
71332 Waiblingen
GERMANY
www.bosch-si.de

daniel.vanmaren@bosch-si.com<mailto:nikolaj.langner@bosch-si.com>

Registered office: Immenstaad, Register court: Amtsgericht Ulm, HRB 631888;
Executives: Heinz Derenbach; Erica Fölsche, Thomas Cotic, Michael Hahn, Klaus Hüftle


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message