lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Policeman Jenkins Server <jenk...@thetaphi.de>
Subject [JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_15) - Build # 4812 - Still Failing!
Date Thu, 21 Mar 2013 19:22:07 GMT
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/4812/
Java: 32bit/jdk1.7.0_15 -server -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1614, name=coreLoadExecutor-597-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1614, name=coreLoadExecutor-597-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1567)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:524)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:991)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:883)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:875)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:746)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
	at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a41104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b29f12))
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	... 1 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.RecoveryZkTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1616, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1616, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
	at java.lang.Thread.getStackTrace(Thread.java:1567)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:524)
	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:991)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:883)
	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:875)
	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a))
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
	at java.lang.Thread.run(Thread.java:722)




Build Log:
[...truncated 9196 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.RecoveryZkTest
[junit4:junit4]   2> 0 T1555 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /_ydt/xa
[junit4:junit4]   2> 4 T1555 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-RecoveryZkTest-1363893210418
[junit4:junit4]   2> 5 T1555 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5 T1556 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 8 T1556 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 8 T1556 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 8 T1556 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 9 T1556 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 10 T1556 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-1363893210417/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 105 T1555 oasc.ZkTestServer.run start zk server on port:45653
[junit4:junit4]   2> 106 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18a2dc4
[junit4:junit4]   2> 107 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 107 T1561 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 107 T1561 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 108 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45516
[junit4:junit4]   2> 108 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45516
[junit4:junit4]   2> 108 T1559 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 211 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380000 with negotiated timeout 10000 for client /127.0.0.1:45516
[junit4:junit4]   2> 211 T1561 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380000, negotiated timeout = 10000
[junit4:junit4]   2> 212 T1562 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18a2dc4 name:ZooKeeperConnection Watcher:127.0.0.1:45653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 212 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 212 T1555 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 215 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380000
[junit4:junit4]   2> 216 T1562 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 216 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45516 which had sessionid 0x13d8e5ecd380000
[junit4:junit4]   2> 216 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380000 closed
[junit4:junit4]   2> 217 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@144d967
[junit4:junit4]   2> 218 T1563 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 217 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 218 T1563 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 218 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45517
[junit4:junit4]   2> 218 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45517
[junit4:junit4]   2> 219 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380001 with negotiated timeout 10000 for client /127.0.0.1:45517
[junit4:junit4]   2> 219 T1563 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380001, negotiated timeout = 10000
[junit4:junit4]   2> 219 T1564 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@144d967 name:ZooKeeperConnection Watcher:127.0.0.1:45653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 220 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 220 T1555 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 222 T1555 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 224 T1555 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 226 T1555 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 228 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 228 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 231 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 232 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 235 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 235 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 237 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 238 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 240 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 240 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 242 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 243 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 245 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 245 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 248 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 248 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 250 T1555 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 250 T1555 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 253 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380001
[junit4:junit4]   2> 253 T1564 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 253 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45517 which had sessionid 0x13d8e5ecd380001
[junit4:junit4]   2> 253 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380001 closed
[junit4:junit4]   2> 307 T1555 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 311 T1555 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49567
[junit4:junit4]   2> 312 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 312 T1555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 312 T1555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666
[junit4:junit4]   2> 313 T1555 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/solr.xml
[junit4:junit4]   2> 313 T1555 oasc.CoreContainer.<init> New CoreContainer 18902728
[junit4:junit4]   2> 313 T1555 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/'
[junit4:junit4]   2> 314 T1555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/'
[junit4:junit4]   2> 330 T1555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 331 T1555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 331 T1555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 331 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 332 T1555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 332 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 332 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 333 T1555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 333 T1555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 333 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 339 T1555 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 348 T1555 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45653/solr
[junit4:junit4]   2> 349 T1555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 349 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@c3eaa4
[junit4:junit4]   2> 350 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 350 T1574 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 351 T1574 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 351 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45518
[junit4:junit4]   2> 352 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45518
[junit4:junit4]   2> 353 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380002 with negotiated timeout 20000 for client /127.0.0.1:45518
[junit4:junit4]   2> 353 T1574 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380002, negotiated timeout = 20000
[junit4:junit4]   2> 353 T1575 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c3eaa4 name:ZooKeeperConnection Watcher:127.0.0.1:45653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 353 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 354 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380002
[junit4:junit4]   2> 355 T1575 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 355 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45518 which had sessionid 0x13d8e5ecd380002
[junit4:junit4]   2> 355 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380002 closed
[junit4:junit4]   2> 355 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 358 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@292ab4
[junit4:junit4]   2> 359 T1576 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 359 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 359 T1576 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 359 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45519
[junit4:junit4]   2> 359 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45519
[junit4:junit4]   2> 360 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380003 with negotiated timeout 20000 for client /127.0.0.1:45519
[junit4:junit4]   2> 360 T1576 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380003, negotiated timeout = 20000
[junit4:junit4]   2> 361 T1577 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@292ab4 name:ZooKeeperConnection Watcher:127.0.0.1:45653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 361 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 362 T1555 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 364 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x7 zxid:0x1b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 365 T1555 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 367 T1555 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 369 T1555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49567__ydt%2Fxa
[junit4:junit4]   2> 369 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:delete cxid:0x12 zxid:0x1e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49567__ydt%2Fxa Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49567__ydt%2Fxa
[junit4:junit4]   2> 370 T1555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49567__ydt%2Fxa
[junit4:junit4]   2> 372 T1555 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 377 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:delete cxid:0x20 zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 378 T1555 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 380 T1555 oasc.Overseer.start Overseer (id=89384105438347267-127.0.0.1:49567__ydt%2Fxa-n_0000000000) starting
[junit4:junit4]   2> 380 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x25 zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 381 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x26 zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 382 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x28 zxid:0x27 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 382 T1555 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 384 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x2e zxid:0x29 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 385 T1579 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 386 T1555 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 388 T1555 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 389 T1555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 390 T1578 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 394 T1580 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/collection1
[junit4:junit4]   2> 395 T1580 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 395 T1580 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 396 T1580 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 397 T1580 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/collection1/'
[junit4:junit4]   2> 398 T1580 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/collection1/lib/README' to classloader
[junit4:junit4]   2> 398 T1580 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 422 T1580 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 449 T1580 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 451 T1580 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 454 T1580 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 785 T1580 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 791 T1580 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 793 T1580 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 805 T1580 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 806 T664 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 808 T1580 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 810 T1580 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 811 T1580 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 811 T1580 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 812 T1580 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 812 T1580 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 812 T1580 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 813 T1580 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 813 T1580 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-controljetty-1363893210666/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data/
[junit4:junit4]   2> 813 T1580 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@578681
[junit4:junit4]   2> 813 T1580 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 814 T1580 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data forceNew: false
[junit4:junit4]   2> 814 T1580 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data/index/
[junit4:junit4]   2> 814 T1580 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 814 T1580 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data/index forceNew: false
[junit4:junit4]   2> 815 T1580 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@180ee0e lockFactory=org.apache.lucene.store.NativeFSLockFactory@7de54f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 816 T1580 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 817 T1580 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 817 T1580 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 817 T1580 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 818 T1580 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 818 T1580 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 818 T1580 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 818 T1580 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 819 T1580 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 819 T1580 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 821 T1580 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 823 T1580 oass.SolrIndexSearcher.<init> Opening Searcher@1681c61 main
[junit4:junit4]   2> 823 T1580 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/control/data/tlog
[junit4:junit4]   2> 823 T1580 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 823 T1580 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 826 T1581 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1681c61 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 827 T1580 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 827 T1580 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1893 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 1894 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49567__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49567/_ydt/xa"}
[junit4:junit4]   2> 1894 T1578 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 1894 T1578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 1899 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2526 T672 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 2829 T1580 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2829 T1580 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49567/_ydt/xa collection:control_collection shard:shard1
[junit4:junit4]   2> 2830 T1580 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2836 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:delete cxid:0x82 zxid:0x35 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2836 T1580 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2837 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x83 zxid:0x36 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2839 T1580 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2839 T1580 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2839 T1580 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49567/_ydt/xa/collection1/
[junit4:junit4]   2> 2839 T1580 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2840 T1580 oasc.SyncStrategy.syncToMe http://127.0.0.1:49567/_ydt/xa/collection1/ has no replicas
[junit4:junit4]   2> 2840 T1580 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49567/_ydt/xa/collection1/
[junit4:junit4]   2> 2840 T1580 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2844 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380003 type:create cxid:0x8e zxid:0x3a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3404 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3415 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3448 T1580 oasc.ZkController.register We are http://127.0.0.1:49567/_ydt/xa/collection1/ and leader is http://127.0.0.1:49567/_ydt/xa/collection1/
[junit4:junit4]   2> 3449 T1580 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49567/_ydt/xa
[junit4:junit4]   2> 3449 T1580 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3449 T1580 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3450 T1580 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3452 T1580 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 3454 T1555 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3455 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3455 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3462 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3463 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@56b9d2
[junit4:junit4]   2> 3465 T1583 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3464 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3465 T1583 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 3465 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45524
[junit4:junit4]   2> 3466 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45524
[junit4:junit4]   2> 3467 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380004 with negotiated timeout 10000 for client /127.0.0.1:45524
[junit4:junit4]   2> 3467 T1583 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380004, negotiated timeout = 10000
[junit4:junit4]   2> 3468 T1584 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56b9d2 name:ZooKeeperConnection Watcher:127.0.0.1:45653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3468 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3470 T1555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 3473 T1555 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 3536 T1555 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 3538 T1555 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:48809
[junit4:junit4]   2> 3540 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 3540 T1555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 3541 T1555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887
[junit4:junit4]   2> 3542 T1555 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/solr.xml
[junit4:junit4]   2> 3542 T1555 oasc.CoreContainer.<init> New CoreContainer 31756692
[junit4:junit4]   2> 3543 T1555 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/'
[junit4:junit4]   2> 3544 T1555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/'
[junit4:junit4]   2> 3567 T1555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 3567 T1555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 3568 T1555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 3568 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 3568 T1555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 3568 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 3569 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 3569 T1555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 3569 T1555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 3570 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 3575 T1555 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 3584 T1555 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45653/solr
[junit4:junit4]   2> 3584 T1555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 3584 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@298d8d
[junit4:junit4]   2> 3585 T1594 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3585 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3586 T1594 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 3586 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45525
[junit4:junit4]   2> 3586 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45525
[junit4:junit4]   2> 3587 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380005 with negotiated timeout 20000 for client /127.0.0.1:45525
[junit4:junit4]   2> 3587 T1594 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380005, negotiated timeout = 20000
[junit4:junit4]   2> 3588 T1595 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@298d8d name:ZooKeeperConnection Watcher:127.0.0.1:45653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3588 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3590 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380005
[junit4:junit4]   2> 3591 T1595 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 3591 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45525 which had sessionid 0x13d8e5ecd380005
[junit4:junit4]   2> 3591 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380005 closed
[junit4:junit4]   2> 3591 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 3597 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@c7fb81
[junit4:junit4]   2> 3598 T1596 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 3598 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3599 T1596 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 3599 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45526
[junit4:junit4]   2> 3599 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45526
[junit4:junit4]   2> 3600 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380006 with negotiated timeout 20000 for client /127.0.0.1:45526
[junit4:junit4]   2> 3600 T1596 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380006, negotiated timeout = 20000
[junit4:junit4]   2> 3600 T1597 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c7fb81 name:ZooKeeperConnection Watcher:127.0.0.1:45653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3600 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3601 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:create cxid:0x1 zxid:0x48 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3602 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:create cxid:0x3 zxid:0x49 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3605 T1555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 4152 T680 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 4608 T1555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48809__ydt%2Fxa
[junit4:junit4]   2> 4609 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:delete cxid:0xd zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:48809__ydt%2Fxa Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:48809__ydt%2Fxa
[junit4:junit4]   2> 4610 T1555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:48809__ydt%2Fxa
[junit4:junit4]   2> 4614 T1584 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4614 T1577 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4615 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4615 T1597 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 4624 T1598 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/collection1
[junit4:junit4]   2> 4624 T1598 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 4625 T1598 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 4625 T1598 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 4627 T1598 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/collection1/'
[junit4:junit4]   2> 4628 T1598 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/collection1/lib/README' to classloader
[junit4:junit4]   2> 4629 T1598 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 4665 T1598 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 4692 T1598 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 4693 T1598 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 4696 T1598 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 4922 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 4922 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:49567__ydt%2Fxa_collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"control_collection",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:49567__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:49567/_ydt/xa"}
[junit4:junit4]   2> 4925 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4925 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 4925 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5020 T1598 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5026 T1598 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5028 T1598 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5045 T1598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5050 T1598 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5055 T1598 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5056 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5057 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5057 T1598 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5058 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5059 T1598 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5059 T1598 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5060 T1598 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty1-1363893213887/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1/
[junit4:junit4]   2> 5060 T1598 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@578681
[junit4:junit4]   2> 5061 T1598 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5061 T1598 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1 forceNew: false
[junit4:junit4]   2> 5062 T1598 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1/index/
[junit4:junit4]   2> 5062 T1598 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5063 T1598 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1/index forceNew: false
[junit4:junit4]   2> 5065 T1598 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cc538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cece7d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 5065 T1598 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 5068 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 5069 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 5069 T1598 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 5070 T1598 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 5071 T1598 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 5071 T1598 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 5072 T1598 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 5073 T1598 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 5073 T1598 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 5078 T1598 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 5082 T1598 oass.SolrIndexSearcher.<init> Opening Searcher@314e46 main
[junit4:junit4]   2> 5083 T1598 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty1/tlog
[junit4:junit4]   2> 5084 T1598 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 5084 T1598 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 5091 T1599 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@314e46 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 5093 T1598 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 5093 T1598 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 5787 T686 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 6429 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 6431 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:48809__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:48809/_ydt/xa"}
[junit4:junit4]   2> 6431 T1578 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6431 T1578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6455 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6455 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6455 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7096 T1598 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7096 T1598 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:48809/_ydt/xa collection:collection1 shard:shard1
[junit4:junit4]   2> 7098 T1598 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7107 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:delete cxid:0x4a zxid:0x5a txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7108 T1598 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7109 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:create cxid:0x4b zxid:0x5b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7112 T1598 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7112 T1598 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7113 T1598 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:48809/_ydt/xa/collection1/
[junit4:junit4]   2> 7113 T1598 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7114 T1598 oasc.SyncStrategy.syncToMe http://127.0.0.1:48809/_ydt/xa/collection1/ has no replicas
[junit4:junit4]   2> 7115 T1598 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:48809/_ydt/xa/collection1/
[junit4:junit4]   2> 7115 T1598 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7121 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380006 type:create cxid:0x56 zxid:0x5f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7416 T696 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 7960 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7966 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7966 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7966 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7977 T1598 oasc.ZkController.register We are http://127.0.0.1:48809/_ydt/xa/collection1/ and leader is http://127.0.0.1:48809/_ydt/xa/collection1/
[junit4:junit4]   2> 7977 T1598 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:48809/_ydt/xa
[junit4:junit4]   2> 7977 T1598 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 7977 T1598 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 7977 T1598 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7978 T1598 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 7979 T1555 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 7980 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 7980 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8064 T1555 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8068 T1555 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36542
[junit4:junit4]   2> 8070 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8072 T1555 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8073 T1555 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397
[junit4:junit4]   2> 8073 T1555 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/solr.xml
[junit4:junit4]   2> 8074 T1555 oasc.CoreContainer.<init> New CoreContainer 7219324
[junit4:junit4]   2> 8074 T1555 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/'
[junit4:junit4]   2> 8075 T1555 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/'
[junit4:junit4]   2> 8101 T1555 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8102 T1555 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8103 T1555 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8103 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8104 T1555 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8104 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8104 T1555 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8105 T1555 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8105 T1555 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8106 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8112 T1555 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8123 T1555 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:45653/solr
[junit4:junit4]   2> 8124 T1555 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8124 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@10bae8f
[junit4:junit4]   2> 8125 T1610 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 8125 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8127 T1610 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 8127 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45564
[junit4:junit4]   2> 8129 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45564
[junit4:junit4]   2> 8132 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380007 with negotiated timeout 20000 for client /127.0.0.1:45564
[junit4:junit4]   2> 8132 T1610 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380007, negotiated timeout = 20000
[junit4:junit4]   2> 8132 T1611 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10bae8f name:ZooKeeperConnection Watcher:127.0.0.1:45653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8132 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8134 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380007
[junit4:junit4]   2> 8134 T1611 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8134 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45564 which had sessionid 0x13d8e5ecd380007
[junit4:junit4]   2> 8134 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380007 closed
[junit4:junit4]   2> 8135 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8139 T1555 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:45653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@123809
[junit4:junit4]   2> 8140 T1612 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost.localdomain/127.0.0.1:45653. Will not attempt to authenticate using SASL (access denied ("javax.security.auth.AuthPermission" "getLoginConfiguration"))
[junit4:junit4]   2> 8140 T1555 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8140 T1612 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost.localdomain/127.0.0.1:45653, initiating session
[junit4:junit4]   2> 8141 T1557 oazs.NIOServerCnxnFactory.run Accepted socket connection from /127.0.0.1:45565
[junit4:junit4]   2> 8141 T1557 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /127.0.0.1:45565
[junit4:junit4]   2> 8142 T1559 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d8e5ecd380008 with negotiated timeout 20000 for client /127.0.0.1:45565
[junit4:junit4]   2> 8142 T1612 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost.localdomain/127.0.0.1:45653, sessionid = 0x13d8e5ecd380008, negotiated timeout = 20000
[junit4:junit4]   2> 8142 T1613 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@123809 name:ZooKeeperConnection Watcher:127.0.0.1:45653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8142 T1555 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8143 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380008 type:create cxid:0x1 zxid:0x6c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8144 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380008 type:create cxid:0x3 zxid:0x6d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8146 T1555 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper... 
[junit4:junit4]   2> 9042 T704 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 9149 T1555 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36542__ydt%2Fxa
[junit4:junit4]   2> 9150 T1560 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d8e5ecd380008 type:delete cxid:0xd zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:36542__ydt%2Fxa Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:36542__ydt%2Fxa
[junit4:junit4]   2> 9150 T1555 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36542__ydt%2Fxa
[junit4:junit4]   2> 9155 T1597 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9155 T1613 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9155 T1577 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9155 T1584 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9155 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9156 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9156 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9164 T1614 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/collection1
[junit4:junit4]   2> 9165 T1614 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9166 T1614 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9166 T1614 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9168 T1614 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/collection1/'
[junit4:junit4]   2> 9169 T1614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/collection1/lib/README' to classloader
[junit4:junit4]   2> 9169 T1614 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9205 T1614 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 9234 T1614 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9235 T1614 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9239 T1614 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9470 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 9471 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:48809__ydt%2Fxa_collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:48809__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:48809/_ydt/xa"}
[junit4:junit4]   2> 9474 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9474 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9474 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9474 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9552 T1614 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 9558 T1614 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 9560 T1614 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 9575 T1614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9579 T1614 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 9583 T1614 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9584 T1614 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 9585 T1614 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 9585 T1614 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 9586 T1614 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 9586 T1614 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 9587 T1614 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 9587 T1614 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.RecoveryZkTest-jetty2-1363893218397/collection1/, dataDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/
[junit4:junit4]   2> 9587 T1614 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@578681
[junit4:junit4]   2> 9588 T1614 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 9589 T1614 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2 forceNew: false
[junit4:junit4]   2> 9589 T1614 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index/
[junit4:junit4]   2> 9590 T1614 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 9590 T1614 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index forceNew: false
[junit4:junit4]   2> 9592 T1614 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@977832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@108dd54),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 9592 T1614 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 9594 T1614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 9594 T1614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 9595 T1614 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 9596 T1614 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 9596 T1614 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 9597 T1614 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 9597 T1614 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 9598 T1614 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 9598 T1614 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 9602 T1614 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 9605 T1614 oass.SolrIndexSearcher.<init> Opening Searcher@109a954 main
[junit4:junit4]   2> 9605 T1614 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/tlog
[junit4:junit4]   2> 9606 T1614 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 9606 T1614 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 9612 T1615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@109a954 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 9614 T1614 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 9615 T1614 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 10668 T712 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 10979 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 10980 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":null,
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":null,
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36542__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36542/_ydt/xa"}
[junit4:junit4]   2> 10980 T1578 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 10981 T1578 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 10987 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10987 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10987 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10987 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11617 T1614 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11617 T1614 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36542/_ydt/xa collection:collection1 shard:shard1
[junit4:junit4]   2> 11634 T1614 oasc.ZkController.register We are http://127.0.0.1:36542/_ydt/xa/collection1/ and leader is http://127.0.0.1:48809/_ydt/xa/collection1/
[junit4:junit4]   2> 11635 T1614 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36542/_ydt/xa
[junit4:junit4]   2> 11635 T1614 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 11635 T1614 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C55 name=collection1 org.apache.solr.core.SolrCore@4e340 url=http://127.0.0.1:36542/_ydt/xa/collection1 node=127.0.0.1:36542__ydt%2Fxa C55_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:36542__ydt%2Fxa, base_url=http://127.0.0.1:36542/_ydt/xa}
[junit4:junit4]   2> 11636 T1616 C55 P36542 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 11637 T1614 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 11638 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 11638 T1616 C55 P36542 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 11638 T1616 C55 P36542 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11639 T1555 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 11640 T1616 C55 P36542 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11640 T1555 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 11641 T1555 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11647 T1555 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> ASYNC  NEW_CORE C56 name=collection1 org.apache.solr.core.SolrCore@1391190 url=http://127.0.0.1:49567/_ydt/xa/collection1 node=127.0.0.1:49567__ydt%2Fxa C56_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:49567__ydt%2Fxa, base_url=http://127.0.0.1:49567/_ydt/xa, leader=true}
[junit4:junit4]   2> 11662 T1571 C56 P49567 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@180ee0e lockFactory=org.apache.lucene.store.NativeFSLockFactory@7de54f),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11663 T1571 C56 P49567 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11666 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[1 (1430145699227369472)]} 0 7
[junit4:junit4]   2> 11666 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10001 (1430145699228418048)]} 0 7
[junit4:junit4]   2> ASYNC  NEW_CORE C57 name=collection1 org.apache.solr.core.SolrCore@fa2234 url=http://127.0.0.1:48809/_ydt/xa/collection1 node=127.0.0.1:48809__ydt%2Fxa C57_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:48809__ydt%2Fxa, base_url=http://127.0.0.1:48809/_ydt/xa, leader=true}
[junit4:junit4]   2> 11683 T1591 C57 P48809 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cc538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cece7d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11684 T1591 C57 P48809 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11688 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[1 (1430145699249389568)]} 0 8
[junit4:junit4]   2> 11688 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10001 (1430145699249389569)]} 0 8
[junit4:junit4]   2> 11694 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[1 (-1430145699264069632)]} 0 1
[junit4:junit4]   2> 11694 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[10001 (-1430145699264069633)]} 0 1
[junit4:junit4]   2> 11698 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[10001 (-1430145699268263936)]} 0 0
[junit4:junit4]   2> 11699 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[1 (-1430145699268263937)]} 0 1
[junit4:junit4]   2> 11703 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10002 (1430145699272458240)]} 0 1
[junit4:junit4]   2> 11704 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[2 (1430145699273506816)]} 0 2
[junit4:junit4]   2> 11709 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10002 (1430145699278749696)]} 0 2
[junit4:junit4]   2> 11709 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[2 (1430145699278749697)]} 0 1
[junit4:junit4]   2> 11809 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10003 (1430145699382558720)]} 0 2
[junit4:junit4]   2> 11809 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[3 (1430145699383607296)]} 0 2
[junit4:junit4]   2> 11813 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10003 (1430145699387801600)]} 0 1
[junit4:junit4]   2> 11814 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[3 (1430145699388850176)]} 0 1
[junit4:junit4]   2> 11871 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10004 (1430145699448619008)]} 0 2
[junit4:junit4]   2> 11872 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[4 (1430145699448619009)]} 0 2
[junit4:junit4]   2> 11875 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10004 (1430145699453861888)]} 0 1
[junit4:junit4]   2> 11876 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[4 (1430145699453861889)]} 0 1
[junit4:junit4]   2> 11948 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10005 (1430145699529359360)]} 0 1
[junit4:junit4]   2> 11949 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[5 (1430145699530407936)]} 0 2
[junit4:junit4]   2> 11952 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10005 (1430145699533553664)]} 0 1
[junit4:junit4]   2> 11953 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[5 (1430145699534602240)]} 0 1
[junit4:junit4]   2> 11986 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[10002 (-1430145699570253824)]} 0 0
[junit4:junit4]   2> 11986 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[2 (-1430145699570253825)]} 0 0
[junit4:junit4]   2> 11990 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[10002 (-1430145699574448128)]} 0 1
[junit4:junit4]   2> 11990 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[2 (-1430145699574448129)]} 0 0
[junit4:junit4]   2> 11994 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10006 (1430145699577593856)]} 0 1
[junit4:junit4]   2> 11995 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[6 (1430145699577593857)]} 0 2
[junit4:junit4]   2> 11998 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10006 (1430145699582836736)]} 0 1
[junit4:junit4]   2> 11999 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[6 (1430145699582836737)]} 0 1
[junit4:junit4]   2> 12059 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10007 (1430145699645751296)]} 0 1
[junit4:junit4]   2> 12060 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[7 (1430145699646799872)]} 0 1
[junit4:junit4]   2> 12063 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10007 (1430145699649945600)]} 0 1
[junit4:junit4]   2> 12064 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[7 (1430145699650994176)]} 0 1
[junit4:junit4]   2> 12075 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10008 (1430145699662528512)]} 0 2
[junit4:junit4]   2> 12075 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[8 (1430145699663577088)]} 0 1
[junit4:junit4]   2> 12079 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10008 (1430145699666722816)]} 0 1
[junit4:junit4]   2> 12080 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[8 (1430145699667771392)]} 0 1
[junit4:junit4]   2> 12106 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10009 (1430145699696082944)]} 0 1
[junit4:junit4]   2> 12107 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[9 (1430145699696082945)]} 0 1
[junit4:junit4]   2> 12111 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10009 (1430145699700277248)]} 0 1
[junit4:junit4]   2> 12112 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[9 (1430145699701325824)]} 0 1
[junit4:junit4]   2> 12197 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10010 (1430145699790454784)]} 0 1
[junit4:junit4]   2> 12198 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10 (1430145699791503360)]} 0 1
[junit4:junit4]   2> 12201 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10010 (1430145699795697664)]} 0 0
[junit4:junit4]   2> 12202 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10 (1430145699796746240)]} 0 1
[junit4:junit4]   2> 12274 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10011 (1430145699871195136)]} 0 1
[junit4:junit4]   2> 12275 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[11 (1430145699872243712)]} 0 1
[junit4:junit4]   2> 12278 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10011 (1430145699875389440)]} 0 1
[junit4:junit4]   2> 12279 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[11 (1430145699876438016)]} 0 1
[junit4:junit4]   2> 12291 T720 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 12353 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10008 (-1430145699955081216)]} 0 0
[junit4:junit4]   2> 12354 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[8 (-1430145699956129792)]} 0 1
[junit4:junit4]   2> 12357 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[10008 (-1430145699959275520)]} 0 0
[junit4:junit4]   2> 12357 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[8 (-1430145699959275521)]} 0 0
[junit4:junit4]   2> 12362 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10012 (1430145699963469824)]} 0 2
[junit4:junit4]   2> 12362 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[12 (1430145699963469825)]} 0 1
[junit4:junit4]   2> 12366 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10012 (1430145699967664128)]} 0 1
[junit4:junit4]   2> 12367 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[12 (1430145699968712704)]} 0 1
[junit4:junit4]   2> 12398 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10011 (-1430145700001218560)]} 0 1
[junit4:junit4]   2> 12399 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[11 (-1430145700003315712)]} 0 1
[junit4:junit4]   2> 12401 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[10011 (-1430145700005412864)]} 0 0
[junit4:junit4]   2> 12403 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[11 (-1430145700006461440)]} 0 1
[junit4:junit4]   2> 12406 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10013 (1430145700009607168)]} 0 1
[junit4:junit4]   2> 12407 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[13 (1430145700010655744)]} 0 1
[junit4:junit4]   2> 12411 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10013 (1430145700014850048)]} 0 1
[junit4:junit4]   2> 12411 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[13 (1430145700014850049)]} 0 1
[junit4:junit4]   2> 12478 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10013 (-1430145700086153216)]} 0 0
[junit4:junit4]   2> 12479 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[13 (-1430145700087201792)]} 0 0
[junit4:junit4]   2> 12482 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10013 (-1430145700090347520)]} 0 0
[junit4:junit4]   2> 12483 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[13 (-1430145700091396096)]} 0 1
[junit4:junit4]   2> 12487 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10014 (1430145700094541824)]} 0 1
[junit4:junit4]   2> 12487 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[14 (1430145700094541825)]} 0 1
[junit4:junit4]   2> 12491 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10014 (1430145700098736128)]} 0 1
[junit4:junit4]   2> 12492 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[14 (1430145700099784704)]} 0 1
[junit4:junit4]   2> 12494 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 12495 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:36542__ydt%2Fxa_collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"recovering",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36542__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36542/_ydt/xa"}
[junit4:junit4]   2> 12501 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10015 (1430145700109221888)]} 0 1
[junit4:junit4]   2> 12501 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12501 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12501 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12501 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12502 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[15 (1430145700110270464)]} 0 2
[junit4:junit4]   2> 12506 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10015 (1430145700114464768)]} 0 1
[junit4:junit4]   2>  C55_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:36542__ydt%2Fxa, base_url=http://127.0.0.1:36542/_ydt/xa}
[junit4:junit4]   2> 12517 T1609 C55 P36542 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@977832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@108dd54),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 12519 T1609 C55 P36542 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 12521 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[15 (1430145700115513344)]} 0 6
[junit4:junit4]   2> 12522 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[15 (1430145700115513344)]} 0 16
[junit4:junit4]   2> 12541 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10015 (-1430145700152213504)]} 0 0
[junit4:junit4]   2> 12548 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10015 (-1430145700155359232)]} 0 0
[junit4:junit4]   2> 12549 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10015 (-1430145700155359232)]} 0 5
[junit4:junit4]   2> 12553 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10016 (1430145700164796416)]} 0 1
[junit4:junit4]   2> 12557 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[15 (-1430145700168990720)]} 0 0
[junit4:junit4]   2> 12562 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10016 (1430145700168990720)]} 0 1
[junit4:junit4]   2> 12564 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10016 (1430145700168990720)]} 0 7
[junit4:junit4]   2> 12565 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[15 (-1430145700172136448)]} 0 1
[junit4:junit4]   2> 12566 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[15 (-1430145700172136448)]} 0 6
[junit4:junit4]   2> 12570 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[16 (1430145700181573632)]} 0 1
[junit4:junit4]   2> 12577 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[16 (1430145700186816512)]} 0 1
[junit4:junit4]   2> 12578 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[16 (1430145700186816512)]} 0 4
[junit4:junit4]   2> 12592 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10017 (1430145700204642304)]} 0 1
[junit4:junit4]   2> 12600 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10017 (1430145700208836608)]} 0 1
[junit4:junit4]   2> 12601 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10017 (1430145700208836608)]} 0 6
[junit4:junit4]   2> 12604 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[17 (1430145700218273792)]} 0 0
[junit4:junit4]   2> 12612 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[17 (1430145700222468096)]} 0 1
[junit4:junit4]   2> 12613 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[17 (1430145700222468096)]} 0 5
[junit4:junit4]   2> 12649 T1593 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:36542__ydt%252Fxa_collection1&state=recovering&nodeName=127.0.0.1:36542__ydt%252Fxa&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001 
[junit4:junit4]   2> 12669 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10018 (1430145700285382656)]} 0 1
[junit4:junit4]   2> 12677 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10018 (1430145700290625536)]} 0 1
[junit4:junit4]   2> 12678 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10018 (1430145700290625536)]} 0 6
[junit4:junit4]   2> 12681 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[18 (1430145700297965568)]} 0 1
[junit4:junit4]   2> 12687 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[18 (1430145700301111296)]} 0 1
[junit4:junit4]   2> 12688 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[18 (1430145700301111296)]} 0 5
[junit4:junit4]   2> 12717 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10018 (-1430145700336762880)]} 0 0
[junit4:junit4]   2> 12723 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10018 (-1430145700339908608)]} 0 0
[junit4:junit4]   2> 12724 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[10018 (-1430145700339908608)]} 0 4
[junit4:junit4]   2> 12728 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[18 (-1430145700348297216)]} 0 1
[junit4:junit4]   2> 12729 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10019 (1430145700348297217)]} 0 1
[junit4:junit4]   2> 12734 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[18 (-1430145700351442944)]} 0 0
[junit4:junit4]   2> 12735 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[18 (-1430145700351442944)]} 0 4
[junit4:junit4]   2> 12737 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10019 (1430145700353540096)]} 0 1
[junit4:junit4]   2> 12738 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10019 (1430145700353540096)]} 0 6
[junit4:junit4]   2> 12740 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[19 (1430145700359831552)]} 0 1
[junit4:junit4]   2> 12747 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[19 (1430145700364025856)]} 0 1
[junit4:junit4]   2> 12748 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[19 (1430145700364025856)]} 0 5
[junit4:junit4]   2> 12810 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10020 (1430145700434280448)]} 0 0
[junit4:junit4]   2> 12818 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10020 (1430145700438474752)]} 0 1
[junit4:junit4]   2> 12819 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[10020 (1430145700438474752)]} 0 5
[junit4:junit4]   2> 12821 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[20 (1430145700444766208)]} 0 1
[junit4:junit4]   2> 12828 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[20 (1430145700448960512)]} 0 1
[junit4:junit4]   2> 12829 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[20 (1430145700448960512)]} 0 5
[junit4:junit4]   2> 12921 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10021 (1430145700549623808)]} 0 1
[junit4:junit4]   2> 12929 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10021 (1430145700553818112)]} 0 1
[junit4:junit4]   2> 12930 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10021 (1430145700553818112)]} 0 6
[junit4:junit4]   2> 12931 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[21 (1430145700560109568)]} 0 1
[junit4:junit4]   2> 12938 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[21 (1430145700564303872)]} 0 1
[junit4:junit4]   2> 12939 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[21 (1430145700564303872)]} 0 5
[junit4:junit4]   2> 12961 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10022 (1430145700592615424)]} 0 1
[junit4:junit4]   2> 12969 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10022 (1430145700596809728)]} 0 1
[junit4:junit4]   2> 12970 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10022 (1430145700596809728)]} 0 5
[junit4:junit4]   2> 12971 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[22 (1430145700602052608)]} 0 1
[junit4:junit4]   2> 12979 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[22 (1430145700606246912)]} 0 0
[junit4:junit4]   2> 12980 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[22 (1430145700606246912)]} 0 6
[junit4:junit4]   2> 13053 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10023 (1430145700689084416)]} 0 0
[junit4:junit4]   2> 13061 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10023 (1430145700693278720)]} 0 1
[junit4:junit4]   2> 13062 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10023 (1430145700693278720)]} 0 5
[junit4:junit4]   2> 13063 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[23 (1430145700698521600)]} 0 1
[junit4:junit4]   2> 13071 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[23 (1430145700702715904)]} 0 1
[junit4:junit4]   2> 13072 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[23 (1430145700702715904)]} 0 6
[junit4:junit4]   2> 13151 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10024 (1430145700790796288)]} 0 1
[junit4:junit4]   2> 13158 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10024 (1430145700794990592)]} 0 0
[junit4:junit4]   2> 13160 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10024 (1430145700794990592)]} 0 6
[junit4:junit4]   2> 13160 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[24 (1430145700800233472)]} 0 1
[junit4:junit4]   2> 13168 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[24 (1430145700805476352)]} 0 1
[junit4:junit4]   2> 13169 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[24 (1430145700805476352)]} 0 6
[junit4:junit4]   2> 13224 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10025 (1430145700867342336)]} 0 1
[junit4:junit4]   2> 13232 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10025 (1430145700872585216)]} 0 1
[junit4:junit4]   2> 13233 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[25 (1430145700877828096)]} 0 1
[junit4:junit4]   2> 13233 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[10025 (1430145700872585216)]} 0 5
[junit4:junit4]   2> 13241 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[25 (1430145700882022400)]} 0 1
[junit4:junit4]   2> 13242 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[25 (1430145700882022400)]} 0 5
[junit4:junit4]   2> 13337 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10026 (1430145700985831424)]} 0 1
[junit4:junit4]   2> 13345 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10026 (1430145700990025728)]} 0 1
[junit4:junit4]   2> 13346 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[26 (1430145700995268608)]} 0 1
[junit4:junit4]   2> 13346 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10026 (1430145700990025728)]} 0 6
[junit4:junit4]   2> 13353 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[26 (1430145700999462912)]} 0 0
[junit4:junit4]   2> 13355 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[26 (1430145700999462912)]} 0 6
[junit4:junit4]   2> 13444 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10027 (1430145701098029056)]} 0 1
[junit4:junit4]   2> 13452 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10027 (1430145701102223360)]} 0 1
[junit4:junit4]   2> 13452 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[27 (1430145701106417664)]} 0 1
[junit4:junit4]   2> 13453 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10027 (1430145701102223360)]} 0 6
[junit4:junit4]   2> 13460 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[27 (1430145701110611968)]} 0 1
[junit4:junit4]   2> 13461 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[27 (1430145701110611968)]} 0 6
[junit4:junit4]   2> 13529 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10028 (1430145701188206592)]} 0 1
[junit4:junit4]   2> 13537 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[28 (1430145701195546624)]} 0 1
[junit4:junit4]   2> 13537 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10028 (1430145701192400896)]} 0 1
[junit4:junit4]   2> 13539 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10028 (1430145701192400896)]} 0 7
[junit4:junit4]   2> 13545 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[28 (1430145701200789504)]} 0 1
[junit4:junit4]   2> 13546 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[28 (1430145701200789504)]} 0 5
[junit4:junit4]   2> 13639 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10029 (1430145701302501376)]} 0 1
[junit4:junit4]   2> 13646 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[29 (1430145701309841408)]} 0 1
[junit4:junit4]   2> 13647 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10029 (1430145701306695680)]} 0 1
[junit4:junit4]   2> 13648 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10029 (1430145701306695680)]} 0 6
[junit4:junit4]   2> 13650 T1555 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 36542
[junit4:junit4]   2> 13651 T1555 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=7219324
[junit4:junit4]   2> 13654 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[29 (1430145701314035712)]} 0 1
[junit4:junit4]   2> 13655 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[29 (1430145701314035712)]} 0 6
[junit4:junit4]   2> 13698 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[10026 (-1430145701365415936)]} 0 1
[junit4:junit4]   2> 13704 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10026 (-1430145701368561664)]} 0 0
[junit4:junit4]   2> 13704 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[26 (-1430145701371707392)]} 0 0
[junit4:junit4]   2> 13705 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10026 (-1430145701368561664)]} 0 4
[junit4:junit4]   2> 13709 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10030 (1430145701376950272)]} 0 1
[junit4:junit4]   2> 13711 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[26 (-1430145701375901696)]} 0 0
[junit4:junit4]   2> 13712 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[26 (-1430145701375901696)]} 0 4
[junit4:junit4]   2> 13716 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[30 (1430145701383241728)]} 0 1
[junit4:junit4]   2> 13717 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10030 (1430145701381144576)]} 0 1
[junit4:junit4]   2> 13718 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10030 (1430145701381144576)]} 0 5
[junit4:junit4]   2> 13724 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[30 (1430145701387436032)]} 0 1
[junit4:junit4]   2> 13725 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[30 (1430145701387436032)]} 0 6
[junit4:junit4]   2> 13821 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[10029 (-1430145701494390784)]} 0 0
[junit4:junit4]   2> 13827 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[29 (-1430145701500682240)]} 0 0
[junit4:junit4]   2> 13828 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10029 (-1430145701498585088)]} 0 0
[junit4:junit4]   2> 13829 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10029 (-1430145701498585088)]} 0 4
[junit4:junit4]   2> 13833 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10031 (1430145701506973696)]} 0 1
[junit4:junit4]   2> 13834 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[29 (-1430145701504876544)]} 0 0
[junit4:junit4]   2> 13835 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[29 (-1430145701504876544)]} 0 4
[junit4:junit4]   2> 13839 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[31 (1430145701512216576)]} 0 1
[junit4:junit4]   2> 13841 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10031 (1430145701511168000)]} 0 1
[junit4:junit4]   2> 13842 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10031 (1430145701511168000)]} 0 5
[junit4:junit4]   2> 13847 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[31 (1430145701516410880)]} 0 1
[junit4:junit4]   2> 13848 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[31 (1430145701516410880)]} 0 6
[junit4:junit4]   2> 13927 T728 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 13936 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[10030 (-1430145701614977024)]} 0 0
[junit4:junit4]   2> 13942 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[30 (-1430145701620219904)]} 0 1
[junit4:junit4]   2> 13943 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[10030 (-1430145701619171328)]} 0 1
[junit4:junit4]   2> 13944 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[10030 (-1430145701619171328)]} 0 5
[junit4:junit4]   2> 13948 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {delete=[30 (-1430145701624414208)]} 0 0
[junit4:junit4]   2> 13948 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10032 (1430145701626511360)]} 0 1
[junit4:junit4]   2> 13949 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[30 (-1430145701624414208)]} 0 4
[junit4:junit4]   2> 13953 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[32 (1430145701631754240)]} 0 1
[junit4:junit4]   2> 13956 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10032 (1430145701631754240)]} 0 0
[junit4:junit4]   2> 13957 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10032 (1430145701631754240)]} 0 6
[junit4:junit4]   2> 13961 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[32 (1430145701636997120)]} 0 1
[junit4:junit4]   2> 13962 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[32 (1430145701636997120)]} 0 5
[junit4:junit4]   2> 14008 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 14009 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:36542__ydt%2Fxa_collection1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"down",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36542__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36542/_ydt/xa"}
[junit4:junit4]   2> 14015 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14015 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14015 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14015 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14023 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10033 (1430145701705154560)]} 0 1
[junit4:junit4]   2> 14028 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10033 (1430145701710397440)]} 0 1
[junit4:junit4]   2> 14028 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[33 (1430145701710397440)]} 0 1
[junit4:junit4]   2> 14032 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[33 (1430145701714591744)]} 0 1
[junit4:junit4]   2> 14067 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10033 (-1430145701752340480)]} 0 0
[junit4:junit4]   2> 14071 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10033 (-1430145701756534784)]} 0 0
[junit4:junit4]   2> 14071 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[33 (-1430145701756534784)]} 0 0
[junit4:junit4]   2> 14075 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[33 (-1430145701760729088)]} 0 0
[junit4:junit4]   2> 14075 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10034 (1430145701760729088)]} 0 0
[junit4:junit4]   2> 14079 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[34 (1430145701763874816)]} 0 1
[junit4:junit4]   2> 14079 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10034 (1430145701764923392)]} 0 0
[junit4:junit4]   2> 14083 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[34 (1430145701768069120)]} 0 1
[junit4:junit4]   2> 14174 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[10034 (-1430145701864538112)]} 0 0
[junit4:junit4]   2> 14178 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[34 (-1430145701868732416)]} 0 0
[junit4:junit4]   2> 14178 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10034 (-1430145701868732416)]} 0 0
[junit4:junit4]   2> 14182 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[34 (-1430145701872926720)]} 0 1
[junit4:junit4]   2> 14183 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10035 (1430145701872926720)]} 0 1
[junit4:junit4]   2> 14186 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[35 (1430145701876072448)]} 0 1
[junit4:junit4]   2> 14187 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10035 (1430145701877121024)]} 0 1
[junit4:junit4]   2> 14190 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[35 (1430145701880266752)]} 0 1
[junit4:junit4]   2> 14249 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10036 (1430145701943181312)]} 0 1
[junit4:junit4]   2> 14252 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[36 (1430145701946327040)]} 0 1
[junit4:junit4]   2> 14254 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10036 (1430145701947375616)]} 0 1
[junit4:junit4]   2> 14256 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[36 (1430145701950521344)]} 0 1
[junit4:junit4]   2> 14313 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10037 (1430145702009241600)]} 0 1
[junit4:junit4]   2> 14315 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[37 (1430145702011338752)]} 0 1
[junit4:junit4]   2> 14316 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10037 (1430145702012387328)]} 0 1
[junit4:junit4]   2> 14319 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[37 (1430145702015533056)]} 0 1
[junit4:junit4]   2> 14370 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10038 (1430145702069010432)]} 0 1
[junit4:junit4]   2> 14373 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[38 (1430145702072156160)]} 0 1
[junit4:junit4]   2> 14374 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10038 (1430145702074253312)]} 0 1
[junit4:junit4]   2> 14377 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[38 (1430145702076350464)]} 0 1
[junit4:junit4]   2> 14420 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10039 (1430145702121439232)]} 0 1
[junit4:junit4]   2> 14422 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[39 (1430145702123536384)]} 0 1
[junit4:junit4]   2> 14424 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[10039 (1430145702125633536)]} 0 1
[junit4:junit4]   2> 14426 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[39 (1430145702127730688)]} 0 1
[junit4:junit4]   2> 14519 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10040 (1430145702226296832)]} 0 1
[junit4:junit4]   2> 14521 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[40 (1430145702228393984)]} 0 1
[junit4:junit4]   2> 14523 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10040 (1430145702230491136)]} 0 1
[junit4:junit4]   2> 14525 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[40 (1430145702232588288)]} 0 0
[junit4:junit4]   2> 14618 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10037 (-1430145702330105856)]} 0 0
[junit4:junit4]   2> 14620 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[37 (-1430145702332203008)]} 0 0
[junit4:junit4]   2> 14622 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[10037 (-1430145702334300160)]} 0 0
[junit4:junit4]   2> 14624 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[37 (-1430145702336397312)]} 0 1
[junit4:junit4]   2> 14626 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10041 (1430145702338494464)]} 0 1
[junit4:junit4]   2> 14628 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[41 (1430145702339543040)]} 0 1
[junit4:junit4]   2> 14631 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10041 (1430145702342688768)]} 0 1
[junit4:junit4]   2> 14632 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[41 (1430145702343737344)]} 0 1
[junit4:junit4]   2>  C55_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=down, core=collection1, collection=collection1, node_name=127.0.0.1:36542__ydt%2Fxa, base_url=http://127.0.0.1:36542/_ydt/xa}
[junit4:junit4]   2> 14650 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:48809/_ydt/xa/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 14650 T1616 C55 P36542 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:36542/_ydt/xa START replicas=[http://127.0.0.1:48809/_ydt/xa/collection1/] nUpdates=100
[junit4:junit4]   2> 14653 T1616 C55 P36542 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 14653 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 14653 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 14653 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 14653 T1593 C57 P48809 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0 
[junit4:junit4]   2> 14653 T1616 C55 P36542 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/tlog/tlog.0000000000000000000 refcount=1}}
[junit4:junit4]   2> 14654 T1616 C55 P36542 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:48809/_ydt/xa/collection1/. core=collection1
[junit4:junit4]   2> 14654 T1616 C55 P36542 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 14657 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10042 (1430145702368903168)]} 0 2
[junit4:junit4]   2> 14657 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[42 (1430145702371000320)]} 0 0
[junit4:junit4]   2> 14661 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10042 (1430145702375194624)]} 0 1
[junit4:junit4]   2> 14662 T1593 C57 P48809 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 14662 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[42 (1430145702375194625)]} 0 1
[junit4:junit4]   2> 14668 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10043 (1430145702381486080)]} 0 1
[junit4:junit4]   2> 14669 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[43 (1430145702382534656)]} 0 1
[junit4:junit4]   2> 14676 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[43 (1430145702388826112)]} 0 3
[junit4:junit4]   2> 14686 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10043 (1430145702385680384)]} 0 15
[junit4:junit4]   2> 14687 T1593 C57 P48809 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cc538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cece7d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3cc538 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1cece7d),segFN=segments_2,generation=2,filenames=[_1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _0_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 14688 T1593 C57 P48809 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _0_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 14698 T1593 C57 P48809 oass.SolrIndexSearcher.<init> Opening Searcher@cace76 realtime
[junit4:junit4]   2> 14698 T1593 C57 P48809 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 14699 T1593 C57 P48809 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 37
[junit4:junit4]   2> 14700 T1616 C55 P36542 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 14700 T1616 C55 P36542 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 14704 T1590 C57 P48809 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 14704 T1590 C57 P48809 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2 
[junit4:junit4]   2> 14705 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 14706 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 14706 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 14709 T1591 C57 P48809 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0 
[junit4:junit4]   2> 14712 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 51
[junit4:junit4]   2> 14713 T1616 C55 P36542 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index.20130321211345125 forceNew: false
[junit4:junit4]   2> 14713 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)) fullCopy=true
[junit4:junit4]   2> 14716 T1593 C57 P48809 REQ /replication {file=_1_Lucene42_0.dvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14719 T1592 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14720 T1616 C55 P36542 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_1_MockFixedIntBlock_0.pyl, size=0}
[junit4:junit4]   2> 14723 T1590 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14725 T1591 C57 P48809 REQ /replication {file=_1_Lucene42_0.dvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14728 T1593 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14731 T1592 C57 P48809 REQ /replication {file=_1.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14734 T1590 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14737 T1591 C57 P48809 REQ /replication {file=_1.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14740 T1593 C57 P48809 REQ /replication {file=_0.fnm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14745 T1592 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14749 T1590 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14754 T1591 C57 P48809 REQ /replication {file=_0_Memory_0.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14756 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[44 (1430145702473760768)]} 0 1
[junit4:junit4]   2> 14757 T1593 C57 P48809 REQ /replication {file=_1.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14759 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[44 (1430145702476906496)]} 0 1
[junit4:junit4]   2> 14762 T1590 C57 P48809 REQ /replication {file=_0_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14766 T1591 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14766 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10044 (1430145702485295104)]} 0 0
[junit4:junit4]   2> 14770 T1592 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14770 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[10044 (1430145702489489408)]} 0 1
[junit4:junit4]   2> 14774 T1590 C57 P48809 REQ /replication {file=_1.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14779 T1591 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14785 T1593 C57 P48809 REQ /replication {file=_1.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14790 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[41 (-1430145702510460928)]} 0 1
[junit4:junit4]   2> 14791 T1592 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14793 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[41 (-1430145702513606656)]} 0 0
[junit4:junit4]   2> 14796 T1591 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14796 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[45 (1430145702516752384)]} 0 0
[junit4:junit4]   2> 14800 T1592 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14800 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[45 (1430145702519898112)]} 0 1
[junit4:junit4]   2> 14801 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10041 (-1430145702521995264)]} 0 0
[junit4:junit4]   2> 14805 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[10041 (-1430145702526189568)]} 0 0
[junit4:junit4]   2> 14806 T1591 C57 P48809 REQ /replication {file=_0_Lucene42_0.dvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14809 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10045 (1430145702530383872)]} 0 1
[junit4:junit4]   2> 14813 T1593 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14814 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10045 (1430145702534578176)]} 0 1
[junit4:junit4]   2> 14818 T1590 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14823 T1591 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14824 T1616 C55 P36542 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_1_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4]   2> 14828 T1592 C57 P48809 REQ /replication {file=_0_Lucene42_0.dvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14832 T1593 C57 P48809 REQ /replication {file=_0_Memory_1.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14837 T1590 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14838 T1616 C55 P36542 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_0_MockVariableIntBlock_0.pyl, size=0}
[junit4:junit4]   2> 14841 T1591 C57 P48809 REQ /replication {file=_1_Memory_0.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14845 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[43 (-1430145702568132608)]} 0 1
[junit4:junit4]   2> 14846 T1592 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.pyl&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14848 T1616 C55 P36542 oash.SnapPuller$DirectoryFileFetcher.fetchPackets WARNING No content recieved for file: {name=_0_MockFixedIntBlock_0.pyl, size=0}
[junit4:junit4]   2> 14848 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[43 (-1430145702571278336)]} 0 0
[junit4:junit4]   2> 14852 T1590 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.skp&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14853 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[46 (1430145702575472640)]} 0 1
[junit4:junit4]   2> 14858 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[46 (1430145702580715520)]} 0 2
[junit4:junit4]   2> 14857 T1592 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14859 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10043 (-1430145702582812672)]} 0 1
[junit4:junit4]   2> 14862 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[10043 (-1430145702585958400)]} 0 0
[junit4:junit4]   2> 14863 T1590 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14885 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10046 (1430145702610075648)]} 0 1
[junit4:junit4]   2> 14886 T1591 C57 P48809 REQ /replication {file=_0.nvd&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14888 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10046 (1430145702613221376)]} 0 0
[junit4:junit4]   2> 14889 T1593 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14892 T1590 C57 P48809 REQ /replication {file=_0.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14894 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[47 (1430145702618464256)]} 0 1
[junit4:junit4]   2> 14896 T1591 C57 P48809 REQ /replication {file=_0_MockVariableIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14897 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[47 (1430145702621609984)]} 0 1
[junit4:junit4]   2> 14899 T1593 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.tii&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14902 T1590 C57 P48809 REQ /replication {file=_1_Memory_1.ram&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14932 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10047 (1430145702658310144)]} 0 1
[junit4:junit4]   2> 14935 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10047 (1430145702661455872)]} 0 1
[junit4:junit4]   2> 14957 T1592 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.pos&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14961 T1593 C57 P48809 REQ /replication {file=_1_1.del&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14965 T1591 C57 P48809 REQ /replication {file=_0.nvm&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14969 T1590 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14974 T1592 C57 P48809 REQ /replication {file=_1_MockVariableIntBlock_0.frq&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14978 T1593 C57 P48809 REQ /replication {file=_0_MockFixedIntBlock_0.doc&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14983 T1591 C57 P48809 REQ /replication {file=_1.si&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14988 T1590 C57 P48809 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=1 
[junit4:junit4]   2> 14993 T1592 C57 P48809 REQ /replication {file=_1_MockFixedIntBlock_0.tib&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 14996 T1570 C56 P49567 /update {wt=javabin&version=2} {delete=[44 (-1430145702726467584)]} 0 0
[junit4:junit4]   2> 14997 T1593 C57 P48809 REQ /replication {file=_0.fdx&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 15000 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[44 (-1430145702730661888)]} 0 1
[junit4:junit4]   2> 15001 T1590 C57 P48809 REQ /replication {file=_0.fdt&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0 
[junit4:junit4]   2> 15003 T1616 C55 P36542 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15004 T1616 C55 P36542 oash.SnapPuller.modifyIndexProps New index installed. Updating index properties... index=index.20130321211345125
[junit4:junit4]   2> 15005 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[48 (1430145702734856192)]} 0 1
[junit4:junit4]   2> 15007 T1616 C55 P36542 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15008 T1616 C55 P36542 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15008 T1616 C55 P36542 oasu.DefaultSolrCoreState.newIndexWriter Rollback old IndexWriter... core=collection1
[junit4:junit4]   2> 15010 T1616 C55 P36542 oasc.SolrCore.getNewIndexDir New index directory detected: old=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index/ new=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index.20130321211345125
[junit4:junit4]   2> 15011 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[48 (1430145702741147648)]} 0 2
[junit4:junit4]   2> 15014 T1616 C55 P36542 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)),segFN=segments_2,generation=2,filenames=[_1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _0_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 15014 T1616 C55 P36542 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _0_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 15015 T1616 C55 P36542 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15016 T1616 C55 P36542 oass.SolrIndexSearcher.<init> Opening Searcher@1955741 main
[junit4:junit4]   2> 15017 T1615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1955741 main{StandardDirectoryReader(segments_2:7:nrt _0(5.0):C64/22 _1(5.0):C20/4)}
[junit4:junit4]   2> 15018 T1616 C55 P36542 oash.SnapPuller$2.preClose removing old index files MockDirWrapper(org.apache.lucene.store.RAMDirectory@977832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@108dd54)
[junit4:junit4]   2> 15018 T1616 C55 P36542 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/index
[junit4:junit4]   2> 15018 T1616 C55 P36542 oash.SnapPuller$2.postClose removing old index directory MockDirWrapper(org.apache.lucene.store.RAMDirectory@977832 lockFactory=org.apache.lucene.store.NativeFSLockFactory@108dd54)
[junit4:junit4]   2> 15018 T1616 C55 P36542 oasc.RecoveryStrategy.replay Replaying buffered documents. core=collection1
[junit4:junit4]   2> 15019 T1624 C55 P36542 oasu.UpdateLog$LogReplayer.doReplay WARNING Starting log replay tlog{file=./org.apache.solr.cloud.RecoveryZkTest-1363893210417/jetty2/tlog/tlog.0000000000000000000 refcount=2} active=true starting pos=3846
[junit4:junit4]   2> 15019 T1624 C55 P36542 oasu.DirectUpdateHandler2.commit start commit{flags=2,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15020 T1624 C55 P36542 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)),segFN=segments_2,generation=2,filenames=[_1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _0_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.tib, _1.fdx, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockVariableIntBlock_0.tib, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, segments_2, _0.fdx, _0.fdt]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)),segFN=segments_3,generation=3,filenames=[_1_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvm, _0_Lucene42_0.dvd, _1_MockFixedIntBlock_0.doc, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.pyl, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvd, _0_MockFixedIntBlock_0.tib, _1.fnm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.tii, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.tii, _1_Memory_1.ram, _0_Memory_0.ram, _1_MockVariableIntBlock_0.pos, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.frq, _1.fdx, _1_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockFixedIntBlock_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.tib, _0.fdx, _1_MockFixedIntBlock_0.pos, _0.fdt, segments_3]
[junit4:junit4]   2> 15020 T1624 C55 P36542 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_1_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvm, _0_Lucene42_0.dvd, _1_MockFixedIntBlock_0.doc, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.pyl, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvd, _0_MockFixedIntBlock_0.tib, _1.fnm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.tii, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.tii, _1_Memory_1.ram, _0_Memory_0.ram, _1_MockVariableIntBlock_0.pos, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.frq, _1.fdx, _1_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockFixedIntBlock_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.tib, _0.fdx, _1_MockFixedIntBlock_0.pos, _0.fdt, segments_3]
[junit4:junit4]   2> 15021 T1624 C55 P36542 oass.SolrIndexSearcher.<init> Opening Searcher@7a82d7 main
[junit4:junit4]   2> 15021 T1624 C55 P36542 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15021 T1615 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a82d7 main{StandardDirectoryReader(segments_2:7:nrt _0(5.0):C64/22 _1(5.0):C20/4)}
[junit4:junit4]   2> 15022 T1624 C55 P36542 [collection1] {} 0 3
[junit4:junit4]   2> 15022 T1624 C55 P36542 oasu.UpdateLog$LogReplayer.run WARNING Log replay finished. recoveryInfo=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=3846}
[junit4:junit4]   2> 15022 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15022 T1616 C55 P36542 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15023 T1616 C55 P36542 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15024 T1616 C55 P36542 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 15034 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10044 (-1430145702766313472)]} 0 0
[junit4:junit4]   2> 15038 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[10044 (-1430145702770507776)]} 0 0
[junit4:junit4]   2> 15042 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10048 (1430145702773653504)]} 0 1
[junit4:junit4]   2> 15046 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10048 (1430145702778896384)]} 0 1
[junit4:junit4]   2> 15087 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[48 (-1430145702821888000)]} 0 0
[junit4:junit4]   2> 15091 T1590 C57 P48809 /update {wt=javabin&version=2} {delete=[48 (-1430145702826082304)]} 0 0
[junit4:junit4]   2> 15095 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[49 (1430145702829228032)]} 0 1
[junit4:junit4]   2> 15099 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[49 (1430145702833422336)]} 0 1
[junit4:junit4]   2> 15123 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10048 (-1430145702859636736)]} 0 0
[junit4:junit4]   2> 15127 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[10048 (-1430145702862782464)]} 0 1
[junit4:junit4]   2> 15131 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10049 (1430145702866976768)]} 0 1
[junit4:junit4]   2> 15135 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10049 (1430145702871171072)]} 0 1
[junit4:junit4]   2> 15135 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[50 (1430145702872219648)]} 0 0
[junit4:junit4]   2> 15140 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[50 (1430145702876413952)]} 0 1
[junit4:junit4]   2> 15171 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10050 (1430145702909968384)]} 0 1
[junit4:junit4]   2> 15175 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10050 (1430145702914162688)]} 0 0
[junit4:junit4]   2> 15231 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[49 (-1430145702972882944)]} 0 0
[junit4:junit4]   2> 15235 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[49 (-1430145702977077248)]} 0 0
[junit4:junit4]   2> 15239 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[51 (1430145702980222976)]} 0 1
[junit4:junit4]   2> 15243 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[51 (1430145702985465856)]} 0 1
[junit4:junit4]   2> 15247 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[52 (1430145702989660160)]} 0 1
[junit4:junit4]   2> 15251 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[52 (1430145702993854464)]} 0 0
[junit4:junit4]   2> 15267 T1571 C56 P49567 /update {wt=javabin&version=2} {delete=[10049 (-1430145703010631680)]} 0 0
[junit4:junit4]   2> 15271 T1591 C57 P48809 /update {wt=javabin&version=2} {delete=[10049 (-1430145703014825984)]} 0 1
[junit4:junit4]   2> 15272 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[53 (1430145703014825984)]} 0 1
[junit4:junit4]   2> 15274 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10051 (1430145703017971712)]} 0 0
[junit4:junit4]   2> 15276 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[53 (1430145703019020288)]} 0 1
[junit4:junit4]   2> 15279 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10051 (1430145703022166016)]} 0 1
[junit4:junit4]   2> 15283 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[10052 (1430145703026360320)]} 0 1
[junit4:junit4]   2> 15287 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[10052 (1430145703030554624)]} 0 1
[junit4:junit4]   2> 15307 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10053 (1430145703051526144)]} 0 1
[junit4:junit4]   2> 15311 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10053 (1430145703056769024)]} 0 1
[junit4:junit4]   2> 15362 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[54 (1430145703110246400)]} 0 0
[junit4:junit4]   2> 15367 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[54 (1430145703114440704)]} 0 1
[junit4:junit4]   2> 15398 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10054 (1430145703146946560)]} 0 1
[junit4:junit4]   2> 15402 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10054 (1430145703151140864)]} 0 1
[junit4:junit4]   2> 15413 T1572 C56 P49567 /update {wt=javabin&version=2} {delete=[51 (-1430145703163723776)]} 0 0
[junit4:junit4]   2> 15417 T1593 C57 P48809 /update {wt=javabin&version=2} {delete=[51 (-1430145703167918080)]} 0 1
[junit4:junit4]   2> 15421 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[55 (1430145703171063808)]} 0 1
[junit4:junit4]   2> 15425 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[55 (1430145703175258112)]} 0 1
[junit4:junit4]   2> 15448 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[10051 (-1430145703200423936)]} 0 0
[junit4:junit4]   2> 15452 T1592 C57 P48809 /update {wt=javabin&version=2} {delete=[10051 (-1430145703204618240)]} 0 1
[junit4:junit4]   2> 15456 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[10055 (1430145703207763968)]} 0 1
[junit4:junit4]   2> 15460 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[10055 (1430145703211958272)]} 0 1
[junit4:junit4]   2> 15522 T1578 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper... 
[junit4:junit4]   2> 15523 T1578 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2> 	  "operation":"state",
[junit4:junit4]   2> 	  "core_node_name":"127.0.0.1:36542__ydt%2Fxa_collection1",
[junit4:junit4]   2> 	  "numShards":"1",
[junit4:junit4]   2> 	  "shard":"shard1",
[junit4:junit4]   2> 	  "roles":null,
[junit4:junit4]   2> 	  "state":"active",
[junit4:junit4]   2> 	  "core":"collection1",
[junit4:junit4]   2> 	  "collection":"collection1",
[junit4:junit4]   2> 	  "node_name":"127.0.0.1:36542__ydt%2Fxa",
[junit4:junit4]   2> 	  "base_url":"http://127.0.0.1:36542/_ydt/xa"}
[junit4:junit4]   2> 15528 T1572 C56 P49567 /update {wt=javabin&version=2} {add=[56 (1430145703283261440)]} 0 1
[junit4:junit4]   2> 15529 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15529 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15529 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15529 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2>  C55_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36542__ydt%2Fxa, base_url=http://127.0.0.1:36542/_ydt/xa}
[junit4:junit4]   2> 15538 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[56 (1430145703288504320)]} 0 3
[junit4:junit4]   2> 15539 T1593 C57 P48809 /update {wt=javabin&version=2} {add=[56 (1430145703288504320)]} 0 8
[junit4:junit4]   2> 15563 T1573 C56 P49567 /update {wt=javabin&version=2} {add=[10056 (1430145703321010176)]} 0 1
[junit4:junit4]   2> 15571 T1609 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10056 (1430145703325204480)]} 0 1
[junit4:junit4]   2> 15572 T1592 C57 P48809 /update {wt=javabin&version=2} {add=[10056 (1430145703325204480)]} 0 5
[junit4:junit4]   2> 15620 T1570 C56 P49567 /update {wt=javabin&version=2} {add=[57 (1430145703380779008)]} 0 0
[junit4:junit4]   2> 15628 T1604 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[57 (1430145703384973312)]} 0 1
[junit4:junit4]   2> 15629 T1590 C57 P48809 /update {wt=javabin&version=2} {add=[57 (1430145703384973312)]} 0 5
[junit4:junit4]   2> 15653 T1555 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15654 T1571 C56 P49567 /update {wt=javabin&version=2} {add=[10057 (1430145703415382016)]} 0 1
[junit4:junit4]   2> 15654 T1555 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15659 T1555 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4e340
[junit4:junit4]   2> 15660 T736 oasc.OverseerCollectionProcessor.run WARNING Overseer cannot talk to ZK
[junit4:junit4]   2> 15661 T1607 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[10057 (1430145703419576320)]} 0 0
[junit4:junit4]   2> 15662 T1607 C55 P36542 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4e340
[junit4:junit4]   2> 15664 T1607 C55 P36542 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: Failed to unregister info bean: /update
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:178)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.clear(JmxMonitoredMap.java:116)
[junit4:junit4]   2> 		at org.apache.solr.core.SolrCore.close(SolrCore.java:972)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:374)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	Caused by: javax.management.InstanceNotFoundException: solr/collection1:type=/update,id=org.apache.solr.handler.UpdateRequestHandler
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1095)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:427)
[junit4:junit4]   2> 		at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
[junit4:junit4]   2> 		at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:537)
[junit4:junit4]   2> 		at org.apache.solr.core.JmxMonitoredMap.unregister(JmxMonitoredMap.java:175)
[junit4:junit4]   2> 		... 29 more
[junit4:junit4]   2> 	
[junit4:junit4]   2> 15665 T1607 C55 P36542 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=4,adds=4,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=39,cumulative_deletesById=10,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15667 T1607 C55 P36542 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 15668 T1607 C55 P36542 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 15668 T1607 C55 P36542 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 15669 T1555 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=4,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=39,cumulative_deletesById=10,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15675 T1607 C55 P36542 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)),segFN=segments_3,generation=3,filenames=[_1_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvm, _0_Lucene42_0.dvd, _1_MockFixedIntBlock_0.doc, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.pyl, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_MockVariableIntBlock_0.skp, _1_Lucene42_0.dvd, _0_MockFixedIntBlock_0.tib, _1.fnm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _0_MockFixedIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.tii, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _1_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.tii, _1_Memory_1.ram, _0_Memory_0.ram, _1_MockVariableIntBlock_0.pos, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _1_1.del, _0.nvm, _1_MockVariableIntBlock_0.tib, _0_MockFixedIntBlock_0.frq, _1.fdx, _1_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.skp, _1.fdt, _0_MockFixedIntBlock_0.doc, _1.si, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.tib, _0.fdx, _1_MockFixedIntBlock_0.pos, _0.fdt, segments_3]
[junit4:junit4]   2> 		commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a)),segFN=segments_4,generation=4,filenames=[_4_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _4.fnm, _0_MockVariableIntBlock_0.skp, _4_MockVariableIntBlock_0.tii, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _4.fdx, _4_Lucene42_0.dvd, _0_MockFixedIntBlock_0.tii, _4_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _4_MockFixedIntBlock_0.pos, _1_MockFixedIntBlock_0.frq, _4_Lucene42_0.dvm, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _4_MockFixedIntBlock_0.skp, _4_MockVariableIntBlock_0.doc, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _4_Memory_1.ram, _1_MockVariableIntBlock_0.tib, _1.fdx, _4_MockVariableIntBlock_0.pyl, _4_MockVariableIntBlock_0.tib, _1_MockFixedIntBlock_0.skp, _1.fdt, _4_MockFixedIntBlock_0.doc, _4_Memory_0.ram, _4_MockFixedIntBlock_0.frq, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _4_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockVariableIntBlock_0.tib, _4_MockVariableIntBlock_0.pos, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _4_MockFixedIntBlock_0.pyl, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _4_MockVariableIntBlock_0.frq, _4.si, _4.fdt, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _4.nvd, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, _0.fdx, _4.nvm, _0.fdt, segments_4]
[junit4:junit4]   2> 15676 T1607 C55 P36542 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[_4_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvm, _1_MockFixedIntBlock_0.pyl, _4.fnm, _0_MockVariableIntBlock_0.skp, _4_MockVariableIntBlock_0.tii, _0_MockFixedIntBlock_0.tib, _1_Lucene42_0.dvd, _1.fnm, _4.fdx, _4_Lucene42_0.dvd, _0_MockFixedIntBlock_0.tii, _4_MockFixedIntBlock_0.tii, _1.nvm, _0.fnm, _4_MockFixedIntBlock_0.pos, _1_MockFixedIntBlock_0.frq, _4_Lucene42_0.dvm, _0_Memory_0.ram, _1_MockVariableIntBlock_0.tii, _4_MockFixedIntBlock_0.skp, _4_MockVariableIntBlock_0.doc, _1.nvd, _0_1.del, _0_MockFixedIntBlock_0.pos, _4_Memory_1.ram, _1_MockVariableIntBlock_0.tib, _1.fdx, _4_MockVariableIntBlock_0.pyl, _4_MockVariableIntBlock_0.tib, _1_MockFixedIntBlock_0.skp, _1.fdt, _4_MockFixedIntBlock_0.doc, _4_Memory_0.ram, _4_MockFixedIntBlock_0.frq, _0_MockVariableIntBlock_0.frq, _1_MockFixedIntBlock_0.pos, _1_MockVariableIntBlock_0.skp, _0_Lucene42_0.dvd, _0_MockVariableIntBlock_0.doc, _1_MockFixedIntBlock_0.doc, _1_MockVariableIntBlock_0.pyl, _0_Lucene42_0.dvm, _0_Memory_1.ram, _0_MockVariableIntBlock_0.pyl, _1_Memory_0.ram, _4_MockVariableIntBlock_0.skp, _0_MockFixedIntBlock_0.pyl, _0_MockFixedIntBlock_0.skp, _0_MockVariableIntBlock_0.tib, _4_MockVariableIntBlock_0.pos, _1_MockVariableIntBlock_0.doc, _0.nvd, _0_MockVariableIntBlock_0.pos, _4_MockFixedIntBlock_0.pyl, _0.si, _0_MockVariableIntBlock_0.tii, _1_MockFixedIntBlock_0.tii, _1_Memory_1.ram, _1_MockVariableIntBlock_0.pos, _4_MockVariableIntBlock_0.frq, _4.si, _4.fdt, _0.nvm, _1_1.del, _0_MockFixedIntBlock_0.frq, _1_MockVariableIntBlock_0.frq, _4.nvd, _0_MockFixedIntBlock_0.doc, _1.si, _1_MockFixedIntBlock_0.tib, _0.fdx, _4.nvm, _0.fdt, segments_4]
[junit4:junit4]   2> 15680 T1607 C55 P36542 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15680 T1555 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15684 T1573 C56 P49567 /update {wt=javabin&version=2} {delete=[53 (-1430145703447887872)]} 0 0
[junit4:junit4]   2> 15731 T1591 C57 P48809 /update {wt=javabin&version=2} {add=[10057 (1430145703419576320)]} 0 74
[junit4:junit4]   2> 15731 T1560 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d8e5ecd380008
[junit4:junit4]   2> 15732 T1613 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 15732 T1557 oazs.NIOServerCnxn.closeSock Closed socket connection for client /127.0.0.1:45565 which had sessionid 0x13d8e5ecd380008
[junit4:junit4]   2> 15732 T1613 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 15732 T1555 oaz.ZooKeeper.close Session: 0x13d8e5ecd380008 closed
[junit4:junit4]   2> 15733 T1613 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 15733 T1597 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15733 T1577 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15733 T1584 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 15733 T1597 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 15733 T1613 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 15734 T1584 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 15733 T1577 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 15734 T1608 C55 P36542 oass.SolrIndexSearcher.<init> Opening Searcher@25a600 main
[junit4:junit4]   2> 15735 T1608 C55 P36542 /update {distrib.from=http://127.0.0.1:48809/_ydt/xa/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {} 0 44
[junit4:junit4]   2> 15736 T1608 C55 P36542 oasc.SolrException.log SEVERE org.apache.solr.common.SolrException: SolrCoreState already closed
[junit4:junit4]   2> 		at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:85)
[junit4:junit4]   2> 		at org.apache.solr.update.DirectUpdateHandler2.delete(DirectUpdateHandler2.java:283)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:77)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:456)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDelete(DistributedUpdateProcessor.java:1031)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:717)
[junit4:junit4]   2> 		at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
[junit4:junit4]   2> 		at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:346)
[junit4:junit4]   2> 		at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:277)
[junit4:junit4]   2> 		at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:173)
[junit4:junit4]   2> 		at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
[junit4:junit4]   2> 		at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
[junit4:junit4]   2> 		at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4]   2> 		at org.apache.solr.core.SolrCore.execute(SolrCore.java:1813)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4]   2> 		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]   2> 		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]   2> 		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]   2> 		at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4]   2> 		at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	
[junit4:junit4]   2> 15737 T1608 C55 P36542 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4e340
[junit4:junit4]   2> 15737 T1608 C55 P36542 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=4,deletesById=1,deletesByQuery=0,errors=0,cumulative_adds=39,cumulative_deletesById=11,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 15738 T1608 C55 P36542 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 15740 T1592 C57 P48809 oasc.SolrException.log SEVERE shard update error StdNode: http://127.0.0.1:36542/_ydt/xa/collection1/:org.apache.solr.common.SolrException: Server at http://127.0.0.1:36542/_ydt/xa/collection1 returned non ok status:503, message:Service Unavailable
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4]   2> 		at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4]   2> 		at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4]   2> 		at org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4]   2> 		at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 		at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4]   2> 		at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> 		at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> 		at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> 	
[junit4:junit4]   2> 15740 T1592 C57 P48809 oasup.DistributedUpdateProcessor.doFinish try and ask http://127.0.0.1:36542/_ydt/xa to recover
[junit4:junit4]   2> 15740 T1592 C57 P48809 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 15745 T1555 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/_ydt/xa,null}
[junit4:junit4]   2> 15747 T1592 C57 P48809 oasup.DistributedUpdateProcessor.doFinish Could not tell a replica to recover org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:36542/_ydt/xa
[junit4:junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:409)
[junit4:junit4]   2> 	at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish(DistributedUpdateProcessor.java:440)
[junit4:junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.finish(DistributedUpdateProcessor.java:1086)
[junit4:junit4]   2> 	at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:179)
[junit4:junit4]   2> 	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
[junit4:junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
[junit4:junit4]   2> 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1813)
[junit4:junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639)
[junit4:junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
[junit4:junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]   2> 	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]   2> 	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]   2> 	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
[junit4:junit4]   2> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
[junit4:junit4]   2> 	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
[junit4:junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]   2> 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]   2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:36542 refused
[junit4:junit4]   2> 	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClie

[...truncated too long message...]

eIntBlock(baseBlockSize=63), timestamp=MockFixedIntBlock(blockSize=1447), id=PostingsFormat(name=Memory doPackFST= false), a_t=MockFixedIntBlock(blockSize=1447), range_facet_sl=PostingsFormat(name=Memory doPackFST= false), range_facet_si=MockFixedIntBlock(blockSize=1447), other_tl1=MockVariableIntBlock(baseBlockSize=63), multiDefault=MockFixedIntBlock(blockSize=1447), a_si=MockVariableIntBlock(baseBlockSize=63)}, docValues:{timestamp=DocValuesFormat(name=Lucene42)}, sim=DefaultSimilarity, locale=sr_RS_#Latn, timezone=Europe/Uzhgorod
[junit4:junit4]   2> NOTE: Linux 3.2.0-39-generic i386/Oracle Corporation 1.7.0_15 (32-bit)/cpus=8,threads=1,free=340230040,total=427819008
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestCopyFieldCollectionResource, ClusterStateUpdateTest, DirectUpdateHandlerOptimizeTest, TestTrie, SliceStateTest, SoftAutoCommitTest, DistributedTermsComponentTest, UpdateRequestProcessorFactoryTest, PluginInfoTest, TestPseudoReturnFields, OverseerTest, CachingDirectoryFactoryTest, PrimUtilsTest, ChaosMonkeyNothingIsSafeTest, DocumentBuilderTest, LegacyHTMLStripCharFilterTest, ScriptEngineTest, QueryElevationComponentTest, SpellingQueryConverterTest, BadComponentTest, TestJoin, ShardRoutingTest, PrimitiveFieldTypeTest, ReturnFieldsTest, TestSort, TestStressRecovery, TestReversedWildcardFilterFactory, TestLMDirichletSimilarityFactory, ZkControllerTest, QueryEqualityTest, DateFieldTest, TestRandomDVFaceting, BasicFunctionalityTest, TestFunctionQuery, TestFuzzyAnalyzedSuggestions, DocValuesTest, TestExtendedDismaxParser, TestStressVersions, TestFaceting, JsonLoaderTest, TestPerFieldSimilarity, DistributedSpellCheckComponentTest, TestRemoteStreaming, TestCoreContainer, RecoveryZkTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=RecoveryZkTest -Dtests.seed=2C859135E16E8F66 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr_RS_#Latn -Dtests.timezone=Europe/Uzhgorod -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   0.00s J0 | RecoveryZkTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1614, name=coreLoadExecutor-597-thread-1, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:524)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:991)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:883)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:875)
[junit4:junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4]    > 	at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:244)
[junit4:junit4]    > 	at org.apache.solr.core.SolrCore.initIndex(SolrCore.java:458)
[junit4:junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:746)
[junit4:junit4]    > 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4]    > 	at org.apache.solr.core.CoreContainer.createFromZk(CoreContainer.java:872)
[junit4:junit4]    > 	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:955)
[junit4:junit4]    > 	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:579)
[junit4:junit4]    > 	at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:574)
[junit4:junit4]    > 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]    > 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4]    > 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]    > 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1a41104 lockFactory=org.apache.lucene.store.NativeFSLockFactory@b29f12))
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	... 1 more
[junit4:junit4]    > Throwable #2: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=1616, name=RecoveryThread, state=RUNNABLE, group=TGRP-RecoveryZkTest], registration stack trace below.
[junit4:junit4]    > 	at java.lang.Thread.getStackTrace(Thread.java:1567)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:524)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:991)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:883)
[junit4:junit4]    > 	at org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:875)
[junit4:junit4]    > 	at org.apache.solr.core.MockDirectoryFactory.create(MockDirectoryFactory.java:37)
[junit4:junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:362)
[junit4:junit4]    > 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:322)
[junit4:junit4]    > 	at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:383)
[junit4:junit4]    > 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:281)
[junit4:junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:153)
[junit4:junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:409)
[junit4:junit4]    > 	at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:223)
[junit4:junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1e32321 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4f306a))
[junit4:junit4]    > 	at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > 	at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4]    > 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > 	at java.lang.Thread.run(Thread.java:722)
[junit4:junit4] Completed on J0 in 35.67s, 1 test, 2 errors <<< FAILURES!

[...truncated 519 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:355: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:183: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1210: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:874: There were test failures: 272 suites, 1144 tests, 2 suite-level errors, 13 ignored (7 assumptions)

Total time: 39 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Description set: Java: 32bit/jdk1.7.0_15 -server -XX:+UseG1GC
Email was triggered for: Failure
Sending email for trigger: Failure



Mime
View raw message