lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar <susheel2...@gmail.com>
Subject Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
Date Thu, 06 Jul 2017 03:17:49 GMT
PLease send all your attachments via dropbox etc.  Do not send them via
attachment as they will be deleted automatically..

On Wed, Jul 5, 2017 at 6:23 PM, Bhalla, Rahat <rbhalla@healthplan.com>
wrote:

> Hi Ere/Other Users,
>
>
>
> *@Ere:* Thank you for the response this morning please find more detailed
> logs from this afternoon.
>
>
>
> We are looking into some of the posts that you suggested but they don’t
> seem to be matching completely as per our current analysis. We are still
> working on analyzing the Heap Dumps on how GC is working. We are also
> looking into why packets are being dropped on the zookeeper nodes.
>
>
>
> What we are now seeing are three other things:
>
>
>
> [ERROR] 2017-07-05 01:57:51 (SolrException.java:log:159) - :
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work – *Only
> Once*
>
>
>
> [ERROR] 2017-07-05 01:58:19 (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader – *Multiple
> times for all collections which have shards on that Solr Node.*
>
>
>
> 34811545 [qtp778720569-21] [ERROR] 2017-07-05 02:01:08 (SolrException.java:log:148)
> - java.io.IOException: Error opening /configs/feedsInBoundNonEx/db-
> data-config.xml – *For all collections which have shards on that node.*
>
>
>
> *If any of you have seen this error before and have any suggestions.
> Please let me know.*
>
>
>
> Thanks,
>
> Rahat Bhalla
>
>
> Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> <http://www.mail-archive.com/solr-user@lucene.apache.org/msg131843.html>
>
> 2017-07-05
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=date:20170705&o=newest&f=1>
>  Thread
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=subject:%22Solr+Prod+Issue+%5C%7C+KeeperErrorCode+%3D+ConnectionLoss+for+%5C%2Foverseer_elect%5C%2Fleader%22&o=newest&f=1>
>  *Ere* *Maijala*
> <http://www.mail-archive.com/search?l=solr-user%40lucene.apache.org&q=from:%22Ere+Maijala%22&o=newest&f=1>
>
>
>
> From the fact that someone has tried to access /etc/passwd file via
>
> your Solr (see all those WARN messages), it seems you have it exposed to
>
> the world, unless of course it's a security scanner you use internally.
>
> Internet is a hostile place, and the very first thing I would do is
>
> shield Solr from external traffic. Even if it's your own security
>
> scanning, I wouldn't do it until you have the system stable.
>
>
>
>
>
> Doing the above you'll reduce noise in the logs and might be able to
>
> better identify the issue.
>
>
>
>
>
> Losing the Zookeeper connection is typically a Java garbage collection
>
> issue. If GC causes too long pauses, the connection may time out. So I
>
> would recommend you start by reading
>
> https://wiki.apache.org/solr/SolrPerformanceProblems and
>
> https://wiki.apache.org/solr/ShawnHeisey. Also make sure that
>
> Zookeeper's Java settings are good.
>
>
>
>
>
> --Ere
>
>
>
>
>
> *From:* Bhalla, Rahat
> *Sent:* Wednesday, July 05, 2017 4:05 AM
> *To:* 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org>
> *Subject:* RE: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> *Importance:* High
>
>
>
> Hi
>
>
>
> I’m not sure if any of you have had a chance to see this email yet.
>
>
>
> We had a reoccurrence of the Issue Today, and I’m attaching the Logs from
> today as well inline below.
>
>
>
> Please let me know if any of you have seen this issue before as this would
> really help me to get to the root of the problem to fix it. I’m a little
> lost here and not entirely sure what to do.
>
>
>
> Thanks,
>
> Rahat Bhalla
>
>
>
> 8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20
> (HttpParser.java:parseNext:1391) - parse exception: java.lang.IllegalArgumentException:
> No Authority for HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}
>
> java.lang.IllegalArgumentException: No Authority
>
>         at org.eclipse.jetty.http.HostPortHttpField.<init>(
> HostPortHttpField.java:43)
>
>         at org.eclipse.jetty.http.HttpParser.parsedHeader(
> HttpParser.java:877)
>
>         at org.eclipse.jetty.http.HttpParser.parseHeaders(
> HttpParser.java:1050)
>
>         at org.eclipse.jetty.http.HttpParser.parseNext(
> HttpParser.java:1266)
>
>         at org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(
> HttpConnection.java:344)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:227)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
> HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../..
> /../../../../../../../../etc/passwd}
>
> 8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for
> HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}
>
> 8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for
> HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-
> solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/
> %2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%
> 2e%2e/%2e%2e/etc/passwd}
> <HttpChannelOverHttp@19a624ec%7br=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd%7d>
>
> 8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%c0%ae%c0%ae/%c0%
> ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%
> ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%
> ae%c0%ae/%c0%ae%c0%ae/etc/passwd org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception:
> Not valid UTF8! byte C0 in state 0
>
> 8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%
> ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte 80 in state 4
>
> 8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%f0%80%80%ae%f0%
> 80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%
> ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%
> 80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%
> ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%
> 80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte 80 in state 6
>
> 8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22
> (URIUtil.java:decodePath:348) - /solr/admin/%f8%80%80%80%ae%
> f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%
> 80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%
> 80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/
> %f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%
> 80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%
> 80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd
> org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8!
> byte F8 in state 0
>
> 8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version
> for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}
>
> 8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version
> for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}
>
> 8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44
> (HttpParser.java:<init>:1740) - Illegal character 0xA in
> state=HEADER_IN_NAME for buffer HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST
> /wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan:
> VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\
> x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}
>
> 8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal character
> 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}
>
> 8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}
>
> 8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}
>
> 8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33
> (HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not
> supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}
>
> 67338023 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:57:42
> (SolrException.java:log:159) - :org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
>
>         at org.apache.solr.common.cloud.SolrZkClient$6.execute(
> SolrZkClient.java:331)
>
>         at org.apache.solr.common.cloud.SolrZkClient$6.execute(
> SolrZkClient.java:328)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getChildren(
> SolrZkClient.java:328)
>
>         at org.apache.solr.cloud.DistributedQueue.fetchZkChildren(
> DistributedQueue.java:299)
>
>         at org.apache.solr.cloud.DistributedQueue.firstChild(
> DistributedQueue.java:279)
>
>         at org.apache.solr.cloud.DistributedQueue.peekElements(
> DistributedQueue.java:328)
>
>         at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(
> OverseerTaskQueue.java:236)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:226)
>
>         at java.lang.Thread.run(Unknown Source)
>
>
>
> 67365997 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:10
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26
> (SolrException.java:log:148) - java.io.IOException: Error opening
> /configs/feedsInBoundCarrier/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/
> db-data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26
> (SolrException.java:log:159) - null:java.io.IOException: Error opening
> /configs/feedsInBoundCarrier/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/feedsInBoundCarrier/
> db-data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67393217 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:58:37
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67420293 [OverseerCollectionConfigSetProcessor-98247501534199809-prod-
> solr-node01:9080_solr-n_0000000158] [ERROR] 2017-07-04 05:59:04
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21
> (SolrException.java:log:148) - java.io.IOException: Error opening
> /configs/txnInBoundNonEx/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-
> data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
> 67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21
> (SolrException.java:log:159) - null:java.io.IOException: Error opening
> /configs/txnInBoundNonEx/db-data-config.xml
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:111)
>
>         at org.apache.solr.handler.dataimport.DataImporter.
> maybeReloadConfiguration(DataImporter.java:125)
>
>         at org.apache.solr.handler.dataimport.DataImportHandler.
> handleRequestBody(DataImportHandler.java:176)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)
>
>         at org.apache.solr.servlet.HttpSolrCall.execute(
> HttpSolrCall.java:657)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:464)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /configs/txnInBoundNonEx/db-
> data-config.xml
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:314)
>
>         at org.apache.solr.common.cloud.SolrZkClient$5.execute(
> SolrZkClient.java:311)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.exists(
> SolrZkClient.java:311)
>
>         at org.apache.solr.cloud.ZkController.pathExists(
> ZkController.java:796)
>
>         at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(
> ZkSolrResourceLoader.java:89)
>
>         ... 32 more
>
>
>
>
>
>
>
> *From:* Bhalla, Rahat
> *Sent:* Monday, July 03, 2017 8:38 PM
> *To:* solr-user@lucene.apache.org
> *Subject:* Solr Prod Issue | KeeperErrorCode = ConnectionLoss for
> /overseer_elect/leader
> *Importance:* High
>
>
>
>
>
> Hi Solr Users,
>
>
>
> I hope this email finds you all in the best of spirits and in a mood where
> you’d be willing to help a young developer (me J ) with issues that I’m
> facing in regards with the Solr Cloud.
>
>
>
> At my organization, we are running a Solr Cloud with 5 Nodes for Solr
> Instances with 13 collections spread across the 5 nodes and an ensemble of
> 3 zookeeper instances spread across three different nodes.
>
>
>
> Over the last one week, our leader node seems to be going down every other
> day and while we restart the solr instances they still go down within the
> next 24 Hours or more.
>
>
>
> We have tried rebooting the nodes that host the solr instances and that
> hasn’t helped. We plan to clear out the zookeeper logs and data folders
> before the restart of the zookeeper instances.
>
>
>
> As of now, I’m the only one supporting Solr in my organization and any
> insight from you could help me a great deal to fix the issue. I’m copying
> the Exception stack trace from this morning. Any recommendations that you
> might have will be great appreciated.
>
>
>
> *Below is a snapshot of one of the zoo nodes:*
>
>
>
>
>
> *Exception Stacktrace*
>
>
>
> 138127149 [OverseerCollectionConfigSetProcessor-98234161688412161-prod-
> solr-node01:9080_solr-n_0000000140] [ERROR] 2017-07-03 05:02:55
> (OverseerTaskProcessor.java:amILeader:392) -
>
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /overseer_elect/leader
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(
> OverseerTaskProcessor.java:384)
>
>         at org.apache.solr.cloud.OverseerTaskProcessor.run(
> OverseerTaskProcessor.java:191)
>
>         at java.lang.Thread.run(Unknown Source)
>
> 138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01
> (SolrException.java:log:148) - org.apache.solr.common.SolrException:
> Could not load collection from ZK: feedsOutBoundToExchange
>
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:1047)
>
>         at org.apache.solr.common.cloud.ZkStateReader$
> LazyCollectionRef.get(ZkStateReader.java:610)
>
>         at org.apache.solr.common.cloud.ClusterState.getCollectionsMap(
> ClusterState.java:248)
>
>         at org.apache.solr.handler.admin.CollectionsHandler$
> CollectionOperation$20.call(CollectionsHandler.java:674)
>
>         at org.apache.solr.handler.admin.CollectionsHandler.
> handleRequestBody(CollectionsHandler.java:195)
>
>         at org.apache.solr.handler.RequestHandlerBase.handleRequest(
> RequestHandlerBase.java:156)
>
>         at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(
> HttpSolrCall.java:663)
>
>         at org.apache.solr.servlet.HttpSolrCall.call(
> HttpSolrCall.java:445)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:257)
>
>         at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
> SolrDispatchFilter.java:208)
>
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
> doFilter(ServletHandler.java:1668)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(
> ServletHandler.java:581)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:143)
>
>         at org.eclipse.jetty.security.SecurityHandler.handle(
> SecurityHandler.java:548)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doHandle(SessionHandler.java:226)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doHandle(ContextHandler.java:1160)
>
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(
> ServletHandler.java:511)
>
>         at org.eclipse.jetty.server.session.SessionHandler.
> doScope(SessionHandler.java:185)
>
>         at org.eclipse.jetty.server.handler.ContextHandler.
> doScope(ContextHandler.java:1092)
>
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(
> ScopedHandler.java:141)
>
>         at org.eclipse.jetty.server.handler.ContextHandlerCollection.
> handle(ContextHandlerCollection.java:213)
>
>         at org.eclipse.jetty.server.handler.HandlerCollection.
> handle(HandlerCollection.java:119)
>
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
> HandlerWrapper.java:134)
>
>         at org.eclipse.jetty.server.Server.handle(Server.java:518)
>
>         at org.eclipse.jetty.server.HttpChannel.handle(
> HttpChannel.java:308)
>
>         at org.eclipse.jetty.server.HttpConnection.onFillable(
> HttpConnection.java:244)
>
>         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(
> AbstractConnection.java:273)
>
>         at org.eclipse.jetty.io.FillInterest.fillable(
> FillInterest.java:95)
>
>         at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
> SelectChannelEndPoint.java:93)
>
>         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.
> produceAndRun(ExecuteProduceConsume.java:246)
>
>         at org.eclipse.jetty.util.thread.strategy.
> ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(
> QueuedThreadPool.java:654)
>
>         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
> QueuedThreadPool.java:572)
>
>         at java.lang.Thread.run(Unknown Source)
>
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /collections/
> feedsOutBoundToExchange/state.json
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:99)
>
>         at org.apache.zookeeper.KeeperException.create(
> KeeperException.java:51)
>
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:348)
>
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(
> ZkCmdExecutor.java:60)
>
>         at org.apache.solr.common.cloud.SolrZkClient.getData(
> SolrZkClient.java:345)
>
>         at org.apache.solr.common.cloud.ZkStateReader.
> fetchCollectionState(ZkStateReader.java:1059)
>
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(
> ZkStateReader.java:1045)
>
>         ... 33 more
>
>
>
> *Thanks,*
>
>
>
> *Rahat Bhalla*
>
> *HealthPlan Services*
> Phone: (813) 289-1000  EXT: 7002249 <(813)%20289-1000>
>
> rbhalla@healthplan.com
>
> *www.healthplan.com <http://www.healthplan.com/> *
>
>
>
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> CONFIDENTIALITY NOTICE: This email message, including any attachments, is
> for the sole use of the intended recipient(s) and may contain confidential
> and privileged information and/or Protected Health Information (PHI)
> subject to protection under the law, including the Health Insurance
> Portability and Accountability Act of 1996, as amended (HIPAA). If you are
> not the intended recipient or the person responsible for delivering the
> email to the intended recipient, be advised that you have received this
> email in error and that any use, disclosure, distribution, forwarding,
> printing, or copying of this email is strictly prohibited. If you have
> received this email in error, please notify the sender immediately and
> destroy all copies of the original message.
>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message