cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shubham Srivastava <Shubham.Srivast...@makemytrip.com>
Subject RE: All host pools Marked Down
Date Tue, 29 May 2012 13:06:11 GMT
My webapp connects to the LoadBalancer IP which has the actual nodes in its pool.

If there is by any chance a connection break then will hector not retry to re-establish connection
I guess it should retry every XX seconds based on  retryDownedHostsDelayInSeconds .

Regards,
Shubham
________________________________
From: cem [cayiroglu@gmail.com]
Sent: Tuesday, May 29, 2012 6:13 PM
To: user@cassandra.apache.org
Subject: Re: All host pools Marked Down

Since all hosts are seem to be down, Hector will not do retry. There should be at least one
node up in a cluster. Make sure that you have a proper connection from your webapps to your
cluster.

Cem.

On Tue, May 29, 2012 at 1:46 PM, Shubham Srivastava <Shubham.Srivastava@makemytrip.com<mailto:Shubham.Srivastava@makemytrip.com>>
wrote:
Any takers on this. Hitting us badly right now.

Regards,
Shubham
________________________________
From: Shubham Srivastava
Sent: Tuesday, May 29, 2012 12:55 PM
To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Subject: All host pools Marked Down

I am getting this exception lot of times

me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden
pushed out to client.

What this causes is no data read/write from the ring from my WebApp.

I have retries as 3 and can see that max retries 3 getting exhausted with the same error as
above.

Checked cfstats and tpstats nothing seem to be a problem.

However through the logs I see lot of time taken in compactions like the below

INFO [CompactionExecutor:73] 2012-05-29 11:03:01,605 CompactionManager.java (line 608) Compacted
to /opt/cassandra-data/data/LH/UserPrefrences-tmp-g-8906-Data.db.  36,986,932 to 36,961,554
(~99% of original) bytes for 132,743 keys.  Time: 112,910ms.

The time taken here seems pretty high. Will this cause a pause or read timeout etc.

I have the connection from my web app through a hardware loadbalancer . Cassandra version
is 0.8.6 with multi-DC ring on 6 nodes each in one DC.
CL:1 and RF:3.

Memeory:8Gb heap -> 14Gb Server memory with 8Core CPU.

How do I move ahead in this.

Shubham Srivastava | Technical Lead - Technology Development

+91 124 4910 548<tel:%2B91%20124%204910%20548>   |  MakeMyTrip.com, 243 SP Infocity,
Udyog Vihar Phase 1, Gurgaon, Haryana - 122 016, India


[http://www.mailmktg.makemytrip.com/signature/images/bulb.gif]What's new? My Trip Rewards
- An exclusive loyalty program for MakeMyTrip customers.<https://rewards.makemytrip.com/MTR>

[http://www.mailmktg.makemytrip.com/signature/images/MMT-signature-footer-V4.gif]<http://www.makemytrip.com/>

[http://www.mailmktg.makemytrip.com/signature/images/map-icon.gif]<http://www.makemytrip.com/support/gurgaon-travel-agent-office.php>
Office Map

[http://www.mailmktg.makemytrip.com/signature/images/facebook-icon.gif]<http://www.facebook.com/pages/MakeMyTrip-Deals/120740541030?ref=search&sid=100000077980239.1422657277..1>
Facebook

[http://www.mailmktg.makemytrip.com/signature/images/twitter-icon.gif]<http://twitter.com/makemytripdeals>
Twitter





Mime
View raw message