cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jji...@gmail.com>
Subject Re: Multi-node repair fails after upgrading to 3.0.14
Date Wed, 20 Sep 2017 06:16:45 GMT
It certainly violates the principle of least astonishment. 

Generally, people with large clusters do it the same way they did in 2.1 - with ring aware
scheduling (which people running large clusters can probably do because they’re less likely
to be using vnodes)

The conversation beyond this belongs on the ticket - the three people most likely to fix it
are aware (Paulo is on the thread, I CC’d Blake and Marcus on the ticket). Further commentary
likely belongs on the ticket.

-- 
Jeff Jirsa


> On Sep 19, 2017, at 10:51 PM, Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> 
> Hi,
>  
> no offense to anybody, but I would even say repair is broken in C* 3.0 (or beginning
with 2.2?). The need for adding some kind of third-party (reaper, etc.) to the deployment
is a smelly sign, that something does not work out-of-the box. I have no idea how users e.g.
with Clusters > 100 nodes do handle repairs with 3.0 (2.2?), if it is not reliable to kick
off repair on several nodes in parallel.
>  
> In my humble opinion, if 3.0 is being classified as “production-ready”, this needs
immediate attention, even if this would mean some sort of backward compatiblity break in a
bug-fix release.
>  
> Just my 2 cents from someone having > 300 Cassandra 2.1 JVMs out there spread around
the world.
>  
> Thanks,
> Thomas
>  
> From: kurt greaves [mailto:kurt@instaclustr.com] 
> Sent: Dienstag, 19. September 2017 23:54
> To: User <user@cassandra.apache.org>
> Subject: RE: Multi-node repair fails after upgrading to 3.0.14
>  
> You're right of course. Part of the reason it's changing so frequently is to try and
improve repairs so that they at least actually work reliably. C* 3 hasn't been the smoothest
ride for repairs. Incremental repairs wasn't really ready for 3.0 so it was a mistake to make
it a default. 
> Unfortunately it's hard to change that back now as it will just lead to more confusion
and problems for users unaware of the change.
>  
> On 20 Sep. 2017 00:25, "Durity, Sean R" <SEAN_R_DURITY@homedepot.com> wrote:
> Required maintenance for a cluster should not be this complicated and should not be changing
so often. To me, this is a major flaw in Cassandra.
>  
>  
> Sean Durity
>  
> From: Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com] 
> Sent: Tuesday, September 19, 2017 2:33 AM
> To: user@cassandra.apache.org
> Subject: RE: Multi-node repair fails after upgrading to 3.0.14
>  
> Hi Kurt,
>  
> thanks for the link!
>  
> Honestly, a pity, that in 3.0, we can’t get the simple, reliable and predictable way
back to run a full repair for very low data volume CFs being kicked off on all nodes in parallel,
without all the magic behind the scene introduced by incremental repairs, even if not used,
as anticompaction even with –full has been introduced with 2.2+ J
>  
>  
> Regards,
> Thomas
>  
> From: kurt greaves [mailto:kurt@instaclustr.com] 
> Sent: Dienstag, 19. September 2017 06:24
> To: User <user@cassandra.apache.org>
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>  
> https://issues.apache.org/jira/browse/CASSANDRA-13153 implies full repairs still triggers
anti-compaction on non-repaired SSTables (if I'm reading that right), so might need to make
sure you don't run multiple repairs at the same time across your nodes (if your using vnodes),
otherwise could still end up trying to run anti-compaction on the same SSTable from 2 repairs.
>  
> Anyone else feel free to jump in and correct me if my interpretation is wrong.
>  
> On 18 September 2017 at 17:11, Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> Jeff,
>  
> what should be the expected outcome when running with 3.0.14:
>  
> nodetool repair –full –pr keyspace cfs
>  
> ·         Should –full trigger anti-compaction?
> 
> ·         Should this be the same operation as nodetool repair –pr keyspace cfs in
2.1?
> 
> ·         Should I be able to  run this on several nodes in parallel as in 2.1 without
troubles, where incremental repair was not the default?
> 
>  
> Still confused if I’m missing something obvious. Sorry about that. J
>  
> Thanks,
> Thomas
>  
> From: Jeff Jirsa [mailto:jjirsa@gmail.com] 
> Sent: Montag, 18. September 2017 16:10
> 
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>  
> Sorry I may be wrong about the cause - didn't see -full
>  
> Mea culpa, its early here and I'm not awake
> 
> 
> -- 
> Jeff Jirsa
>  
> 
> On Sep 18, 2017, at 7:01 AM, Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> 
> Hi Jeff,
>  
> understood. That’s quite a change then coming from 2.1 from an operational POV.
>  
> Thanks again.
>  
> Thomas
>  
> From: Jeff Jirsa [mailto:jjirsa@gmail.com] 
> Sent: Montag, 18. September 2017 15:56
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>  
> The command you're running will cause anticompaction and the range borders for all instances
at the same time
>  
> Since only one repair session can anticompact any given sstable, it's almost guaranteed
to fail
>  
> Run it on one instance at a time
> 
> 
> -- 
> Jeff Jirsa
>  
> 
> On Sep 18, 2017, at 1:11 AM, Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> 
> Hi Alex,
>  
> I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel and this
may pop up now:
>  
> 0.176.38.128 (progress: 1%)
> [2017-09-18 07:59:17,145] Some repair failed
> [2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds
> error: Repair job has failed with the error message: [2017-09-18 07:59:17,145] Some repair
failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: [2017-09-18
07:59:17,145] Some repair failed
>         at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
>         at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
>  
> 2017-09-18 07:59:17 repair finished
>  
>  
> If running the above nodetool call sequentially on all nodes, repair finishes without
printing a stack trace.
>  
> The error message and stack trace isn’t really useful here. Any further ideas/experiences?
>  
> Thanks,
> Thomas
>  
> From: Alexander Dejanovski [mailto:alex@thelastpickle.com] 
> Sent: Freitag, 15. September 2017 11:30
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>  
> Right, you should indeed add the "--full" flag to perform full repairs, and you can then
keep the "-pr" flag.
>  
> I'd advise to monitor the status of your SSTables as you'll probably end up with a pool
of SSTables marked as repaired, and another pool marked as unrepaired which won't be compacted
 together (hence the suggestion of running subrange repairs).
> Use sstablemetadata to check on the "Repaired at" value for each. 0 means unrepaired
and any other value (a timestamp) means the SSTable has been repaired.
> I've had behaviors in the past where running "-pr" on the whole cluster would still not
mark all SSTables as repaired, but I can't say if that behavior has changed in latest versions.
>  
> Having separate pools of SStables that cannot be compacted means that you might have
tombstones that don't get evicted due to partitions living in both states (repaired/unrepaired).
>  
> To sum up the recommendations : 
> - Run a full repair with both "--full" and "-pr" and check that SSTables are properly
marked as repaired
> - Use a tight repair schedule to avoid keeping partitions for too long in both repaired
and unrepaired state
> - Switch to subrange repair if you want to fully avoid marking SSTables as repaired (which
you don't need anyway since you're not using incremental repairs). If you wish to do this,
you'll have to mark back all your sstables to unrepaired, using nodetool sstablerepairedset.
>  
> Cheers,
>  
> On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> Hi Alex,
>  
> thanks a lot. Somehow missed that incremental repairs are the default now.
>  
> We have been happy with full repair so far, cause data what we currently manually invoke
 for being prepared is a small (~1GB or even smaller).
>  
> So I guess with full repairs across all nodes, we still can stick with the partition
range (-pr) option, but with 3.0 we additionally have to provide the –full option, right?
>  
> Thanks again,
> Thomas
>  
> From: Alexander Dejanovski [mailto:alex@thelastpickle.com] 
> Sent: Freitag, 15. September 2017 09:45
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>  
> Hi Thomas,
>  
> in 2.1.18, the default repair mode was full repair while since 2.2 it is incremental
repair.
> So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't trigger the same
operation.
>  
> Incremental repair cannot run on more than one node at a time on a cluster, because you
risk to have conflicts with sessions trying to anticompact and run validation compactions
on the same SSTables (which will make the validation phase fail, like your logs are showing).
> Furthermore, you should never use "-pr" with incremental repair because it is useless
in that mode, and won't properly perform anticompaction on all nodes.
>  
> If you were happy with full repairs in 2.1.18, I'd suggest to stick with those in 3.0.14
as well because there are still too many caveats with incremental repairs that should hopefully
be fixed in 4.0+.
> Note that full repair will also trigger anticompaction and mark SSTables as repaired
in your release of Cassandra, and only full subrange repairs are the only flavor that will
skip anticompaction. 
>  
> You will need some tooling to help with subrange repairs though, and I'd recommend to
use Reaper which handles automation for you : http://cassandra-reaper.io/
>  
> If you decide to stick with incremental repairs, first perform a rolling restart of your
cluster to make sure no repair session still runs, and run "nodetool repair" on a single node
at a time. Move on to the next node only when nodetool or the logs show that repair is over
(which will include the anticompaction phase).
>  
> Cheers,  
>  
>  
>  
> On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com>
wrote:
> Hello,
>  
> we are currently in the process of upgrading from 2.1.18 to 3.0.14. After upgrading a
few test environments, we start to see some suspicious log entries regarding repair issues.
>  
> We have a cron job on all nodes basically executing the following repair call on a daily
basis:
>  
> nodetool repair –pr <list of CFs>
>  
> This gets started on all nodes at the same time. While this has worked with 2.1.18 (at
least we haven’t seen anything suspicious in Cassandra log), with 3.0.14 we get something
similar like that on all nodes (see below; IP addresses and KS/CF faked).
>  
> Any pointers are appreciated. Thanks.
> Thomas
>  
>  
> INFO  [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.153, /FAKE.34.171 on range [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
> INFO  [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Requesting merkle trees for YYY (to [/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
> INFO  [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - [repair #075d2720-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.57, /FAKE.34.171 on range [(-5410955131843184047,-5390722609201388849],
(-2429793939970389370,-2402273315769352748], (8085575576842594575,8086965740279021106], (-8802193901675845653,-8790472027607832351],
(-3900412470120874591,-3892641480459306647], (5455804264750818305,5465037357825542970], (4930767198829659527,4939587074207662799],
(8086965740279021106,8087442741329154201], (-8933201045321260661,-8926445549049070674], (-4841328524165418854,-4838895482794593338],
(628107265570603622,682509946926464280], (7043245467621414187,7055126022831789025], (624871765540463735,627374995781897409],
(9219228482330263660,9221294940422311559], (-2335215188301493066,-2315034243278984017], (-6216599212198827632,-6211460136507414133],
(-3276490559558850323,-3273110814046238767], (7204991007334459472,7214826985711309418], (1815809811279373566,1846961604192445001],
(8743912118048160970,8751518028513315549], (-9204701745739426439,-9200185935622985719], (7926527126882050773,7941554683778488797],
(-1307707180308444994,-1274682085495751899], (8354147540115782875,8358523989614737607], (-5418282332713406631,-5415077779309282099],
(2436459402559272117,2441988676982099299], (3718536403335860823,3722612994041485455], (613072756805503750,624871765540463735],
(3348338086790700840,3352211222405547568], (7941554683778488797,7949493423692701952], (-319434281654804937,-312029192727845573],
(6560852500852746428,6577389024948220270], (3468872066705970665,3469611600245714638], (-4691666557374653357,-4691655196509961176],
(1517867403379415135,1524487506534693145], (-7422802504145504981,-7402342710717945820], (1766173970230627147,1798177111825664424],
(-4402244320098942276,-4399211415161403124], (-6024004998426219783,-6013031695922198269],
(-6930256177103656090,-6926607739769548590], (-3578196517591043633,-3561058036097453011],
(-7923695406605667073,-7885733750925254770], (-8063817427378208986,-8061632046535266293],
(-7341733690813664665,-7287847431444491429], (-8327456680139422800,-8322944736494135341],
(885662136143255295,898069608029365525], (-5555349213881020405,-5555145874347997903], (-4895525453201761515,-4892963646364151947],
(4925572085497096406,4926057814710187377], (7785700473508314561,7813617525484553901], (-1598850926292630432,-1593204162764888475],
(-2583030464997858536,-2581717714589673598], (7780476570449984013,7785700473508314561], (-3558045406807923433,-3550343462935144866],
(7571721681971063633,7578715430118769516], (8865905084309486257,8879414598472867800], (-4984170160639031182,-4973560491294201055],
(-5469249030580263062,-5432196900285376733], (-340636581410346966,-324449230085298889], (9120471287581648922,9127588925404102522],
(8366987539912250171,8368772126617619299], (-9125573241533654210,-9123870950381500195], (1193377366975272040,1205407211929215537],
(8879414598472867800,8900714457425949896], (7055126022831789025,7065799572034111661], (-1588707445511226686,-1539910209795101605],
(6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ]
> INFO  [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - [repair #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.57, /FAKE.34.90 on range [(-3078252446668567055,-3069777768104388887],
(3648099278098819362,3665843214181043507], (5797678959887947223,5820452778888079981], (-2642545928835445644,-2631494184435884123],
(-1612062874902452641,-1610692114449831135], (-1707971415501754602,-1692411707392102715],
(291977720343160936,302780457185309086], (-8155767539645304082,-8140296327332660530], (5553475010776119070,5565820507856584918],
(3560286576304809625,3566372473455883512], (5226123540640728306,5228400303590993685], (3421729237821036215,3428685863718017921],
(1227377289103207735,1240348980976161342], (3246531324407742399,3247845316596819400], (5304633131413824443,5306083993155890840],
(-8619514398407319973,-8606439320645645548], (5911017071945481872,5912872298340550901], (-5637986723275926091,-5632042588860715397],
(-7132684921710202809,-7132365563419969085], (8246342224027692193,8283424185055442516], (-4044458382788374703,-4036186919446747968],
(-5613445605139257263,-5596969946550242654], (1083135773045625799,1084813234330856986], (-8140296327332660530,-8138240730730282332],
(2996936679493130945,3002752531484603869], (-4490677154018906600,-4487029511866186581], (8456154651893349894,8463606226270547769],
(-2656313914110802764,-2642545928835445644], (7193156006022031982,7197068598281426786], (8318481021259042556,8345872746994439720],
(-1692411707392102715,-1673006435564531865], (1097191643473406518,1104630053604629743], (-3613177001569108681,-3599878627600515791],
(-1662308889719802294,-1656482847672279831], (-6811080679276087557,-6804940650782637386],
(7392275851743093948,7393304112887579707], (4240513765329086484,4257716503015670881], (-5036265422304136579,-5029426061972234042],
(-4230577892781670592,-4228053239714848467], (-8159151100170546488,-8155767539645304082],
(-5492523660879132140,-5478464594772026508], (3421125671176312710,3421729237821036215], (5262611881402902860,5275876653180871852],
(1577722798587790899,1613215617842269344], (6666156845317387896,6676801654642938276], (5155814764198615180,5168412530555049843],
(-6824229014637940118,-6819748162411093540], (-3666684985038092211,-3613177001569108681],
(-8874623295866980549,-8872063750661463000], (-3599878627600515791,-3599733923716673512],
(-5960723066215554398,-5949787783010678970], (258158907638948329,259127284829404565], (6661762711488840391,6666156845317387896],
(-6367100347601878936,-6363911697345955296], (-6250542414915159381,-6229550501272299394],
(-6917261891898915460,-6904807338337882298], (-7449919126757571330,-7438527202807402113],
(-6904807338337882298,-6881332882762615208], (-7942161058117423218,-7938222136384107977],
(781504901963705642,791302021433124535], (6527741860207715339,6537050054260012793], (7186200312153798494,7187207897161667549]]
for XXX.[YYY, ZZZ]
> INFO  [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - [repair #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.153, /FAKE.34.90 on range [(3075097238958732453,3095612227388458535],
(-3791204388712221856,-3785200705970537637], (8410714452405602873,8417924634749091961], (5114137550939144346,5121098642749542165],
(2740032807677652551,2746972622905345611], (2197312222134575375,2220347677927119914], (4864988928009437565,4869276508135102594],
(-3451050463660365008,-3450091362870187260], (7424782039380072835,7472858232992173041], (-5338516343760774634,-5325637469358550877],
(2731359217194343571,2734544646630861913], (382200642783712616,390184326306122373], (430251086609083397,431821763463501369],
(-5338895526845396808,-5338516343760774634], (-2677272836643942252,-2671813555202977248],
(-2132202487362637132,-2128425620095084071], (2165546227633605752,2174360971273026818], (2763720958753148061,2764848409944217454],
(3538741049060169597,3547917038270802802], (-6741608677619654195,-6731111079555719429], (2764848409944217454,2779298419000989854],
(374883901669627753,382200642783712616], (-2706148319144852162,-2694312255222799319], (7132646356350938192,7136352706932176819],
(194821142331236531,228318513249291325], (-2047674613927860008,-2032759017860426681], (-2032759017860426681,-2011077831196488705],
(7472858232992173041,7477981949405760270], (431821763463501369,449911816694531926], (4347942912351042626,4349624658068191606],
(311672118330363382,320380777495445406], (3099877433733294763,3112419968835610433], (-7182182373680571317,-7177421545891675761],
(-6742074278675667310,-6741608677619654195], (-2770159943814078393,-2757633164368231752],
(3095612227388458535,3099877433733294763], (-3741816028443553809,-3741630281022069399], (-738951315633674297,-727109613092919188],
(4851195187346961355,4864988928009437565], (-769229207399684543,-744901822064683937], (-4143548408671617625,-4141029975015318194],
(-7957562451432593113,-7952882742232241654], (-511680378144663402,-494227476671052554], (-832115128813031544,-819699210830328959],
(6784361874104123161,6792435444931864417], (-8284973647739186524,-8256379601422884439], (-5321596150219749918,-5309942391653520721],
(4723799060589319973,4734570102349898748], (5077208816402461873,5091797538025862046], (7487792874019513006,7489186250794409815],
(-1647123563569159073,-1620470309759457440], (2779298419000989854,2784870878309140225], (320380777495445406,344020839234233467],
(3112419968835610433,3119057615014897057], (-8506504972343997648,-8502433630545238869], (-2810980092222625726,-2810271001960433176],
(344020839234233467,347093216302465566], (-8545779729891984163,-8519832061775919840], (2633165392330429557,2680549740474631393],
(6391794517057168460,6431615673158447973], (6440766036098716417,6458083094779720657], (-8735147357564473077,-8730522574908969951],
(7484355688408489651,7487792874019513006], (4325444310827485194,4347942912351042626], (2220347677927119914,2223568835084606157],
(6802149418704156481,6819713717509862346], (4827439895247379322,4851195187346961355], (-4016529712384875309,-4009393333251988044],
(2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ]
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 ActiveRepairService.java:554 - Cannot
start multiple repair sessions over the same sstables
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 - Failed creating
a merkle tree for [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for details)
> INFO  [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Received merkle tree for YYY from /FAKE.33.64
> ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 - Exception in thread
Thread[Repair#1:1,5,RMI Runtime]
> com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException:
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>         at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
~[guava-18.0.jar:na]
>         at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
~[guava-18.0.jar:na]
>         at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) ~[apache-cassandra-3.0.14.jar:3.0.14]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_102]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_102]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
on XXX/YYY, [(8195393703879512303,8196334842725538685], (8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>         at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
>         ... 4 common frames omitted
> WARN  [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
YYY sync failed
> ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Session completed with the following error
> org.apache.cassandra.exceptions.RepairException: [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
on XXX/YYY, [(8195393703879512303,8196334842725538685], (8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>         at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
~[apache-cassandra-3.0.14.jar:3.0.14]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[apache-cassandra-3.0.14.jar:3.0.14]
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>  
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
> --
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>  
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
>  
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
>  
> 
> The information in this Internet Email is confidential and may be legally privileged.
It is intended solely for the addressee. Access to this Email by anyone else is unauthorized.
If you are not the intended recipient, any disclosure, copying, distribution or any action
taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed
to our clients any opinions or advice contained in this Email are subject to the terms and
conditions expressed in any applicable governing The Home Depot terms of business or client
engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy
and content of this attachment and for any damages or losses arising from any inaccuracies,
errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature,
which may be contained in this attachment and shall not be liable for direct, indirect, consequential
or special damages in connection with this e-mail message or its attachment.
> The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313

Mime
View raw message