cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steinmaurer, Thomas" <thomas.steinmau...@dynatrace.com>
Subject RE: Multi-node repair fails after upgrading to 3.0.14
Date Fri, 15 Sep 2017 08:27:31 GMT
Hi Alex,

thanks a lot. Somehow missed that incremental repairs are the default now.

We have been happy with full repair so far, cause data what we currently manually invoke for
being prepared is a small (~1GB or even smaller).

So I guess with full repairs across all nodes, we still can stick with the partition range
(-pr) option, but with 3.0 we additionally have to provide the –full option, right?

Thanks again,
Thomas

From: Alexander Dejanovski [mailto:alex@thelastpickle.com]
Sent: Freitag, 15. September 2017 09:45
To: user@cassandra.apache.org
Subject: Re: Multi-node repair fails after upgrading to 3.0.14

Hi Thomas,

in 2.1.18, the default repair mode was full repair while since 2.2 it is incremental repair.
So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't trigger the same operation.

Incremental repair cannot run on more than one node at a time on a cluster, because you risk
to have conflicts with sessions trying to anticompact and run validation compactions on the
same SSTables (which will make the validation phase fail, like your logs are showing).
Furthermore, you should never use "-pr" with incremental repair because it is useless in that
mode, and won't properly perform anticompaction on all nodes.

If you were happy with full repairs in 2.1.18, I'd suggest to stick with those in 3.0.14 as
well because there are still too many caveats with incremental repairs that should hopefully
be fixed in 4.0+.
Note that full repair will also trigger anticompaction and mark SSTables as repaired in your
release of Cassandra, and only full subrange repairs are the only flavor that will skip anticompaction.

You will need some tooling to help with subrange repairs though, and I'd recommend to use
Reaper which handles automation for you : http://cassandra-reaper.io/

If you decide to stick with incremental repairs, first perform a rolling restart of your cluster
to make sure no repair session still runs, and run "nodetool repair" on a single node at a
time. Move on to the next node only when nodetool or the logs show that repair is over (which
will include the anticompaction phase).

Cheers,



On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas <thomas.steinmaurer@dynatrace.com<mailto:thomas.steinmaurer@dynatrace.com>>
wrote:
Hello,

we are currently in the process of upgrading from 2.1.18 to 3.0.14. After upgrading a few
test environments, we start to see some suspicious log entries regarding repair issues.

We have a cron job on all nodes basically executing the following repair call on a daily basis:

nodetool repair –pr <list of CFs>

This gets started on all nodes at the same time. While this has worked with 2.1.18 (at least
we haven’t seen anything suspicious in Cassandra log), with 3.0.14 we get something similar
like that on all nodes (see below; IP addresses and KS/CF faked).

Any pointers are appreciated. Thanks.
Thomas


INFO  [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.153, /FAKE.34.171 on range [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
INFO  [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Requesting merkle trees for YYY (to [/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
INFO  [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - [repair #075d2720-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.57, /FAKE.34.171 on range [(-5410955131843184047,-5390722609201388849],
(-2429793939970389370,-2402273315769352748], (8085575576842594575,8086965740279021106], (-8802193901675845653,-8790472027607832351],
(-3900412470120874591,-3892641480459306647], (5455804264750818305,5465037357825542970], (4930767198829659527,4939587074207662799],
(8086965740279021106,8087442741329154201], (-8933201045321260661,-8926445549049070674], (-4841328524165418854,-4838895482794593338],
(628107265570603622,682509946926464280], (7043245467621414187,7055126022831789025], (624871765540463735,627374995781897409],
(9219228482330263660,9221294940422311559], (-2335215188301493066,-2315034243278984017], (-6216599212198827632,-6211460136507414133],
(-3276490559558850323,-3273110814046238767], (7204991007334459472,7214826985711309418], (1815809811279373566,1846961604192445001],
(8743912118048160970,8751518028513315549], (-9204701745739426439,-9200185935622985719], (7926527126882050773,7941554683778488797],
(-1307707180308444994,-1274682085495751899], (8354147540115782875,8358523989614737607], (-5418282332713406631,-5415077779309282099],
(2436459402559272117,2441988676982099299], (3718536403335860823,3722612994041485455], (613072756805503750,624871765540463735],
(3348338086790700840,3352211222405547568], (7941554683778488797,7949493423692701952], (-319434281654804937,-312029192727845573],
(6560852500852746428,6577389024948220270], (3468872066705970665,3469611600245714638], (-4691666557374653357,-4691655196509961176],
(1517867403379415135,1524487506534693145], (-7422802504145504981,-7402342710717945820], (1766173970230627147,1798177111825664424],
(-4402244320098942276,-4399211415161403124], (-6024004998426219783,-6013031695922198269],
(-6930256177103656090,-6926607739769548590], (-3578196517591043633,-3561058036097453011],
(-7923695406605667073,-7885733750925254770], (-8063817427378208986,-8061632046535266293],
(-7341733690813664665,-7287847431444491429], (-8327456680139422800,-8322944736494135341],
(885662136143255295,898069608029365525], (-5555349213881020405,-5555145874347997903], (-4895525453201761515,-4892963646364151947],
(4925572085497096406,4926057814710187377], (7785700473508314561,7813617525484553901], (-1598850926292630432,-1593204162764888475],
(-2583030464997858536,-2581717714589673598], (7780476570449984013,7785700473508314561], (-3558045406807923433,-3550343462935144866],
(7571721681971063633,7578715430118769516], (8865905084309486257,8879414598472867800], (-4984170160639031182,-4973560491294201055],
(-5469249030580263062,-5432196900285376733], (-340636581410346966,-324449230085298889], (9120471287581648922,9127588925404102522],
(8366987539912250171,8368772126617619299], (-9125573241533654210,-9123870950381500195], (1193377366975272040,1205407211929215537],
(8879414598472867800,8900714457425949896], (7055126022831789025,7065799572034111661], (-1588707445511226686,-1539910209795101605],
(6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ]
INFO  [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - [repair #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.57, /FAKE.34.90 on range [(-3078252446668567055,-3069777768104388887],
(3648099278098819362,3665843214181043507], (5797678959887947223,5820452778888079981], (-2642545928835445644,-2631494184435884123],
(-1612062874902452641,-1610692114449831135], (-1707971415501754602,-1692411707392102715],
(291977720343160936,302780457185309086], (-8155767539645304082,-8140296327332660530], (5553475010776119070,5565820507856584918],
(3560286576304809625,3566372473455883512], (5226123540640728306,5228400303590993685], (3421729237821036215,3428685863718017921],
(1227377289103207735,1240348980976161342], (3246531324407742399,3247845316596819400], (5304633131413824443,5306083993155890840],
(-8619514398407319973,-8606439320645645548], (5911017071945481872,5912872298340550901], (-5637986723275926091,-5632042588860715397],
(-7132684921710202809,-7132365563419969085], (8246342224027692193,8283424185055442516], (-4044458382788374703,-4036186919446747968],
(-5613445605139257263,-5596969946550242654], (1083135773045625799,1084813234330856986], (-8140296327332660530,-8138240730730282332],
(2996936679493130945,3002752531484603869], (-4490677154018906600,-4487029511866186581], (8456154651893349894,8463606226270547769],
(-2656313914110802764,-2642545928835445644], (7193156006022031982,7197068598281426786], (8318481021259042556,8345872746994439720],
(-1692411707392102715,-1673006435564531865], (1097191643473406518,1104630053604629743], (-3613177001569108681,-3599878627600515791],
(-1662308889719802294,-1656482847672279831], (-6811080679276087557,-6804940650782637386],
(7392275851743093948,7393304112887579707], (4240513765329086484,4257716503015670881], (-5036265422304136579,-5029426061972234042],
(-4230577892781670592,-4228053239714848467], (-8159151100170546488,-8155767539645304082],
(-5492523660879132140,-5478464594772026508], (3421125671176312710,3421729237821036215], (5262611881402902860,5275876653180871852],
(1577722798587790899,1613215617842269344], (6666156845317387896,6676801654642938276], (5155814764198615180,5168412530555049843],
(-6824229014637940118,-6819748162411093540], (-3666684985038092211,-3613177001569108681],
(-8874623295866980549,-8872063750661463000], (-3599878627600515791,-3599733923716673512],
(-5960723066215554398,-5949787783010678970], (258158907638948329,259127284829404565], (6661762711488840391,6666156845317387896],
(-6367100347601878936,-6363911697345955296], (-6250542414915159381,-6229550501272299394],
(-6917261891898915460,-6904807338337882298], (-7449919126757571330,-7438527202807402113],
(-6904807338337882298,-6881332882762615208], (-7942161058117423218,-7938222136384107977],
(781504901963705642,791302021433124535], (6527741860207715339,6537050054260012793], (7186200312153798494,7187207897161667549]]
for XXX.[YYY, ZZZ]
INFO  [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - [repair #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0]
new session: will sync /FAKE.33.64, /FAKE.35.153, /FAKE.34.90 on range [(3075097238958732453,3095612227388458535],
(-3791204388712221856,-3785200705970537637], (8410714452405602873,8417924634749091961], (5114137550939144346,5121098642749542165],
(2740032807677652551,2746972622905345611], (2197312222134575375,2220347677927119914], (4864988928009437565,4869276508135102594],
(-3451050463660365008,-3450091362870187260], (7424782039380072835,7472858232992173041], (-5338516343760774634,-5325637469358550877],
(2731359217194343571,2734544646630861913], (382200642783712616,390184326306122373], (430251086609083397,431821763463501369],
(-5338895526845396808,-5338516343760774634], (-2677272836643942252,-2671813555202977248],
(-2132202487362637132,-2128425620095084071], (2165546227633605752,2174360971273026818], (2763720958753148061,2764848409944217454],
(3538741049060169597,3547917038270802802], (-6741608677619654195,-6731111079555719429], (2764848409944217454,2779298419000989854],
(374883901669627753,382200642783712616], (-2706148319144852162,-2694312255222799319], (7132646356350938192,7136352706932176819],
(194821142331236531,228318513249291325], (-2047674613927860008,-2032759017860426681], (-2032759017860426681,-2011077831196488705],
(7472858232992173041,7477981949405760270], (431821763463501369,449911816694531926], (4347942912351042626,4349624658068191606],
(311672118330363382,320380777495445406], (3099877433733294763,3112419968835610433], (-7182182373680571317,-7177421545891675761],
(-6742074278675667310,-6741608677619654195], (-2770159943814078393,-2757633164368231752],
(3095612227388458535,3099877433733294763], (-3741816028443553809,-3741630281022069399], (-738951315633674297,-727109613092919188],
(4851195187346961355,4864988928009437565], (-769229207399684543,-744901822064683937], (-4143548408671617625,-4141029975015318194],
(-7957562451432593113,-7952882742232241654], (-511680378144663402,-494227476671052554], (-832115128813031544,-819699210830328959],
(6784361874104123161,6792435444931864417], (-8284973647739186524,-8256379601422884439], (-5321596150219749918,-5309942391653520721],
(4723799060589319973,4734570102349898748], (5077208816402461873,5091797538025862046], (7487792874019513006,7489186250794409815],
(-1647123563569159073,-1620470309759457440], (2779298419000989854,2784870878309140225], (320380777495445406,344020839234233467],
(3112419968835610433,3119057615014897057], (-8506504972343997648,-8502433630545238869], (-2810980092222625726,-2810271001960433176],
(344020839234233467,347093216302465566], (-8545779729891984163,-8519832061775919840], (2633165392330429557,2680549740474631393],
(6391794517057168460,6431615673158447973], (6440766036098716417,6458083094779720657], (-8735147357564473077,-8730522574908969951],
(7484355688408489651,7487792874019513006], (4325444310827485194,4347942912351042626], (2220347677927119914,2223568835084606157],
(6802149418704156481,6819713717509862346], (4827439895247379322,4851195187346961355], (-4016529712384875309,-4009393333251988044],
(2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ]
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 ActiveRepairService.java:554 - Cannot
start multiple repair sessions over the same sstables
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 - Failed creating
a merkle tree for [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for details)
INFO  [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Received merkle tree for YYY from /FAKE.33.64
ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 - Exception in thread
Thread[Repair#1:1,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: org.apache.cassandra.exceptions.RepairException:
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
(8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
~[guava-18.0.jar:na]
        at com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
        at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_102]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
on XXX/YYY, [(8195393703879512303,8196334842725538685], (8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
        ... 4 common frames omitted
WARN  [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
YYY sync failed
ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0]
Session completed with the following error
org.apache.cassandra.exceptions.RepairException: [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
on XXX/YYY, [(8195393703879512303,8196334842725538685], (8166975326273137878,8182604850967732931],
(-7246799942440641887,-7227869626613009045], (-8371707510273823988,-8365977215604569699],
(-141862581573028594,-140310864869418908], (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
(-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
(-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
(6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461], (1886545362164970333,1886646959491599822],
(-4511817721998311568,-4507491187192881115], (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
(-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594], (481579232521229473,491242114841289497],
(4052464144722307684,4059745901618136723], (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
(7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333], (8151204058820798561,8154760186218662205],
(-1040398370287131739,-1033770179677543189], (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
(-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832], (8183526518331102304,8186908810225025483],
(-5685737903527826627,-5672136154194382932], (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
(-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890], (-3357667382515377172,-3356304907368646189],
(-5370953856683870319,-5345971445444542485], (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
(3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692], (-82499927325251331,-79866884352549492],
(3952144214544622998,3955602392726495936], (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
(-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
(-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858], (-5380038118840759647,-5370953856683870319],
(-3221630728515706463,-3206856875356976885], (-1193448110686154165,-1161640137086921883],
(-3356304907368646189,-3346460884208327912], (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_102]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freist<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>ädterstra<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>ße
313<https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
--
-----------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com<http://www.thelastpickle.com/>
The contents of this e-mail are intended for the named addressee only. It contains information
that may be confidential. Unless you are the named addressee or an authorized designee, you
may not copy or use it, or disclose it to anyone else. If you received it in error please
notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN
91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria,
Freistädterstraße 313
Mime
View raw message