cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anthony P. Scism" <APSC...@UP.COM>
Subject RE: Multi-node repair fails after upgrading to 3.0.14
Date Tue, 19 Sep 2017 15:19:31 GMT
unsubscribe
____________________________________
Anthony P. Scism
Info Tech-Risk Mgmt/Client Sys - Capacity Planning
Work: 402-544-0361 Mobile: 402-707-4446



From:   "Durity, Sean R" <SEAN_R_DURITY@homedepot.com>
To:     "user@cassandra.apache.org" <user@cassandra.apache.org>
Date:   09/19/2017 09:25 AM
Subject:        RE: Multi-node repair fails after upgrading to 3.0.14



This email originated from outside of the company. Please use discretion 
if opening attachments or clicking on links. 
Required maintenance for a cluster should not be this complicated and 
should not be changing so often. To me, this is a major flaw in Cassandra.
 
 
Sean Durity
 
From: Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com] 
Sent: Tuesday, September 19, 2017 2:33 AM
To: user@cassandra.apache.org
Subject: RE: Multi-node repair fails after upgrading to 3.0.14
 
Hi Kurt,
 
thanks for the link!
 
Honestly, a pity, that in 3.0, we can’t get the simple, reliable and 
predictable way back to run a full repair for very low data volume CFs 
being kicked off on all nodes in parallel, without all the magic behind 
the scene introduced by incremental repairs, even if not used, as 
anticompaction even with –full has been introduced with 2.2+ J
 
 
Regards,
Thomas
 
From: kurt greaves [mailto:kurt@instaclustr.com] 
Sent: Dienstag, 19. September 2017 06:24
To: User <user@cassandra.apache.org>
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
 
https://issues.apache.org/jira/browse/CASSANDRA-13153 implies full repairs 
still triggers anti-compaction on non-repaired SSTables (if I'm reading 
that right), so might need to make sure you don't run multiple repairs at 
the same time across your nodes (if your using vnodes), otherwise could 
still end up trying to run anti-compaction on the same SSTable from 2 
repairs.
 
Anyone else feel free to jump in and correct me if my interpretation is 
wrong.
 
On 18 September 2017 at 17:11, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
Jeff,
 
what should be the expected outcome when running with 3.0.14:
 
nodetool repair –full –pr keyspace cfs
 
·         Should –full trigger anti-compaction?
·         Should this be the same operation as nodetool repair –pr 
keyspace cfs in 2.1?
·         Should I be able to  run this on several nodes in parallel as in 
2.1 without troubles, where incremental repair was not the default?
 
Still confused if I’m missing something obvious. Sorry about that. J
 
Thanks,
Thomas
 
From: Jeff Jirsa [mailto:jjirsa@gmail.com] 
Sent: Montag, 18. September 2017 16:10

To: user@cassandra.apache.org
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
 
Sorry I may be wrong about the cause - didn't see -full
 
Mea culpa, its early here and I'm not awake


-- 
Jeff Jirsa
 

On Sep 18, 2017, at 7:01 AM, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
Hi Jeff,
 
understood. That’s quite a change then coming from 2.1 from an operational 
POV.
 
Thanks again.
 
Thomas
 
From: Jeff Jirsa [mailto:jjirsa@gmail.com] 
Sent: Montag, 18. September 2017 15:56
To: user@cassandra.apache.org
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
 
The command you're running will cause anticompaction and the range borders 
for all instances at the same time
 
Since only one repair session can anticompact any given sstable, it's 
almost guaranteed to fail
 
Run it on one instance at a time


-- 
Jeff Jirsa
 

On Sep 18, 2017, at 1:11 AM, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
Hi Alex,
 
I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel 
and this may pop up now:
 
0.176.38.128 (progress: 1%)
[2017-09-18 07:59:17,145] Some repair failed
[2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds
error: Repair job has failed with the error message: [2017-09-18 
07:59:17,145] Some repair failed
-- StackTrace --
java.lang.RuntimeException: Repair job has failed with the error message: 
[2017-09-18 07:59:17,145] Some repair failed
        at 
org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
        at 
org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
        at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
        at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
        at 
com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
        at 
com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
 
2017-09-18 07:59:17 repair finished
 
 
If running the above nodetool call sequentially on all nodes, repair 
finishes without printing a stack trace.
 
The error message and stack trace isn’t really useful here. Any further 
ideas/experiences?
 
Thanks,
Thomas
 
From: Alexander Dejanovski [mailto:alex@thelastpickle.com] 
Sent: Freitag, 15. September 2017 11:30
To: user@cassandra.apache.org
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
 
Right, you should indeed add the "--full" flag to perform full repairs, 
and you can then keep the "-pr" flag.
 
I'd advise to monitor the status of your SSTables as you'll probably end 
up with a pool of SSTables marked as repaired, and another pool marked as 
unrepaired which won't be compacted together (hence the suggestion of 
running subrange repairs).
Use sstablemetadata to check on the "Repaired at" value for each. 0 means 
unrepaired and any other value (a timestamp) means the SSTable has been 
repaired.
I've had behaviors in the past where running "-pr" on the whole cluster 
would still not mark all SSTables as repaired, but I can't say if that 
behavior has changed in latest versions.
 
Having separate pools of SStables that cannot be compacted means that you 
might have tombstones that don't get evicted due to partitions living in 
both states (repaired/unrepaired).
 
To sum up the recommendations : 
- Run a full repair with both "--full" and "-pr" and check that SSTables 
are properly marked as repaired
- Use a tight repair schedule to avoid keeping partitions for too long in 
both repaired and unrepaired state
- Switch to subrange repair if you want to fully avoid marking SSTables as 
repaired (which you don't need anyway since you're not using incremental 
repairs). If you wish to do this, you'll have to mark back all your 
sstables to unrepaired, using nodetool sstablerepairedset.
 
Cheers,
 
On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
Hi Alex,
 
thanks a lot. Somehow missed that incremental repairs are the default now.
 
We have been happy with full repair so far, cause data what we currently 
manually invoke for being prepared is a small (~1GB or even smaller).
 
So I guess with full repairs across all nodes, we still can stick with the 
partition range (-pr) option, but with 3.0 we additionally have to provide 
the –full option, right?
 
Thanks again,
Thomas
 
From: Alexander Dejanovski [mailto:alex@thelastpickle.com] 
Sent: Freitag, 15. September 2017 09:45
To: user@cassandra.apache.org
Subject: Re: Multi-node repair fails after upgrading to 3.0.14
 
Hi Thomas,
 
in 2.1.18, the default repair mode was full repair while since 2.2 it is 
incremental repair.
So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't 
trigger the same operation.
 
Incremental repair cannot run on more than one node at a time on a 
cluster, because you risk to have conflicts with sessions trying to 
anticompact and run validation compactions on the same SSTables (which 
will make the validation phase fail, like your logs are showing).
Furthermore, you should never use "-pr" with incremental repair because it 
is useless in that mode, and won't properly perform anticompaction on all 
nodes.
 
If you were happy with full repairs in 2.1.18, I'd suggest to stick with 
those in 3.0.14 as well because there are still too many caveats with 
incremental repairs that should hopefully be fixed in 4.0+.
Note that full repair will also trigger anticompaction and mark SSTables 
as repaired in your release of Cassandra, and only full subrange repairs 
are the only flavor that will skip anticompaction. 
 
You will need some tooling to help with subrange repairs though, and I'd 
recommend to use Reaper which handles automation for you : 
http://cassandra-reaper.io/
 
If you decide to stick with incremental repairs, first perform a rolling 
restart of your cluster to make sure no repair session still runs, and run 
"nodetool repair" on a single node at a time. Move on to the next node 
only when nodetool or the logs show that repair is over (which will 
include the anticompaction phase).
 
Cheers, 
 
 
 
On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
Hello,
 
we are currently in the process of upgrading from 2.1.18 to 3.0.14. After 
upgrading a few test environments, we start to see some suspicious log 
entries regarding repair issues.
 
We have a cron job on all nodes basically executing the following repair 
call on a daily basis:
 
nodetool repair –pr <list of CFs>
 
This gets started on all nodes at the same time. While this has worked 
with 2.1.18 (at least we haven’t seen anything suspicious in Cassandra 
log), with 3.0.14 we get something similar like that on all nodes (see 
below; IP addresses and KS/CF faked).
 
Any pointers are appreciated. Thanks.
Thomas
 
 
INFO  [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - 
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync 
/FAKE.33.64, /FAKE.35.153, /FAKE.34.171 on range 
[(8195393703879512303,8196334842725538685], 
(8166975326273137878,8182604850967732931], 
(-7246799942440641887,-7227869626613009045], 
(-8371707510273823988,-8365977215604569699], 
(-141862581573028594,-140310864869418908], 
(3732113975108886193,3743105867152786342], 
(4998127507903069087,5008922734235607550], 
(-5115827291264930140,-5111054924035590372], 
(-2475342271852943287,-2447285553369030332], 
(-8318606053827235336,-8308721754886697230], 
(-5208900659917654871,-5202385837264015269], 
(6618737991399272130,6623100721269775102], 
(-4650650128572424858,-4650260492494258461], 
(1886545362164970333,1886646959491599822], 
(-4511817721998311568,-4507491187192881115], 
(8114903118676615937,8132992506844206601], 
(6224957219376301858,6304379125732293904], 
(-3460547504877234383,-3459262416082517136], 
(-167838948111369123,-141862581573028594], 
(481579232521229473,491242114841289497], 
(4052464144722307684,4059745901618136723], 
(1659668187498418295,1679582585970705122], 
(-1118922763210109192,-1093766915505652874], 
(7504365235878319341,7526188885210185292], 
(-79866884352549492,-77667207866300333], 
(8151204058820798561,8154760186218662205], 
(-1040398370287131739,-1033770179677543189], 
(3767057277953758442,3783780844370292025], 
(-6491678058233994892,-6487797181789288329], 
(-916868210769480248,-907141794196269524], 
(-9005441616028750657,-9002220258513351832], 
(8183526518331102304,8186908810225025483], 
(-5685737903527826627,-5672136154194382932], 
(4976122621177738811,4987871287137312689], 
(6051670147160447042,6051686987147911650], 
(-1161640137086921883,-1159172734746043158], 
(6895951547735922309,6899152466544114890], 
(-3357667382515377172,-3356304907368646189], 
(-5370953856683870319,-5345971445444542485], 
(3824272999898372667,3829315045986248983], 
(8132992506844206601,8149858096109302285], 
(3975126143101303723,3980729378827590597], 
(-956691623200349709,-946602525018301692], 
(-82499927325251331,-79866884352549492], 
(3952144214544622998,3955602392726495936], 
(8154760186218662205,8157079055586089583], 
(3840595196718778916,3866458971850198755], 
(-1066905024007783341,-1055954824488508260], 
(-7252356975874511782,-7246799942440641887], 
(-810612946397276081,-792189809286829222], 
(4964519403172053705,4970446606512414858], 
(-5380038118840759647,-5370953856683870319], 
(-3221630728515706463,-3206856875356976885], 
(-1193448110686154165,-1161640137086921883], 
(-3356304907368646189,-3346460884208327912], 
(3466596314109623830,3468144326691723333], 
(-9050241313548454460,-9005441616028750657], 
(402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
INFO  [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair 
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Requesting merkle trees for YYY (to 
[/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
INFO  [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - 
[repair #075d2720-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync 
/FAKE.33.64, /FAKE.35.57, /FAKE.34.171 on range 
[(-5410955131843184047,-5390722609201388849], 
(-2429793939970389370,-2402273315769352748], 
(8085575576842594575,8086965740279021106], 
(-8802193901675845653,-8790472027607832351], 
(-3900412470120874591,-3892641480459306647], 
(5455804264750818305,5465037357825542970], 
(4930767198829659527,4939587074207662799], 
(8086965740279021106,8087442741329154201], 
(-8933201045321260661,-8926445549049070674], 
(-4841328524165418854,-4838895482794593338], 
(628107265570603622,682509946926464280], 
(7043245467621414187,7055126022831789025], 
(624871765540463735,627374995781897409], 
(9219228482330263660,9221294940422311559], 
(-2335215188301493066,-2315034243278984017], 
(-6216599212198827632,-6211460136507414133], 
(-3276490559558850323,-3273110814046238767], 
(7204991007334459472,7214826985711309418], 
(1815809811279373566,1846961604192445001], 
(8743912118048160970,8751518028513315549], 
(-9204701745739426439,-9200185935622985719], 
(7926527126882050773,7941554683778488797], 
(-1307707180308444994,-1274682085495751899], 
(8354147540115782875,8358523989614737607], 
(-5418282332713406631,-5415077779309282099], 
(2436459402559272117,2441988676982099299], 
(3718536403335860823,3722612994041485455], 
(613072756805503750,624871765540463735], 
(3348338086790700840,3352211222405547568], 
(7941554683778488797,7949493423692701952], 
(-319434281654804937,-312029192727845573], 
(6560852500852746428,6577389024948220270], 
(3468872066705970665,3469611600245714638], 
(-4691666557374653357,-4691655196509961176], 
(1517867403379415135,1524487506534693145], 
(-7422802504145504981,-7402342710717945820], 
(1766173970230627147,1798177111825664424], 
(-4402244320098942276,-4399211415161403124], 
(-6024004998426219783,-6013031695922198269], 
(-6930256177103656090,-6926607739769548590], 
(-3578196517591043633,-3561058036097453011], 
(-7923695406605667073,-7885733750925254770], 
(-8063817427378208986,-8061632046535266293], 
(-7341733690813664665,-7287847431444491429], 
(-8327456680139422800,-8322944736494135341], 
(885662136143255295,898069608029365525], 
(-5555349213881020405,-5555145874347997903], 
(-4895525453201761515,-4892963646364151947], 
(4925572085497096406,4926057814710187377], 
(7785700473508314561,7813617525484553901], 
(-1598850926292630432,-1593204162764888475], 
(-2583030464997858536,-2581717714589673598], 
(7780476570449984013,7785700473508314561], 
(-3558045406807923433,-3550343462935144866], 
(7571721681971063633,7578715430118769516], 
(8865905084309486257,8879414598472867800], 
(-4984170160639031182,-4973560491294201055], 
(-5469249030580263062,-5432196900285376733], 
(-340636581410346966,-324449230085298889], 
(9120471287581648922,9127588925404102522], 
(8366987539912250171,8368772126617619299], 
(-9125573241533654210,-9123870950381500195], 
(1193377366975272040,1205407211929215537], 
(8879414598472867800,8900714457425949896], 
(7055126022831789025,7065799572034111661], 
(-1588707445511226686,-1539910209795101605], 
(6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ]
INFO  [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - 
[repair #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync 
/FAKE.33.64, /FAKE.35.57, /FAKE.34.90 on range 
[(-3078252446668567055,-3069777768104388887], 
(3648099278098819362,3665843214181043507], 
(5797678959887947223,5820452778888079981], 
(-2642545928835445644,-2631494184435884123], 
(-1612062874902452641,-1610692114449831135], 
(-1707971415501754602,-1692411707392102715], 
(291977720343160936,302780457185309086], 
(-8155767539645304082,-8140296327332660530], 
(5553475010776119070,5565820507856584918], 
(3560286576304809625,3566372473455883512], 
(5226123540640728306,5228400303590993685], 
(3421729237821036215,3428685863718017921], 
(1227377289103207735,1240348980976161342], 
(3246531324407742399,3247845316596819400], 
(5304633131413824443,5306083993155890840], 
(-8619514398407319973,-8606439320645645548], 
(5911017071945481872,5912872298340550901], 
(-5637986723275926091,-5632042588860715397], 
(-7132684921710202809,-7132365563419969085], 
(8246342224027692193,8283424185055442516], 
(-4044458382788374703,-4036186919446747968], 
(-5613445605139257263,-5596969946550242654], 
(1083135773045625799,1084813234330856986], 
(-8140296327332660530,-8138240730730282332], 
(2996936679493130945,3002752531484603869], 
(-4490677154018906600,-4487029511866186581], 
(8456154651893349894,8463606226270547769], 
(-2656313914110802764,-2642545928835445644], 
(7193156006022031982,7197068598281426786], 
(8318481021259042556,8345872746994439720], 
(-1692411707392102715,-1673006435564531865], 
(1097191643473406518,1104630053604629743], 
(-3613177001569108681,-3599878627600515791], 
(-1662308889719802294,-1656482847672279831], 
(-6811080679276087557,-6804940650782637386], 
(7392275851743093948,7393304112887579707], 
(4240513765329086484,4257716503015670881], 
(-5036265422304136579,-5029426061972234042], 
(-4230577892781670592,-4228053239714848467], 
(-8159151100170546488,-8155767539645304082], 
(-5492523660879132140,-5478464594772026508], 
(3421125671176312710,3421729237821036215], 
(5262611881402902860,5275876653180871852], 
(1577722798587790899,1613215617842269344], 
(6666156845317387896,6676801654642938276], 
(5155814764198615180,5168412530555049843], 
(-6824229014637940118,-6819748162411093540], 
(-3666684985038092211,-3613177001569108681], 
(-8874623295866980549,-8872063750661463000], 
(-3599878627600515791,-3599733923716673512], 
(-5960723066215554398,-5949787783010678970], 
(258158907638948329,259127284829404565], 
(6661762711488840391,6666156845317387896], 
(-6367100347601878936,-6363911697345955296], 
(-6250542414915159381,-6229550501272299394], 
(-6917261891898915460,-6904807338337882298], 
(-7449919126757571330,-7438527202807402113], 
(-6904807338337882298,-6881332882762615208], 
(-7942161058117423218,-7938222136384107977], 
(781504901963705642,791302021433124535], 
(6527741860207715339,6537050054260012793], 
(7186200312153798494,7187207897161667549]] for XXX.[YYY, ZZZ]
INFO  [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - 
[repair #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync 
/FAKE.33.64, /FAKE.35.153, /FAKE.34.90 on range 
[(3075097238958732453,3095612227388458535], 
(-3791204388712221856,-3785200705970537637], 
(8410714452405602873,8417924634749091961], 
(5114137550939144346,5121098642749542165], 
(2740032807677652551,2746972622905345611], 
(2197312222134575375,2220347677927119914], 
(4864988928009437565,4869276508135102594], 
(-3451050463660365008,-3450091362870187260], 
(7424782039380072835,7472858232992173041], 
(-5338516343760774634,-5325637469358550877], 
(2731359217194343571,2734544646630861913], 
(382200642783712616,390184326306122373], 
(430251086609083397,431821763463501369], 
(-5338895526845396808,-5338516343760774634], 
(-2677272836643942252,-2671813555202977248], 
(-2132202487362637132,-2128425620095084071], 
(2165546227633605752,2174360971273026818], 
(2763720958753148061,2764848409944217454], 
(3538741049060169597,3547917038270802802], 
(-6741608677619654195,-6731111079555719429], 
(2764848409944217454,2779298419000989854], 
(374883901669627753,382200642783712616], 
(-2706148319144852162,-2694312255222799319], 
(7132646356350938192,7136352706932176819], 
(194821142331236531,228318513249291325], 
(-2047674613927860008,-2032759017860426681], 
(-2032759017860426681,-2011077831196488705], 
(7472858232992173041,7477981949405760270], 
(431821763463501369,449911816694531926], 
(4347942912351042626,4349624658068191606], 
(311672118330363382,320380777495445406], 
(3099877433733294763,3112419968835610433], 
(-7182182373680571317,-7177421545891675761], 
(-6742074278675667310,-6741608677619654195], 
(-2770159943814078393,-2757633164368231752], 
(3095612227388458535,3099877433733294763], 
(-3741816028443553809,-3741630281022069399], 
(-738951315633674297,-727109613092919188], 
(4851195187346961355,4864988928009437565], 
(-769229207399684543,-744901822064683937], 
(-4143548408671617625,-4141029975015318194], 
(-7957562451432593113,-7952882742232241654], 
(-511680378144663402,-494227476671052554], 
(-832115128813031544,-819699210830328959], 
(6784361874104123161,6792435444931864417], 
(-8284973647739186524,-8256379601422884439], 
(-5321596150219749918,-5309942391653520721], 
(4723799060589319973,4734570102349898748], 
(5077208816402461873,5091797538025862046], 
(7487792874019513006,7489186250794409815], 
(-1647123563569159073,-1620470309759457440], 
(2779298419000989854,2784870878309140225], 
(320380777495445406,344020839234233467], 
(3112419968835610433,3119057615014897057], 
(-8506504972343997648,-8502433630545238869], 
(-2810980092222625726,-2810271001960433176], 
(344020839234233467,347093216302465566], 
(-8545779729891984163,-8519832061775919840], 
(2633165392330429557,2680549740474631393], 
(6391794517057168460,6431615673158447973], 
(6440766036098716417,6458083094779720657], 
(-8735147357564473077,-8730522574908969951], 
(7484355688408489651,7487792874019513006], 
(4325444310827485194,4347942912351042626], 
(2220347677927119914,2223568835084606157], 
(6802149418704156481,6819713717509862346], 
(4827439895247379322,4851195187346961355], 
(-4016529712384875309,-4009393333251988044], 
(2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ]
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 
ActiveRepairService.java:554 - Cannot start multiple repair sessions over 
the same sstables
ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 - 
Failed creating a merkle tree for [repair 
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, 
[(8195393703879512303,8196334842725538685], 
(8166975326273137878,8182604850967732931], 
(-7246799942440641887,-7227869626613009045], 
(-8371707510273823988,-8365977215604569699], 
(-141862581573028594,-140310864869418908], 
(3732113975108886193,3743105867152786342], 
(4998127507903069087,5008922734235607550], 
(-5115827291264930140,-5111054924035590372], 
(-2475342271852943287,-2447285553369030332], 
(-8318606053827235336,-8308721754886697230], 
(-5208900659917654871,-5202385837264015269], 
(6618737991399272130,6623100721269775102], 
(-4650650128572424858,-4650260492494258461], 
(1886545362164970333,1886646959491599822], 
(-4511817721998311568,-4507491187192881115], 
(8114903118676615937,8132992506844206601], 
(6224957219376301858,6304379125732293904], 
(-3460547504877234383,-3459262416082517136], 
(-167838948111369123,-141862581573028594], 
(481579232521229473,491242114841289497], 
(4052464144722307684,4059745901618136723], 
(1659668187498418295,1679582585970705122], 
(-1118922763210109192,-1093766915505652874], 
(7504365235878319341,7526188885210185292], 
(-79866884352549492,-77667207866300333], 
(8151204058820798561,8154760186218662205], 
(-1040398370287131739,-1033770179677543189], 
(3767057277953758442,3783780844370292025], 
(-6491678058233994892,-6487797181789288329], 
(-916868210769480248,-907141794196269524], 
(-9005441616028750657,-9002220258513351832], 
(8183526518331102304,8186908810225025483], 
(-5685737903527826627,-5672136154194382932], 
(4976122621177738811,4987871287137312689], 
(6051670147160447042,6051686987147911650], 
(-1161640137086921883,-1159172734746043158], 
(6895951547735922309,6899152466544114890], 
(-3357667382515377172,-3356304907368646189], 
(-5370953856683870319,-5345971445444542485], 
(3824272999898372667,3829315045986248983], 
(8132992506844206601,8149858096109302285], 
(3975126143101303723,3980729378827590597], 
(-956691623200349709,-946602525018301692], 
(-82499927325251331,-79866884352549492], 
(3952144214544622998,3955602392726495936], 
(8154760186218662205,8157079055586089583], 
(3840595196718778916,3866458971850198755], 
(-1066905024007783341,-1055954824488508260], 
(-7252356975874511782,-7246799942440641887], 
(-810612946397276081,-792189809286829222], 
(4964519403172053705,4970446606512414858], 
(-5380038118840759647,-5370953856683870319], 
(-3221630728515706463,-3206856875356976885], 
(-1193448110686154165,-1161640137086921883], 
(-3356304907368646189,-3346460884208327912], 
(3466596314109623830,3468144326691723333], 
(-9050241313548454460,-9005441616028750657], 
(402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for 
details)
INFO  [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 
- [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Received merkle tree for 
YYY from /FAKE.33.64
ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 - 
Exception in thread Thread[Repair#1:1,5,RMI Runtime]
com.google.common.util.concurrent.UncheckedExecutionException: 
org.apache.cassandra.exceptions.RepairException: [repair 
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, 
[(8195393703879512303,8196334842725538685], 
(8166975326273137878,8182604850967732931], 
(-7246799942440641887,-7227869626613009045], 
(-8371707510273823988,-8365977215604569699], 
(-141862581573028594,-140310864869418908], 
(3732113975108886193,3743105867152786342], 
(4998127507903069087,5008922734235607550], 
(-5115827291264930140,-5111054924035590372], 
(-2475342271852943287,-2447285553369030332], 
(-8318606053827235336,-8308721754886697230], 
(-5208900659917654871,-5202385837264015269], 
(6618737991399272130,6623100721269775102], 
(-4650650128572424858,-4650260492494258461], 
(1886545362164970333,1886646959491599822], 
(-4511817721998311568,-4507491187192881115], 
(8114903118676615937,8132992506844206601], 
(6224957219376301858,6304379125732293904], 
(-3460547504877234383,-3459262416082517136], 
(-167838948111369123,-141862581573028594], 
(481579232521229473,491242114841289497], 
(4052464144722307684,4059745901618136723], 
(1659668187498418295,1679582585970705122], 
(-1118922763210109192,-1093766915505652874], 
(7504365235878319341,7526188885210185292], 
(-79866884352549492,-77667207866300333], 
(8151204058820798561,8154760186218662205], 
(-1040398370287131739,-1033770179677543189], 
(3767057277953758442,3783780844370292025], 
(-6491678058233994892,-6487797181789288329], 
(-916868210769480248,-907141794196269524], 
(-9005441616028750657,-9002220258513351832], 
(8183526518331102304,8186908810225025483], 
(-5685737903527826627,-5672136154194382932], 
(4976122621177738811,4987871287137312689], 
(6051670147160447042,6051686987147911650], 
(-1161640137086921883,-1159172734746043158], 
(6895951547735922309,6899152466544114890], 
(-3357667382515377172,-3356304907368646189], 
(-5370953856683870319,-5345971445444542485], 
(3824272999898372667,3829315045986248983], 
(8132992506844206601,8149858096109302285], 
(3975126143101303723,3980729378827590597], 
(-956691623200349709,-946602525018301692], 
(-82499927325251331,-79866884352549492], 
(3952144214544622998,3955602392726495936], 
(8154760186218662205,8157079055586089583], 
(3840595196718778916,3866458971850198755], 
(-1066905024007783341,-1055954824488508260], 
(-7252356975874511782,-7246799942440641887], 
(-810612946397276081,-792189809286829222], 
(4964519403172053705,4970446606512414858], 
(-5380038118840759647,-5370953856683870319], 
(-3221630728515706463,-3206856875356976885], 
(-1193448110686154165,-1161640137086921883], 
(-3356304907368646189,-3346460884208327912], 
(3466596314109623830,3468144326691723333], 
(-9050241313548454460,-9005441616028750657], 
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at 
com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525) 
~[guava-18.0.jar:na]
        at 
com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511) 
~[guava-18.0.jar:na]
        at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
~[na:1.8.0_102]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
~[na:1.8.0_102]
        at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)

~[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair 
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, 
[(8195393703879512303,8196334842725538685], 
(8166975326273137878,8182604850967732931], 
(-7246799942440641887,-7227869626613009045], 
(-8371707510273823988,-8365977215604569699], 
(-141862581573028594,-140310864869418908], 
(3732113975108886193,3743105867152786342], 
(4998127507903069087,5008922734235607550], 
(-5115827291264930140,-5111054924035590372], 
(-2475342271852943287,-2447285553369030332], 
(-8318606053827235336,-8308721754886697230], 
(-5208900659917654871,-5202385837264015269], 
(6618737991399272130,6623100721269775102], 
(-4650650128572424858,-4650260492494258461], 
(1886545362164970333,1886646959491599822], 
(-4511817721998311568,-4507491187192881115], 
(8114903118676615937,8132992506844206601], 
(6224957219376301858,6304379125732293904], 
(-3460547504877234383,-3459262416082517136], 
(-167838948111369123,-141862581573028594], 
(481579232521229473,491242114841289497], 
(4052464144722307684,4059745901618136723], 
(1659668187498418295,1679582585970705122], 
(-1118922763210109192,-1093766915505652874], 
(7504365235878319341,7526188885210185292], 
(-79866884352549492,-77667207866300333], 
(8151204058820798561,8154760186218662205], 
(-1040398370287131739,-1033770179677543189], 
(3767057277953758442,3783780844370292025], 
(-6491678058233994892,-6487797181789288329], 
(-916868210769480248,-907141794196269524], 
(-9005441616028750657,-9002220258513351832], 
(8183526518331102304,8186908810225025483], 
(-5685737903527826627,-5672136154194382932], 
(4976122621177738811,4987871287137312689], 
(6051670147160447042,6051686987147911650], 
(-1161640137086921883,-1159172734746043158], 
(6895951547735922309,6899152466544114890], 
(-3357667382515377172,-3356304907368646189], 
(-5370953856683870319,-5345971445444542485], 
(3824272999898372667,3829315045986248983], 
(8132992506844206601,8149858096109302285], 
(3975126143101303723,3980729378827590597], 
(-956691623200349709,-946602525018301692], 
(-82499927325251331,-79866884352549492], 
(3952144214544622998,3955602392726495936], 
(8154760186218662205,8157079055586089583], 
(3840595196718778916,3866458971850198755], 
(-1066905024007783341,-1055954824488508260], 
(-7252356975874511782,-7246799942440641887], 
(-810612946397276081,-792189809286829222], 
(4964519403172053705,4970446606512414858], 
(-5380038118840759647,-5370953856683870319], 
(-3221630728515706463,-3206856875356976885], 
(-1193448110686154165,-1161640137086921883], 
(-3356304907368646189,-3346460884208327912], 
(3466596314109623830,3468144326691723333], 
(-9050241313548454460,-9005441616028750657], 
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at 
org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)

~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)

~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.net
.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_102]
        ... 4 common frames omitted
WARN  [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - 
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] YYY sync failed
ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 - 
[repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Session completed with the 
following error
org.apache.cassandra.exceptions.RepairException: [repair 
#071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, 
[(8195393703879512303,8196334842725538685], 
(8166975326273137878,8182604850967732931], 
(-7246799942440641887,-7227869626613009045], 
(-8371707510273823988,-8365977215604569699], 
(-141862581573028594,-140310864869418908], 
(3732113975108886193,3743105867152786342], 
(4998127507903069087,5008922734235607550], 
(-5115827291264930140,-5111054924035590372], 
(-2475342271852943287,-2447285553369030332], 
(-8318606053827235336,-8308721754886697230], 
(-5208900659917654871,-5202385837264015269], 
(6618737991399272130,6623100721269775102], 
(-4650650128572424858,-4650260492494258461], 
(1886545362164970333,1886646959491599822], 
(-4511817721998311568,-4507491187192881115], 
(8114903118676615937,8132992506844206601], 
(6224957219376301858,6304379125732293904], 
(-3460547504877234383,-3459262416082517136], 
(-167838948111369123,-141862581573028594], 
(481579232521229473,491242114841289497], 
(4052464144722307684,4059745901618136723], 
(1659668187498418295,1679582585970705122], 
(-1118922763210109192,-1093766915505652874], 
(7504365235878319341,7526188885210185292], 
(-79866884352549492,-77667207866300333], 
(8151204058820798561,8154760186218662205], 
(-1040398370287131739,-1033770179677543189], 
(3767057277953758442,3783780844370292025], 
(-6491678058233994892,-6487797181789288329], 
(-916868210769480248,-907141794196269524], 
(-9005441616028750657,-9002220258513351832], 
(8183526518331102304,8186908810225025483], 
(-5685737903527826627,-5672136154194382932], 
(4976122621177738811,4987871287137312689], 
(6051670147160447042,6051686987147911650], 
(-1161640137086921883,-1159172734746043158], 
(6895951547735922309,6899152466544114890], 
(-3357667382515377172,-3356304907368646189], 
(-5370953856683870319,-5345971445444542485], 
(3824272999898372667,3829315045986248983], 
(8132992506844206601,8149858096109302285], 
(3975126143101303723,3980729378827590597], 
(-956691623200349709,-946602525018301692], 
(-82499927325251331,-79866884352549492], 
(3952144214544622998,3955602392726495936], 
(8154760186218662205,8157079055586089583], 
(3840595196718778916,3866458971850198755], 
(-1066905024007783341,-1055954824488508260], 
(-7252356975874511782,-7246799942440641887], 
(-810612946397276081,-792189809286829222], 
(4964519403172053705,4970446606512414858], 
(-5380038118840759647,-5370953856683870319], 
(-3221630728515706463,-3206856875356976885], 
(-1193448110686154165,-1161640137086921883], 
(-3356304907368646189,-3346460884208327912], 
(3466596314109623830,3468144326691723333], 
(-9050241313548454460,-9005441616028750657], 
(402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
        at 
org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)

~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)

~[apache-cassandra-3.0.14.jar:3.0.14]
        at org.apache.cassandra.net
.MessageDeliveryTask.run(MessageDeliveryTask.java:67) 
~[apache-cassandra-3.0.14.jar:3.0.14]
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
~[na:1.8.0_102]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
~[na:1.8.0_102]
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[na:1.8.0_102]
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[na:1.8.0_102]
        at 
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)

[apache-cassandra-3.0.14.jar:3.0.14]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 
-- 
-----------------
Alexander Dejanovski
France
@alexanderdeja
 
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 
-- 
-----------------
Alexander Dejanovski
France
@alexanderdeja
 
Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 
 
The contents of this e-mail are intended for the named addressee only. It 
contains information that may be confidential. Unless you are the named 
addressee or an authorized designee, you may not copy or use it, or 
disclose it to anyone else. If you received it in error please notify us 
immediately and then destroy it. Dynatrace Austria GmbH (registration 
number FN 91482h) is a company registered in Linz whose registered office 
is at 4040 Linz, Austria, Freistädterstraße 313 


The information in this Internet Email is confidential and may be legally 
privileged. It is intended solely for the addressee. Access to this Email 
by anyone else is unauthorized. If you are not the intended recipient, any 
disclosure, copying, distribution or any action taken or omitted to be 
taken in reliance on it, is prohibited and may be unlawful. When addressed 
to our clients any opinions or advice contained in this Email are subject 
to the terms and conditions expressed in any applicable governing The Home 
Depot terms of business or client engagement letter. The Home Depot 
disclaims all responsibility and liability for the accuracy and content of 
this attachment and for any damages or losses arising from any 
inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other 
items of a destructive nature, which may be contained in this attachment 
and shall not be liable for direct, indirect, consequential or special 
damages in connection with this e-mail message or its attachment.



**

This email and any attachments may contain information that is confidential and/or privileged
for the sole use of the intended recipient.  Any use, review, disclosure, copying, distribution
or reliance by others, and any forwarding of this email or its contents, without the express
permission of the sender is strictly prohibited by law.  If you are not the intended recipient,
please contact the sender immediately, delete the e-mail and destroy all copies.
**
Mime
View raw message