cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paulo Motta <pauloricard...@gmail.com>
Subject Re: Multi-node repair fails after upgrading to 3.0.14
Date Tue, 19 Sep 2017 06:55:40 GMT
In 4.0 anti-compaction is no longer run after full repairs, so we
should probably backport this behavior to 3.0, given there are known
limitations with incremental repair on 3.0 and non-incremental users
may want to run keep running full repairs without the additional cost
of anti-compaction.

Would you mind opening a ticket for this?

2017-09-19 1:33 GMT-05:00 Steinmaurer, Thomas
<thomas.steinmaurer@dynatrace.com>:
> Hi Kurt,
>
>
>
> thanks for the link!
>
>
>
> Honestly, a pity, that in 3.0, we can’t get the simple, reliable and
> predictable way back to run a full repair for very low data volume CFs being
> kicked off on all nodes in parallel, without all the magic behind the scene
> introduced by incremental repairs, even if not used, as anticompaction even
> with –full has been introduced with 2.2+ J
>
>
>
>
>
> Regards,
>
> Thomas
>
>
>
> From: kurt greaves [mailto:kurt@instaclustr.com]
> Sent: Dienstag, 19. September 2017 06:24
> To: User <user@cassandra.apache.org>
>
>
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-13153 implies full repairs
> still triggers anti-compaction on non-repaired SSTables (if I'm reading that
> right), so might need to make sure you don't run multiple repairs at the
> same time across your nodes (if your using vnodes), otherwise could still
> end up trying to run anti-compaction on the same SSTable from 2 repairs.
>
>
>
> Anyone else feel free to jump in and correct me if my interpretation is
> wrong.
>
>
>
> On 18 September 2017 at 17:11, Steinmaurer, Thomas
> <thomas.steinmaurer@dynatrace.com> wrote:
>
> Jeff,
>
>
>
> what should be the expected outcome when running with 3.0.14:
>
>
>
> nodetool repair –full –pr keyspace cfs
>
>
>
> ·         Should –full trigger anti-compaction?
>
> ·         Should this be the same operation as nodetool repair –pr keyspace
> cfs in 2.1?
>
> ·         Should I be able to  run this on several nodes in parallel as in
> 2.1 without troubles, where incremental repair was not the default?
>
>
>
> Still confused if I’m missing something obvious. Sorry about that. J
>
>
>
> Thanks,
>
> Thomas
>
>
>
> From: Jeff Jirsa [mailto:jjirsa@gmail.com]
> Sent: Montag, 18. September 2017 16:10
>
>
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Sorry I may be wrong about the cause - didn't see -full
>
>
>
> Mea culpa, its early here and I'm not awake
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Sep 18, 2017, at 7:01 AM, Steinmaurer, Thomas
> <thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Jeff,
>
>
>
> understood. That’s quite a change then coming from 2.1 from an operational
> POV.
>
>
>
> Thanks again.
>
>
>
> Thomas
>
>
>
> From: Jeff Jirsa [mailto:jjirsa@gmail.com]
> Sent: Montag, 18. September 2017 15:56
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> The command you're running will cause anticompaction and the range borders
> for all instances at the same time
>
>
>
> Since only one repair session can anticompact any given sstable, it's almost
> guaranteed to fail
>
>
>
> Run it on one instance at a time
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Sep 18, 2017, at 1:11 AM, Steinmaurer, Thomas
> <thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Alex,
>
>
>
> I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel
> and this may pop up now:
>
>
>
> 0.176.38.128 (progress: 1%)
>
> [2017-09-18 07:59:17,145] Some repair failed
>
> [2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds
>
> error: Repair job has failed with the error message: [2017-09-18
> 07:59:17,145] Some repair failed
>
> -- StackTrace --
>
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-09-18 07:59:17,145] Some repair failed
>
>         at
> org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
>
>         at
> org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
>
>         at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
>
>         at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
>
>         at
> com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
>
>         at
> com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
>
>
>
> 2017-09-18 07:59:17 repair finished
>
>
>
>
>
> If running the above nodetool call sequentially on all nodes, repair
> finishes without printing a stack trace.
>
>
>
> The error message and stack trace isn’t really useful here. Any further
> ideas/experiences?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> From: Alexander Dejanovski [mailto:alex@thelastpickle.com]
> Sent: Freitag, 15. September 2017 11:30
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Right, you should indeed add the "--full" flag to perform full repairs, and
> you can then keep the "-pr" flag.
>
>
>
> I'd advise to monitor the status of your SSTables as you'll probably end up
> with a pool of SSTables marked as repaired, and another pool marked as
> unrepaired which won't be compacted together (hence the suggestion of
> running subrange repairs).
>
> Use sstablemetadata to check on the "Repaired at" value for each. 0 means
> unrepaired and any other value (a timestamp) means the SSTable has been
> repaired.
>
> I've had behaviors in the past where running "-pr" on the whole cluster
> would still not mark all SSTables as repaired, but I can't say if that
> behavior has changed in latest versions.
>
>
>
> Having separate pools of SStables that cannot be compacted means that you
> might have tombstones that don't get evicted due to partitions living in
> both states (repaired/unrepaired).
>
>
>
> To sum up the recommendations :
>
> - Run a full repair with both "--full" and "-pr" and check that SSTables are
> properly marked as repaired
>
> - Use a tight repair schedule to avoid keeping partitions for too long in
> both repaired and unrepaired state
>
> - Switch to subrange repair if you want to fully avoid marking SSTables as
> repaired (which you don't need anyway since you're not using incremental
> repairs). If you wish to do this, you'll have to mark back all your sstables
> to unrepaired, using nodetool sstablerepairedset.
>
>
>
> Cheers,
>
>
>
> On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas
> <thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Alex,
>
>
>
> thanks a lot. Somehow missed that incremental repairs are the default now.
>
>
>
> We have been happy with full repair so far, cause data what we currently
> manually invoke for being prepared is a small (~1GB or even smaller).
>
>
>
> So I guess with full repairs across all nodes, we still can stick with the
> partition range (-pr) option, but with 3.0 we additionally have to provide
> the –full option, right?
>
>
>
> Thanks again,
>
> Thomas
>
>
>
> From: Alexander Dejanovski [mailto:alex@thelastpickle.com]
> Sent: Freitag, 15. September 2017 09:45
> To: user@cassandra.apache.org
> Subject: Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Hi Thomas,
>
>
>
> in 2.1.18, the default repair mode was full repair while since 2.2 it is
> incremental repair.
>
> So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't
> trigger the same operation.
>
>
>
> Incremental repair cannot run on more than one node at a time on a cluster,
> because you risk to have conflicts with sessions trying to anticompact and
> run validation compactions on the same SSTables (which will make the
> validation phase fail, like your logs are showing).
>
> Furthermore, you should never use "-pr" with incremental repair because it
> is useless in that mode, and won't properly perform anticompaction on all
> nodes.
>
>
>
> If you were happy with full repairs in 2.1.18, I'd suggest to stick with
> those in 3.0.14 as well because there are still too many caveats with
> incremental repairs that should hopefully be fixed in 4.0+.
>
> Note that full repair will also trigger anticompaction and mark SSTables as
> repaired in your release of Cassandra, and only full subrange repairs are
> the only flavor that will skip anticompaction.
>
>
>
> You will need some tooling to help with subrange repairs though, and I'd
> recommend to use Reaper which handles automation for you :
> http://cassandra-reaper.io/
>
>
>
> If you decide to stick with incremental repairs, first perform a rolling
> restart of your cluster to make sure no repair session still runs, and run
> "nodetool repair" on a single node at a time. Move on to the next node only
> when nodetool or the logs show that repair is over (which will include the
> anticompaction phase).
>
>
>
> Cheers,
>
>
>
>
>
>
>
> On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas
> <thomas.steinmaurer@dynatrace.com> wrote:
>
> Hello,
>
>
>
> we are currently in the process of upgrading from 2.1.18 to 3.0.14. After
> upgrading a few test environments, we start to see some suspicious log
> entries regarding repair issues.
>
>
>
> We have a cron job on all nodes basically executing the following repair
> call on a daily basis:
>
>
>
> nodetool repair –pr <list of CFs>
>
>
>
> This gets started on all nodes at the same time. While this has worked with
> 2.1.18 (at least we haven’t seen anything suspicious in Cassandra log), with
> 3.0.14 we get something similar like that on all nodes (see below; IP
> addresses and KS/CF faked).
>
>
>
> Any pointers are appreciated. Thanks.
>
> Thomas
>
>
>
>
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 - [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
> /FAKE.35.153, /FAKE.34.171 on range
> [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931],
> (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699],
> (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342],
> (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372],
> (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230],
> (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102],
> (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822],
> (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601],
> (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136],
> (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497],
> (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122],
> (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292],
> (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205],
> (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025],
> (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524],
> (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483],
> (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689],
> (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158],
> (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189],
> (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983],
> (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597],
> (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492],
> (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583],
> (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260],
> (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222],
> (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319],
> (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883],
> (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333],
> (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
>
> INFO  [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Requesting merkle trees for YYY (to
> [/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 - [repair
> #075d2720-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
> /FAKE.35.57, /FAKE.34.171 on range
> [(-5410955131843184047,-5390722609201388849],
> (-2429793939970389370,-2402273315769352748],
> (8085575576842594575,8086965740279021106],
> (-8802193901675845653,-8790472027607832351],
> (-3900412470120874591,-3892641480459306647],
> (5455804264750818305,5465037357825542970],
> (4930767198829659527,4939587074207662799],
> (8086965740279021106,8087442741329154201],
> (-8933201045321260661,-8926445549049070674],
> (-4841328524165418854,-4838895482794593338],
> (628107265570603622,682509946926464280],
> (7043245467621414187,7055126022831789025],
> (624871765540463735,627374995781897409],
> (9219228482330263660,9221294940422311559],
> (-2335215188301493066,-2315034243278984017],
> (-6216599212198827632,-6211460136507414133],
> (-3276490559558850323,-3273110814046238767],
> (7204991007334459472,7214826985711309418],
> (1815809811279373566,1846961604192445001],
> (8743912118048160970,8751518028513315549],
> (-9204701745739426439,-9200185935622985719],
> (7926527126882050773,7941554683778488797],
> (-1307707180308444994,-1274682085495751899],
> (8354147540115782875,8358523989614737607],
> (-5418282332713406631,-5415077779309282099],
> (2436459402559272117,2441988676982099299],
> (3718536403335860823,3722612994041485455],
> (613072756805503750,624871765540463735],
> (3348338086790700840,3352211222405547568],
> (7941554683778488797,7949493423692701952],
> (-319434281654804937,-312029192727845573],
> (6560852500852746428,6577389024948220270],
> (3468872066705970665,3469611600245714638],
> (-4691666557374653357,-4691655196509961176],
> (1517867403379415135,1524487506534693145],
> (-7422802504145504981,-7402342710717945820],
> (1766173970230627147,1798177111825664424],
> (-4402244320098942276,-4399211415161403124],
> (-6024004998426219783,-6013031695922198269],
> (-6930256177103656090,-6926607739769548590],
> (-3578196517591043633,-3561058036097453011],
> (-7923695406605667073,-7885733750925254770],
> (-8063817427378208986,-8061632046535266293],
> (-7341733690813664665,-7287847431444491429],
> (-8327456680139422800,-8322944736494135341],
> (885662136143255295,898069608029365525],
> (-5555349213881020405,-5555145874347997903],
> (-4895525453201761515,-4892963646364151947],
> (4925572085497096406,4926057814710187377],
> (7785700473508314561,7813617525484553901],
> (-1598850926292630432,-1593204162764888475],
> (-2583030464997858536,-2581717714589673598],
> (7780476570449984013,7785700473508314561],
> (-3558045406807923433,-3550343462935144866],
> (7571721681971063633,7578715430118769516],
> (8865905084309486257,8879414598472867800],
> (-4984170160639031182,-4973560491294201055],
> (-5469249030580263062,-5432196900285376733],
> (-340636581410346966,-324449230085298889],
> (9120471287581648922,9127588925404102522],
> (8366987539912250171,8368772126617619299],
> (-9125573241533654210,-9123870950381500195],
> (1193377366975272040,1205407211929215537],
> (8879414598472867800,8900714457425949896],
> (7055126022831789025,7065799572034111661],
> (-1588707445511226686,-1539910209795101605],
> (6990922382177634221,7007948980474566617]] for XXX.[YYY, ZZZ]
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 - [repair
> #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
> /FAKE.35.57, /FAKE.34.90 on range
> [(-3078252446668567055,-3069777768104388887],
> (3648099278098819362,3665843214181043507],
> (5797678959887947223,5820452778888079981],
> (-2642545928835445644,-2631494184435884123],
> (-1612062874902452641,-1610692114449831135],
> (-1707971415501754602,-1692411707392102715],
> (291977720343160936,302780457185309086],
> (-8155767539645304082,-8140296327332660530],
> (5553475010776119070,5565820507856584918],
> (3560286576304809625,3566372473455883512],
> (5226123540640728306,5228400303590993685],
> (3421729237821036215,3428685863718017921],
> (1227377289103207735,1240348980976161342],
> (3246531324407742399,3247845316596819400],
> (5304633131413824443,5306083993155890840],
> (-8619514398407319973,-8606439320645645548],
> (5911017071945481872,5912872298340550901],
> (-5637986723275926091,-5632042588860715397],
> (-7132684921710202809,-7132365563419969085],
> (8246342224027692193,8283424185055442516],
> (-4044458382788374703,-4036186919446747968],
> (-5613445605139257263,-5596969946550242654],
> (1083135773045625799,1084813234330856986],
> (-8140296327332660530,-8138240730730282332],
> (2996936679493130945,3002752531484603869],
> (-4490677154018906600,-4487029511866186581],
> (8456154651893349894,8463606226270547769],
> (-2656313914110802764,-2642545928835445644],
> (7193156006022031982,7197068598281426786],
> (8318481021259042556,8345872746994439720],
> (-1692411707392102715,-1673006435564531865],
> (1097191643473406518,1104630053604629743],
> (-3613177001569108681,-3599878627600515791],
> (-1662308889719802294,-1656482847672279831],
> (-6811080679276087557,-6804940650782637386],
> (7392275851743093948,7393304112887579707],
> (4240513765329086484,4257716503015670881],
> (-5036265422304136579,-5029426061972234042],
> (-4230577892781670592,-4228053239714848467],
> (-8159151100170546488,-8155767539645304082],
> (-5492523660879132140,-5478464594772026508],
> (3421125671176312710,3421729237821036215],
> (5262611881402902860,5275876653180871852],
> (1577722798587790899,1613215617842269344],
> (6666156845317387896,6676801654642938276],
> (5155814764198615180,5168412530555049843],
> (-6824229014637940118,-6819748162411093540],
> (-3666684985038092211,-3613177001569108681],
> (-8874623295866980549,-8872063750661463000],
> (-3599878627600515791,-3599733923716673512],
> (-5960723066215554398,-5949787783010678970],
> (258158907638948329,259127284829404565],
> (6661762711488840391,6666156845317387896],
> (-6367100347601878936,-6363911697345955296],
> (-6250542414915159381,-6229550501272299394],
> (-6917261891898915460,-6904807338337882298],
> (-7449919126757571330,-7438527202807402113],
> (-6904807338337882298,-6881332882762615208],
> (-7942161058117423218,-7938222136384107977],
> (781504901963705642,791302021433124535],
> (6527741860207715339,6537050054260012793],
> (7186200312153798494,7187207897161667549]] for XXX.[YYY, ZZZ]
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 - [repair
> #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync /FAKE.33.64,
> /FAKE.35.153, /FAKE.34.90 on range
> [(3075097238958732453,3095612227388458535],
> (-3791204388712221856,-3785200705970537637],
> (8410714452405602873,8417924634749091961],
> (5114137550939144346,5121098642749542165],
> (2740032807677652551,2746972622905345611],
> (2197312222134575375,2220347677927119914],
> (4864988928009437565,4869276508135102594],
> (-3451050463660365008,-3450091362870187260],
> (7424782039380072835,7472858232992173041],
> (-5338516343760774634,-5325637469358550877],
> (2731359217194343571,2734544646630861913],
> (382200642783712616,390184326306122373],
> (430251086609083397,431821763463501369],
> (-5338895526845396808,-5338516343760774634],
> (-2677272836643942252,-2671813555202977248],
> (-2132202487362637132,-2128425620095084071],
> (2165546227633605752,2174360971273026818],
> (2763720958753148061,2764848409944217454],
> (3538741049060169597,3547917038270802802],
> (-6741608677619654195,-6731111079555719429],
> (2764848409944217454,2779298419000989854],
> (374883901669627753,382200642783712616],
> (-2706148319144852162,-2694312255222799319],
> (7132646356350938192,7136352706932176819],
> (194821142331236531,228318513249291325],
> (-2047674613927860008,-2032759017860426681],
> (-2032759017860426681,-2011077831196488705],
> (7472858232992173041,7477981949405760270],
> (431821763463501369,449911816694531926],
> (4347942912351042626,4349624658068191606],
> (311672118330363382,320380777495445406],
> (3099877433733294763,3112419968835610433],
> (-7182182373680571317,-7177421545891675761],
> (-6742074278675667310,-6741608677619654195],
> (-2770159943814078393,-2757633164368231752],
> (3095612227388458535,3099877433733294763],
> (-3741816028443553809,-3741630281022069399],
> (-738951315633674297,-727109613092919188],
> (4851195187346961355,4864988928009437565],
> (-769229207399684543,-744901822064683937],
> (-4143548408671617625,-4141029975015318194],
> (-7957562451432593113,-7952882742232241654],
> (-511680378144663402,-494227476671052554],
> (-832115128813031544,-819699210830328959],
> (6784361874104123161,6792435444931864417],
> (-8284973647739186524,-8256379601422884439],
> (-5321596150219749918,-5309942391653520721],
> (4723799060589319973,4734570102349898748],
> (5077208816402461873,5091797538025862046],
> (7487792874019513006,7489186250794409815],
> (-1647123563569159073,-1620470309759457440],
> (2779298419000989854,2784870878309140225],
> (320380777495445406,344020839234233467],
> (3112419968835610433,3119057615014897057],
> (-8506504972343997648,-8502433630545238869],
> (-2810980092222625726,-2810271001960433176],
> (344020839234233467,347093216302465566],
> (-8545779729891984163,-8519832061775919840],
> (2633165392330429557,2680549740474631393],
> (6391794517057168460,6431615673158447973],
> (6440766036098716417,6458083094779720657],
> (-8735147357564473077,-8730522574908969951],
> (7484355688408489651,7487792874019513006],
> (4325444310827485194,4347942912351042626],
> (2220347677927119914,2223568835084606157],
> (6802149418704156481,6819713717509862346],
> (4827439895247379322,4851195187346961355],
> (-4016529712384875309,-4009393333251988044],
> (2018021481024026660,2049270980733207626]] for XXX.[YYY, ZZZ]
>
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471
> ActiveRepairService.java:554 - Cannot start multiple repair sessions over
> the same sstables
>
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 -
> Failed creating a merkle tree for [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
> [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931],
> (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699],
> (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342],
> (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372],
> (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230],
> (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102],
> (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822],
> (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601],
> (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136],
> (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497],
> (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122],
> (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292],
> (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205],
> (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025],
> (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524],
> (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483],
> (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689],
> (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158],
> (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189],
> (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983],
> (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597],
> (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492],
> (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583],
> (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260],
> (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222],
> (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319],
> (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883],
> (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333],
> (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for details)
>
> INFO  [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176 -
> [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Received merkle tree for YYY
> from /FAKE.33.64
>
> ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 -
> Exception in thread Thread[Repair#1:1,5,RMI Runtime]
>
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
> [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931],
> (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699],
> (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342],
> (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372],
> (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230],
> (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102],
> (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822],
> (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601],
> (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136],
> (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497],
> (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122],
> (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292],
> (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205],
> (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025],
> (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524],
> (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483],
> (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689],
> (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158],
> (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189],
> (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983],
> (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597],
> (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492],
> (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583],
> (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260],
> (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222],
> (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319],
> (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883],
> (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333],
> (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at
> com.google.common.util.concurrent.Futures.wrapAndThrowUnchecked(Futures.java:1525)
> ~[guava-18.0.jar:na]
>
>         at
> com.google.common.util.concurrent.Futures.getUnchecked(Futures.java:1511)
> ~[guava-18.0.jar:na]
>
>         at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_102]
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_102]
>
>         at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
> [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931],
> (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699],
> (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342],
> (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372],
> (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230],
> (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102],
> (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822],
> (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601],
> (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136],
> (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497],
> (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122],
> (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292],
> (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205],
> (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025],
> (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524],
> (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483],
> (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689],
> (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158],
> (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189],
> (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983],
> (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597],
> (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492],
> (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583],
> (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260],
> (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222],
> (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319],
> (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883],
> (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333],
> (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at
> org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_102]
>
>         ... 4 common frames omitted
>
> WARN  [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 - [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] YYY sync failed
>
> ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 -
> [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Session completed with the
> following error
>
> org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY,
> [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931],
> (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699],
> (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342],
> (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372],
> (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230],
> (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102],
> (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822],
> (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601],
> (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136],
> (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497],
> (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122],
> (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292],
> (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205],
> (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025],
> (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524],
> (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483],
> (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689],
> (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158],
> (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189],
> (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983],
> (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597],
> (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492],
> (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583],
> (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260],
> (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222],
> (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319],
> (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883],
> (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333],
> (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at
> org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:164)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_102]
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_102]
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_102]
>
>         at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313
>
> --
>
> -----------------
>
> Alexander Dejanovski
>
> France
>
> @alexanderdeja
>
>
>
> Consultant
>
> Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313
>
> --
>
> -----------------
>
> Alexander Dejanovski
>
> France
>
> @alexanderdeja
>
>
>
> Consultant
>
> Apache Cassandra Consulting
>
> http://www.thelastpickle.com
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or disclose
> it to anyone else. If you received it in error please notify us immediately
> and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h)
> is a company registered in Linz whose registered office is at 4040 Linz,
> Austria, Freistädterstraße 313


---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message