cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kurt greaves <k...@instaclustr.com>
Subject RE: Multi-node repair fails after upgrading to 3.0.14
Date Tue, 19 Sep 2017 21:53:39 GMT
You're right of course. Part of the reason it's changing so frequently is
to try and improve repairs so that they at least actually work reliably. C*
3 hasn't been the smoothest ride for repairs. Incremental repairs wasn't
really ready for 3.0 so it was a mistake to make it a default.
Unfortunately it's hard to change that back now as it will just lead to
more confusion and problems for users unaware of the change.

On 20 Sep. 2017 00:25, "Durity, Sean R" <SEAN_R_DURITY@homedepot.com> wrote:

> Required maintenance for a cluster should not be this complicated and
> should not be changing so often. To me, this is a major flaw in Cassandra.
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* Steinmaurer, Thomas [mailto:thomas.steinmaurer@dynatrace.com]
> *Sent:* Tuesday, September 19, 2017 2:33 AM
> *To:* user@cassandra.apache.org
> *Subject:* RE: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Hi Kurt,
>
>
>
> thanks for the link!
>
>
>
> Honestly, a pity, that in 3.0, we can’t get the simple, reliable and
> predictable way back to run a full repair for very low data volume CFs
> being kicked off on all nodes in parallel, without all the magic behind the
> scene introduced by incremental repairs, even if not used, as
> anticompaction even with –full has been introduced with 2.2+ J
>
>
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* kurt greaves [mailto:kurt@instaclustr.com <kurt@instaclustr.com>]
> *Sent:* Dienstag, 19. September 2017 06:24
> *To:* User <user@cassandra.apache.org>
> *Subject:* Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-13153
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D13153&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=GIeM1jPMqIq_3Mi_6ksfEhOMVQibPPWLTd_Luolj3a0&e=>
> implies full repairs still triggers anti-compaction on non-repaired
> SSTables (if I'm reading that right), so might need to make sure you don't
> run multiple repairs at the same time across your nodes (if your using
> vnodes), otherwise could still end up trying to run anti-compaction on the
> same SSTable from 2 repairs.
>
>
>
> Anyone else feel free to jump in and correct me if my interpretation is
> wrong.
>
>
>
> On 18 September 2017 at 17:11, Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
> Jeff,
>
>
>
> what should be the expected outcome when running with 3.0.14:
>
>
>
> nodetool repair –full –pr keyspace cfs
>
>
>
> ·         Should –full trigger anti-compaction?
>
> ·         Should this be the same operation as nodetool repair –pr
> keyspace cfs in 2.1?
>
> ·         Should I be able to  run this on several nodes in parallel as
> in 2.1 without troubles, where incremental repair was not the default?
>
>
>
> Still confused if I’m missing something obvious. Sorry about that. J
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com]
> *Sent:* Montag, 18. September 2017 16:10
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Sorry I may be wrong about the cause - didn't see -full
>
>
>
> Mea culpa, its early here and I'm not awake
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Sep 18, 2017, at 7:01 AM, Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Jeff,
>
>
>
> understood. That’s quite a change then coming from 2.1 from an operational
> POV.
>
>
>
> Thanks again.
>
>
>
> Thomas
>
>
>
> *From:* Jeff Jirsa [mailto:jjirsa@gmail.com <jjirsa@gmail.com>]
> *Sent:* Montag, 18. September 2017 15:56
> *To:* user@cassandra.apache.org
> *Subject:* Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> The command you're running will cause anticompaction and the range borders
> for all instances at the same time
>
>
>
> Since only one repair session can anticompact any given sstable, it's
> almost guaranteed to fail
>
>
>
> Run it on one instance at a time
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Sep 18, 2017, at 1:11 AM, Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Alex,
>
>
>
> I now ran nodetool repair –full –pr keyspace cfs on all nodes in parallel
> and this may pop up now:
>
>
>
> 0.176.38.128 (progress: 1%)
>
> [2017-09-18 07:59:17,145] Some repair failed
>
> [2017-09-18 07:59:17,151] Repair command #3 finished in 0 seconds
>
> error: Repair job has failed with the error message: [2017-09-18
> 07:59:17,145] Some repair failed
>
> -- StackTrace --
>
> java.lang.RuntimeException: Repair job has failed with the error message:
> [2017-09-18 07:59:17,145] Some repair failed
>
>         at org.apache.cassandra.tools.RepairRunner.progress(
> RepairRunner.java:115)
>
>         at org.apache.cassandra.utils.progress.jmx.
> JMXNotificationProgressListener.handleNotification(
> JMXNotificationProgressListener.java:77)
>
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.
> dispatchNotification(ClientNotifForwarder.java:583)
>
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$
> NotifFetcher.doRun(ClientNotifForwarder.java:533)
>
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$
> NotifFetcher.run(ClientNotifForwarder.java:452)
>
>         at com.sun.jmx.remote.internal.ClientNotifForwarder$
> LinearExecutor$1.run(ClientNotifForwarder.java:108)
>
>
>
> 2017-09-18 07:59:17 repair finished
>
>
>
>
>
> If running the above nodetool call sequentially on all nodes, repair
> finishes without printing a stack trace.
>
>
>
> The error message and stack trace isn’t really useful here. Any further
> ideas/experiences?
>
>
>
> Thanks,
>
> Thomas
>
>
>
> *From:* Alexander Dejanovski [mailto:alex@thelastpickle.com
> <alex@thelastpickle.com>]
> *Sent:* Freitag, 15. September 2017 11:30
> *To:* user@cassandra.apache.org
> *Subject:* Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Right, you should indeed add the "--full" flag to perform full repairs,
> and you can then keep the "-pr" flag.
>
>
>
> I'd advise to monitor the status of your SSTables as you'll probably end
> up with a pool of SSTables marked as repaired, and another pool marked as
> unrepaired which won't be compacted together (hence the suggestion of
> running subrange repairs).
>
> Use sstablemetadata to check on the "Repaired at" value for each. 0 means
> unrepaired and any other value (a timestamp) means the SSTable has been
> repaired.
>
> I've had behaviors in the past where running "-pr" on the whole cluster
> would still not mark all SSTables as repaired, but I can't say if that
> behavior has changed in latest versions.
>
>
>
> Having separate pools of SStables that cannot be compacted means that you
> might have tombstones that don't get evicted due to partitions living in
> both states (repaired/unrepaired).
>
>
>
> To sum up the recommendations :
>
> - Run a full repair with both "--full" and "-pr" and check that SSTables
> are properly marked as repaired
>
> - Use a tight repair schedule to avoid keeping partitions for too long in
> both repaired and unrepaired state
>
> - Switch to subrange repair if you want to fully avoid marking SSTables as
> repaired (which you don't need anyway since you're not using incremental
> repairs). If you wish to do this, you'll have to mark back all your
> sstables to unrepaired, using nodetool sstablerepairedset
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.datastax.com_en_cassandra_2.1_cassandra_tools_toolsSSTableRepairedSet.html&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=wwVG6PjMgeF54rS6Xcpuj506-JJRwBO9r602jXKKceg&e=>
> .
>
>
>
> Cheers,
>
>
>
> On Fri, Sep 15, 2017 at 10:27 AM Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
> Hi Alex,
>
>
>
> thanks a lot. Somehow missed that incremental repairs are the default now.
>
>
>
> We have been happy with full repair so far, cause data what we currently
> manually invoke for being prepared is a small (~1GB or even smaller).
>
>
>
> So I guess with full repairs across all nodes, we still can stick with the
> partition range (-pr) option, but with 3.0 we additionally have to provide
> the –full option, right?
>
>
>
> Thanks again,
>
> Thomas
>
>
>
> *From:* Alexander Dejanovski [mailto:alex@thelastpickle.com]
> *Sent:* Freitag, 15. September 2017 09:45
> *To:* user@cassandra.apache.org
> *Subject:* Re: Multi-node repair fails after upgrading to 3.0.14
>
>
>
> Hi Thomas,
>
>
>
> in 2.1.18, the default repair mode was full repair while since 2.2 it is
> incremental repair.
>
> So running "nodetool repair -pr" since your upgrade to 3.0.14 doesn't
> trigger the same operation.
>
>
>
> Incremental repair cannot run on more than one node at a time on a
> cluster, because you risk to have conflicts with sessions trying to
> anticompact and run validation compactions on the same SSTables (which will
> make the validation phase fail, like your logs are showing).
>
> Furthermore, you should never use "-pr" with incremental repair because it
> is useless in that mode, and won't properly perform anticompaction on all
> nodes.
>
>
>
> If you were happy with full repairs in 2.1.18, I'd suggest to stick with
> those in 3.0.14 as well because there are still too many caveats with
> incremental repairs that should hopefully be fixed in 4.0+.
>
> Note that full repair will also trigger anticompaction and mark SSTables
> as repaired in your release of Cassandra, and only full subrange repairs
> are the only flavor that will skip anticompaction.
>
>
>
> You will need some tooling to help with subrange repairs though, and I'd
> recommend to use Reaper which handles automation for you :
> http://cassandra-reaper.io/
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra-2Dreaper.io_&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=e_PVNeJObvX9IBhvXg-LvG2O3Kg1jPTywMUYlFiM8Xw&e=>
>
>
>
> If you decide to stick with incremental repairs, first perform a rolling
> restart of your cluster to make sure no repair session still runs, and run
> "nodetool repair" on a single node at a time. Move on to the next node only
> when nodetool or the logs show that repair is over (which will include the
> anticompaction phase).
>
>
>
> Cheers,
>
>
>
>
>
>
>
> On Fri, Sep 15, 2017 at 8:42 AM Steinmaurer, Thomas <
> thomas.steinmaurer@dynatrace.com> wrote:
>
> Hello,
>
>
>
> we are currently in the process of upgrading from 2.1.18 to 3.0.14. After
> upgrading a few test environments, we start to see some suspicious log
> entries regarding repair issues.
>
>
>
> We have a cron job on all nodes basically executing the following repair
> call on a daily basis:
>
>
>
> nodetool repair –pr <list of CFs>
>
>
>
> This gets started on all nodes at the same time. While this has worked
> with 2.1.18 (at least we haven’t seen anything suspicious in Cassandra
> log), with 3.0.14 we get something similar like that on all nodes (see
> below; IP addresses and KS/CF faked).
>
>
>
> Any pointers are appreciated. Thanks.
>
> Thomas
>
>
>
>
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,036 RepairSession.java:224 -
> [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync
> /FAKE.33.64, /FAKE.35.153, /FAKE.34.171 on range [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699], (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822], (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497], (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205], (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483], (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189], (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492], (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319], (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883], (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]] for XXX.[YYY, ZZZ]
>
> INFO  [Repair#1:1] 2017-09-15 03:00:28,419 RepairJob.java:172 - [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Requesting merkle trees for YYY
> (to [/FAKE.35.153, /FAKE.34.171, /FAKE.33.64])
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,434 RepairSession.java:224 -
> [repair #075d2720-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync
> /FAKE.33.64, /FAKE.35.57, /FAKE.34.171 on range [(-5410955131843184047,-5390722609201388849],
> (-2429793939970389370,-2402273315769352748], (8085575576842594575,8086965740279021106],
> (-8802193901675845653,-8790472027607832351], (-3900412470120874591,-3892641480459306647],
> (5455804264750818305,5465037357825542970], (4930767198829659527,4939587074207662799],
> (8086965740279021106,8087442741329154201], (-8933201045321260661,-8926445549049070674],
> (-4841328524165418854,-4838895482794593338], (628107265570603622,682509946926464280],
> (7043245467621414187,7055126022831789025], (624871765540463735,627374995781897409],
> (9219228482330263660,9221294940422311559], (-2335215188301493066,-2315034243278984017],
> (-6216599212198827632,-6211460136507414133], (-3276490559558850323,-3273110814046238767],
> (7204991007334459472,7214826985711309418], (1815809811279373566,1846961604192445001],
> (8743912118048160970,8751518028513315549], (-9204701745739426439,-9200185935622985719],
> (7926527126882050773,7941554683778488797], (-1307707180308444994,-1274682085495751899],
> (8354147540115782875,8358523989614737607], (-5418282332713406631,-5415077779309282099],
> (2436459402559272117,2441988676982099299], (3718536403335860823,3722612994041485455],
> (613072756805503750,624871765540463735], (3348338086790700840,3352211222405547568],
> (7941554683778488797,7949493423692701952], (-319434281654804937,-312029192727845573],
> (6560852500852746428,6577389024948220270], (3468872066705970665,3469611600245714638],
> (-4691666557374653357,-4691655196509961176], (1517867403379415135,1524487506534693145],
> (-7422802504145504981,-7402342710717945820], (1766173970230627147,1798177111825664424],
> (-4402244320098942276,-4399211415161403124], (-6024004998426219783,-6013031695922198269],
> (-6930256177103656090,-6926607739769548590], (-3578196517591043633,-3561058036097453011],
> (-7923695406605667073,-7885733750925254770], (-8063817427378208986,-8061632046535266293],
> (-7341733690813664665,-7287847431444491429], (-8327456680139422800,-8322944736494135341],
> (885662136143255295,898069608029365525], (-5555349213881020405,-5555145874347997903],
> (-4895525453201761515,-4892963646364151947], (4925572085497096406,4926057814710187377],
> (7785700473508314561,7813617525484553901], (-1598850926292630432,-1593204162764888475],
> (-2583030464997858536,-2581717714589673598], (7780476570449984013,7785700473508314561],
> (-3558045406807923433,-3550343462935144866], (7571721681971063633,7578715430118769516],
> (8865905084309486257,8879414598472867800], (-4984170160639031182,-4973560491294201055],
> (-5469249030580263062,-5432196900285376733], (-340636581410346966,-324449230085298889],
> (9120471287581648922,9127588925404102522], (8366987539912250171,8368772126617619299],
> (-9125573241533654210,-9123870950381500195], (1193377366975272040,1205407211929215537],
> (8879414598472867800,8900714457425949896], (7055126022831789025,7065799572034111661],
> (-1588707445511226686,-1539910209795101605], (6990922382177634221,7007948980474566617]]
> for XXX.[YYY, ZZZ]
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,778 RepairSession.java:224 -
> [repair #0791a4a0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync
> /FAKE.33.64, /FAKE.35.57, /FAKE.34.90 on range [(-3078252446668567055,-3069777768104388887],
> (3648099278098819362,3665843214181043507], (5797678959887947223,5820452778888079981],
> (-2642545928835445644,-2631494184435884123], (-1612062874902452641,-1610692114449831135],
> (-1707971415501754602,-1692411707392102715], (291977720343160936,302780457185309086],
> (-8155767539645304082,-8140296327332660530], (5553475010776119070,5565820507856584918],
> (3560286576304809625,3566372473455883512], (5226123540640728306,5228400303590993685],
> (3421729237821036215,3428685863718017921], (1227377289103207735,1240348980976161342],
> (3246531324407742399,3247845316596819400], (5304633131413824443,5306083993155890840],
> (-8619514398407319973,-8606439320645645548], (5911017071945481872,5912872298340550901],
> (-5637986723275926091,-5632042588860715397], (-7132684921710202809,-7132365563419969085],
> (8246342224027692193,8283424185055442516], (-4044458382788374703,-4036186919446747968],
> (-5613445605139257263,-5596969946550242654], (1083135773045625799,1084813234330856986],
> (-8140296327332660530,-8138240730730282332], (2996936679493130945,3002752531484603869],
> (-4490677154018906600,-4487029511866186581], (8456154651893349894,8463606226270547769],
> (-2656313914110802764,-2642545928835445644], (7193156006022031982,7197068598281426786],
> (8318481021259042556,8345872746994439720], (-1692411707392102715,-1673006435564531865],
> (1097191643473406518,1104630053604629743], (-3613177001569108681,-3599878627600515791],
> (-1662308889719802294,-1656482847672279831], (-6811080679276087557,-6804940650782637386],
> (7392275851743093948,7393304112887579707], (4240513765329086484,4257716503015670881],
> (-5036265422304136579,-5029426061972234042], (-4230577892781670592,-4228053239714848467],
> (-8159151100170546488,-8155767539645304082], (-5492523660879132140,-5478464594772026508],
> (3421125671176312710,3421729237821036215], (5262611881402902860,5275876653180871852],
> (1577722798587790899,1613215617842269344], (6666156845317387896,6676801654642938276],
> (5155814764198615180,5168412530555049843], (-6824229014637940118,-6819748162411093540],
> (-3666684985038092211,-3613177001569108681], (-8874623295866980549,-8872063750661463000],
> (-3599878627600515791,-3599733923716673512], (-5960723066215554398,-5949787783010678970],
> (258158907638948329,259127284829404565], (6661762711488840391,6666156845317387896],
> (-6367100347601878936,-6363911697345955296], (-6250542414915159381,-6229550501272299394],
> (-6917261891898915460,-6904807338337882298], (-7449919126757571330,-7438527202807402113],
> (-6904807338337882298,-6881332882762615208], (-7942161058117423218,-7938222136384107977],
> (781504901963705642,791302021433124535], (6527741860207715339,6537050054260012793],
> (7186200312153798494,7187207897161667549]] for XXX.[YYY, ZZZ]
>
> INFO  [Thread-2941] 2017-09-15 03:00:28,942 RepairSession.java:224 -
> [repair #07aaaae0-99c2-11e7-91dc-6132f5fe5fb0] new session: will sync
> /FAKE.33.64, /FAKE.35.153, /FAKE.34.90 on range [(3075097238958732453,3095612227388458535],
> (-3791204388712221856,-3785200705970537637], (8410714452405602873,8417924634749091961],
> (5114137550939144346,5121098642749542165], (2740032807677652551,2746972622905345611],
> (2197312222134575375,2220347677927119914], (4864988928009437565,4869276508135102594],
> (-3451050463660365008,-3450091362870187260], (7424782039380072835,7472858232992173041],
> (-5338516343760774634,-5325637469358550877], (2731359217194343571,2734544646630861913],
> (382200642783712616,390184326306122373], (430251086609083397,431821763463501369],
> (-5338895526845396808,-5338516343760774634], (-2677272836643942252,-2671813555202977248],
> (-2132202487362637132,-2128425620095084071], (2165546227633605752,2174360971273026818],
> (2763720958753148061,2764848409944217454], (3538741049060169597,3547917038270802802],
> (-6741608677619654195,-6731111079555719429], (2764848409944217454,2779298419000989854],
> (374883901669627753,382200642783712616], (-2706148319144852162,-2694312255222799319],
> (7132646356350938192,7136352706932176819], (194821142331236531,228318513249291325],
> (-2047674613927860008,-2032759017860426681], (-2032759017860426681,-2011077831196488705],
> (7472858232992173041,7477981949405760270], (431821763463501369,449911816694531926],
> (4347942912351042626,4349624658068191606], (311672118330363382,320380777495445406],
> (3099877433733294763,3112419968835610433], (-7182182373680571317,-7177421545891675761],
> (-6742074278675667310,-6741608677619654195], (-2770159943814078393,-2757633164368231752],
> (3095612227388458535,3099877433733294763], (-3741816028443553809,-3741630281022069399],
> (-738951315633674297,-727109613092919188], (4851195187346961355,4864988928009437565],
> (-769229207399684543,-744901822064683937], (-4143548408671617625,-4141029975015318194],
> (-7957562451432593113,-7952882742232241654], (-511680378144663402,-494227476671052554],
> (-832115128813031544,-819699210830328959], (6784361874104123161,6792435444931864417],
> (-8284973647739186524,-8256379601422884439], (-5321596150219749918,-5309942391653520721],
> (4723799060589319973,4734570102349898748], (5077208816402461873,5091797538025862046],
> (7487792874019513006,7489186250794409815], (-1647123563569159073,-1620470309759457440],
> (2779298419000989854,2784870878309140225], (320380777495445406,344020839234233467],
> (3112419968835610433,3119057615014897057], (-8506504972343997648,-8502433630545238869],
> (-2810980092222625726,-2810271001960433176], (344020839234233467,347093216302465566],
> (-8545779729891984163,-8519832061775919840], (2633165392330429557,2680549740474631393],
> (6391794517057168460,6431615673158447973], (6440766036098716417,6458083094779720657],
> (-8735147357564473077,-8730522574908969951], (7484355688408489651,7487792874019513006],
> (4325444310827485194,4347942912351042626], (2220347677927119914,2223568835084606157],
> (6802149418704156481,6819713717509862346], (4827439895247379322,4851195187346961355],
> (-4016529712384875309,-4009393333251988044], (2018021481024026660,2049270980733207626]]
> for XXX.[YYY, ZZZ]
>
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471
> ActiveRepairService.java:554 - Cannot start multiple repair sessions over
> the same sstables
>
> ERROR [ValidationExecutor:3] 2017-09-15 03:00:29,471 Validator.java:268 -
> Failed creating a merkle tree for [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0
> on XXX/YYY, [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699], (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822], (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497], (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205], (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483], (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189], (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492], (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319], (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883], (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]], /FAKE.33.64 (see log for
> details)
>
> INFO  [AntiEntropyStage:1] 2017-09-15 03:00:29,473 RepairSession.java:176
> - [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Received merkle tree for
> YYY from /FAKE.33.64
>
> ERROR [Repair#1:1] 2017-09-15 03:00:29,492 CassandraDaemon.java:207 -
> Exception in thread Thread[Repair#1:1,5,RMI Runtime]
>
> com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699], (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822], (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497], (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205], (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483], (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189], (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492], (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319], (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883], (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at com.google.common.util.concurrent.Futures.
> wrapAndThrowUnchecked(Futures.java:1525) ~[guava-18.0.jar:na]
>
>         at com.google.common.util.concurrent.Futures.
> getUnchecked(Futures.java:1511) ~[guava-18.0.jar:na]
>
>         at org.apache.cassandra.repair.RepairJob.run(RepairJob.java:160)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_102]
>
>         at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>
> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699], (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822], (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497], (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205], (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483], (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189], (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492], (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319], (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883], (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:164) ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.net
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=iTkw7x_8SK-JX4Z3qQ2Jy3_b-tabmhuz0oX8eYhAMOc&e=>
> .MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_102]
>
>         ... 4 common frames omitted
>
> WARN  [RepairJobTask:2] 2017-09-15 03:00:29,493 RepairJob.java:153 -
> [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] YYY sync failed
>
> ERROR [RepairJobTask:2] 2017-09-15 03:00:29,498 RepairSession.java:277 -
> [repair #071f81e0-99c2-11e7-91dc-6132f5fe5fb0] Session completed with the
> following error
>
> org.apache.cassandra.exceptions.RepairException: [repair
> #071f81e0-99c2-11e7-91dc-6132f5fe5fb0 on XXX/YYY, [(8195393703879512303,8196334842725538685],
> (8166975326273137878,8182604850967732931], (-7246799942440641887,-7227869626613009045],
> (-8371707510273823988,-8365977215604569699], (-141862581573028594,-140310864869418908],
> (3732113975108886193,3743105867152786342], (4998127507903069087,5008922734235607550],
> (-5115827291264930140,-5111054924035590372], (-2475342271852943287,-2447285553369030332],
> (-8318606053827235336,-8308721754886697230], (-5208900659917654871,-5202385837264015269],
> (6618737991399272130,6623100721269775102], (-4650650128572424858,-4650260492494258461],
> (1886545362164970333,1886646959491599822], (-4511817721998311568,-4507491187192881115],
> (8114903118676615937,8132992506844206601], (6224957219376301858,6304379125732293904],
> (-3460547504877234383,-3459262416082517136], (-167838948111369123,-141862581573028594],
> (481579232521229473,491242114841289497], (4052464144722307684,4059745901618136723],
> (1659668187498418295,1679582585970705122], (-1118922763210109192,-1093766915505652874],
> (7504365235878319341,7526188885210185292], (-79866884352549492,-77667207866300333],
> (8151204058820798561,8154760186218662205], (-1040398370287131739,-1033770179677543189],
> (3767057277953758442,3783780844370292025], (-6491678058233994892,-6487797181789288329],
> (-916868210769480248,-907141794196269524], (-9005441616028750657,-9002220258513351832],
> (8183526518331102304,8186908810225025483], (-5685737903527826627,-5672136154194382932],
> (4976122621177738811,4987871287137312689], (6051670147160447042,6051686987147911650],
> (-1161640137086921883,-1159172734746043158], (6895951547735922309,6899152466544114890],
> (-3357667382515377172,-3356304907368646189], (-5370953856683870319,-5345971445444542485],
> (3824272999898372667,3829315045986248983], (8132992506844206601,8149858096109302285],
> (3975126143101303723,3980729378827590597], (-956691623200349709,-946602525018301692],
> (-82499927325251331,-79866884352549492], (3952144214544622998,3955602392726495936],
> (8154760186218662205,8157079055586089583], (3840595196718778916,3866458971850198755],
> (-1066905024007783341,-1055954824488508260], (-7252356975874511782,-7246799942440641887],
> (-810612946397276081,-792189809286829222], (4964519403172053705,4970446606512414858],
> (-5380038118840759647,-5370953856683870319], (-3221630728515706463,-3206856875356976885],
> (-1193448110686154165,-1161640137086921883], (-3356304907368646189,-3346460884208327912],
> (3466596314109623830,3468144326691723333], (-9050241313548454460,-9005441616028750657],
> (402227699082311580,407458511300218383]]] Validation failed in /FAKE.33.64
>
>         at org.apache.cassandra.repair.ValidationTask.treesReceived(ValidationTask.java:68)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:178)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:486)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(
> RepairMessageVerbHandler.java:164) ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at org.apache.cassandra.net
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.net&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=iTkw7x_8SK-JX4Z3qQ2Jy3_b-tabmhuz0oX8eYhAMOc&e=>
> .MessageDeliveryTask.run(MessageDeliveryTask.java:67)
> ~[apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_102]
>
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_102]
>
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_102]
>
>         at org.apache.cassandra.concurrent.NamedThreadFactory.
> lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> [apache-cassandra-3.0.14.jar:3.0.14]
>
>         at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freist
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
> ädterstra
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
> ße 313
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
>
> --
>
> -----------------
>
> Alexander Dejanovski
>
> France
>
> @alexanderdeja
>
>
>
> Consultant
>
> Apache Cassandra Consulting
>
> http://www.thelastpickle.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=AccWDrJYIVBtQPHn2HooyxcIJ9NBFfdRrDqhssVn0ZE&e=>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
>
> --
>
> -----------------
>
> Alexander Dejanovski
>
> France
>
> @alexanderdeja
>
>
>
> Consultant
>
> Apache Cassandra Consulting
>
> http://www.thelastpickle.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=AccWDrJYIVBtQPHn2HooyxcIJ9NBFfdRrDqhssVn0ZE&e=>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__maps.google.com_-3Fq-3D4040-2BLinz-2C-2BAustria-2C-2BFreist-25C3-25A4dterstra-25C3-259Fe-2B313-26entry-3Dgmail-26source-3Dg&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=lLUGbGIBZQ5fnsEfOIpMF0WfcCqjrsx3QhW3OJyNlFc&s=sDB8Yn-ZhDQVMYXiSgfBIYLieqZ61kiGjY8sj4p03tA&e=>
>
>
>
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

Mime
View raw message