Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 237E79C3F for ; Sun, 20 May 2012 22:44:31 +0000 (UTC) Received: (qmail 81727 invoked by uid 500); 20 May 2012 22:44:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81706 invoked by uid 500); 20 May 2012 22:44:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81697 invoked by uid 99); 20 May 2012 22:44:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 22:44:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_IMAGE_ONLY_20,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of tyler@datastax.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-lpp01m010-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 May 2012 22:44:22 +0000 Received: by lagv3 with SMTP id v3so3534092lag.31 for ; Sun, 20 May 2012 15:44:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=Km0GBE5FJxB4hejtRQ4YP7zP996vyIjT8yBtnaPFkOY=; b=PKkmW6BqESXhtouWgg2IkfLepY5KFdkQpsDvcGPvBox15joNa4J6ejDLAeyLtEoR5T sYPFfZWi+HV0dqDO56erCxpVsEOgzlFZvR9SJYBZAFyilZGrst1iy2IH2nfaZvCn0pn7 ftJiQRUFhW1ke3IpboHi08w3XAb8NmAIJL9TybzoZFSkPpAJjsrJb38mFCtuoSBzTKu+ Nh8bFrOF3vzEJ5EicHuPwbsMMOAk4WmCfVZn8cNZ+bsU8oL659M1ZxPrMadBVhM1nMr3 nPv7C6h5k7DW+VFMhtW8qA0Sv6Eg+C1lXLw3Ti3ORmNZ2rX1EzGIZv0g2DFUdaO6O31S 2GFg== MIME-Version: 1.0 Received: by 10.152.135.105 with SMTP id pr9mr9898796lab.37.1337553841734; Sun, 20 May 2012 15:44:01 -0700 (PDT) Received: by 10.112.59.132 with HTTP; Sun, 20 May 2012 15:44:01 -0700 (PDT) In-Reply-To: References: Date: Sun, 20 May 2012 17:44:01 -0500 Message-ID: Subject: Re: restoring from snapshot - missing data From: Tyler Hobbs To: user@cassandra.apache.org Content-Type: multipart/related; boundary=f46d044271569ac24a04c07f8517 X-Gm-Message-State: ALoCoQklpJvMaliuocTcglfnpNefGPXia5gopW7NShQM0DS0yZcC8YjdQK0QDqmEjSAPusnguhE/ --f46d044271569ac24a04c07f8517 Content-Type: multipart/alternative; boundary=f46d044271569ac24804c07f8516 --f46d044271569ac24804c07f8516 Content-Type: text/plain; charset=ISO-8859-1 Did you use the same tokens for the nodes in both clusters? On Sun, May 20, 2012 at 1:25 PM, Tamar Fraenkel wrote: > Hi! > I am testing backup and restore. > I created the restore using parallel ssh on all 3 nodes. > I created a new 3 ring setup and used the snapshot to test recover. > Snapshot from every original node went to one of the new nodes. > When I compare the content of the data dir it seems that all files from > the original cluster exist on the backup cluster. > *But* when I do some cqlsh queries it seems as though about 1/3 of my > data is missing. > > Any idea what could be the issue? > I thought that snapshot flushes all in-memory writes to disk, so it can't > be that some data was not on the original snapshot. > > Help is much appreciated, > Thanks > > *Tamar Fraenkel * > Senior Software Engineer, TOK Media > > [image: Inline image 1] > > tamar@tok-media.com > Tel: +972 2 6409736 > Mob: +972 54 8356490 > Fax: +972 2 5612956 > > > > -- Tyler Hobbs DataStax --f46d044271569ac24804c07f8516 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Did you use the same tokens for the nodes in both clusters?

On Sun, May 20, 2012 at 1:25 PM, Tamar Fraenkel <tama= r@tok-media.com> wrote:
Hi!
I am testing backup= and restore.
I created the restore using parallel ssh on all 3 n= odes.
I created a new 3 ring setup and used the snapshot to test recover.=A0=
Snapshot from every original node went to one of the new nodes.<= /div>
When I compare the content of the data dir it seems that = all files from the original cluster exist on the backup cluster.=A0
But when I do some cqlsh=A0queries it seems as tho= ugh about 1/3 of my data is missing.

Any idea what= could be the issue?
I thought that snapshot=A0flushes all in-memory writes to = disk, so it can't be that some data was not on the original snapshot.

Help is much appreciated,
Thanks
Tamar Fraenkel=A0
= Senior Software Engineer, TOK Media=A0

3D"Inline

tamar@tok-media.co= m
Tel:=A0=A0=A0+972 2 6409736=A0
Mo= b:=A0=A0+972 54 8356490=A0
Fax:=A0=A0=A0<= a value=3D"+97225612956">+972 2 5612956=A0






--
Tyler Hobbs
DataStax
<= br> --f46d044271569ac24804c07f8516-- --f46d044271569ac24a04c07f8517 Content-Type: image/png; name="tokLogo.png" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: ii_135b91fb888fa9ff iVBORw0KGgoAAAANSUhEUgAAAF0AAAAnCAYAAABtyERkAAAABGdBTUEAALGPC/xhBQAACkNpQ0NQ SUNDIFByb2ZpbGUAAHgBnZZ3VFNZE8Dvey+90BJCkRJ6DU1KAJESepFeRSUkAUIJGBKwV0QFVxQV aYoiiyIuuLoUWSuiWFgUFLAvyCKgrIuriIplX/QcZf/Y/b6z88ec35s7c+/cmbnnPAAovoFCUSas AECGSCIO8/FgxsTGMfHdAAZEgAPWAHB52VlB4d4RABU/Lw4zG3WSsUygz/p1/xe4xfINYTI/m/5/ pcjLEkvQnULQkLl8QTYP5TyU03MlWTL7JMr0xDQZwxgZi9EEUVaVcfIXNv/s84XdZMzPEPFRH1nO WfwMvow7UN6SIxWgjASinJ8jFOSifBtl/XRphhDlNyjTMwTcbAAwFJldIuCloGyFMkUcEcZBeR4A BEryLE6cxRLBMjRPADiZWcvFwuQUCdOYZ8K0dnRkM30FuekCiYQVwuWlccV8JiczI4srWg7AlzvL ooCSrLZMtMj21o729iwbC7T8X+VfF796/TvIevvF42Xo555BjK5vtm+x32yZ1QCwp9Da7PhmSywD oGUTAKr3vtn0DwAgnwdA841Z92HI5iVFIslysrTMzc21EAp4FrKCfpX/6fDV859h1nkWsvO+1o7p KUjiStMlTFlReZnpmVIxMzuLyxMwWX8bYnTr/xw4K61ZeZiHCZIEYoEIPSoKnTKhKBltt4gvlAgz RUyh6J86/B/DZuUgwy9zjQKt5iOgL7EACjfoAPm9C2BoZIDE70dXoK99CyRGAdnLi9Ye/TL3KKPr n/XfFFyEfsLZwmSmzMwJi2DypOIcGaNvQqawgATkAR2oAS2gB4wBC9gAB+AM3IAX8AfBIALEgsWA B1JABhCDXLAKrAf5oBDsAHtAOagCNaAONIAToAWcBhfAZXAd3AR94D4YBCPgGZgEr8EMBEF4iArR IDVIGzKAzCAbiA3Nh7ygQCgMioUSoGRIBEmhVdBGqBAqhsqhg1Ad9CN0CroAXYV6oLvQEDQO/Qm9 gxGYAtNhTdgQtoTZsDscAEfAi+BkeCm8As6Dt8OlcDV8DG6GL8DX4T54EH4GTyEAISMMRAdhIWyE gwQjcUgSIkbWIAVICVKNNCBtSCdyCxlEJpC3GByGhmFiWBhnjC8mEsPDLMWswWzDlGOOYJoxHZhb mCHMJOYjlorVwJphnbB+2BhsMjYXm48twdZim7CXsH3YEexrHA7HwBnhHHC+uFhcKm4lbhtuH64R dx7XgxvGTeHxeDW8Gd4FH4zn4iX4fHwZ/hj+HL4XP4J/QyATtAk2BG9CHEFE2EAoIRwlnCX0EkYJ M0QFogHRiRhM5BOXE4uINcQ24g3iCHGGpEgyIrmQIkippPWkUlID6RLpAeklmUzWJTuSQ8lC8jpy Kfk4+Qp5iPyWokQxpXAo8RQpZTvlMOU85S7lJZVKNaS6UeOoEup2ah31IvUR9Y0cTc5Czk+OL7dW rkKuWa5X7rk8Ud5A3l1+sfwK+RL5k/I35CcUiAqGChwFrsIahQqFUwoDClOKNEVrxWDFDMVtikcV ryqOKeGVDJW8lPhKeUqHlC4qDdMQmh6NQ+PRNtJqaJdoI3Qc3YjuR0+lF9J/oHfTJ5WVlG2Vo5SX KVcon1EeZCAMQ4YfI51RxDjB6Ge8U9FUcVcRqGxVaVDpVZlWnaPqpipQLVBtVO1TfafGVPNSS1Pb qdai9lAdo26qHqqeq75f/ZL6xBz6HOc5vDkFc07MuacBa5hqhGms1Dik0aUxpaml6aOZpVmmeVFz Qouh5aaVqrVb66zWuDZNe762UHu39jntp0xlpjsznVnK7GBO6mjo+OpIdQ7qdOvM6BrpRupu0G3U fahH0mPrJent1mvXm9TX1g/SX6Vfr3/PgGjANkgx2GvQaTBtaGQYbbjZsMVwzEjVyM9ohVG90QNj qrGr8VLjauPbJjgTtkmayT6Tm6awqZ1pimmF6Q0z2MzeTGi2z6zHHGvuaC4yrzYfYFFY7qwcVj1r yIJhEWixwaLF4rmlvmWc5U7LTsuPVnZW6VY1Vvetlaz9rTdYt1n/aWNqw7OpsLk9lzrXe+7aua1z X9ia2Qps99vesaPZBdlttmu3+2DvYC+2b7Afd9B3SHCodBhg09kh7G3sK45YRw/HtY6nHd862TtJ nE44/eHMck5zPuo8Ns9onmBezbxhF10XrstBl8H5zPkJ8w/MH3TVceW6Vrs+dtNz47vVuo26m7in uh9zf+5h5SH2aPKY5jhxVnPOeyKePp4Fnt1eSl6RXuVej7x1vZO9670nfex8Vvqc98X6Bvju9B3w 0/Tj+dX5Tfo7+K/27wigBIQHlAc8DjQNFAe2BcFB/kG7gh4sMFggWtASDIL9gncFPwwxClka8nMo LjQktCL0SZh12KqwznBa+JLwo+GvIzwiiiLuRxpHSiPbo+Sj4qPqoqajPaOLowdjLGNWx1yPVY8V xrbG4eOi4mrjphZ6LdyzcCTeLj4/vn+R0aJli64uVl+cvvjMEvkl3CUnE7AJ0QlHE95zg7nV3KlE v8TKxEkeh7eX94zvxt/NHxe4CIoFo0kuScVJY8kuybuSx1NcU0pSJoQcYbnwRapvalXqdFpw2uG0 T+nR6Y0ZhIyEjFMiJVGaqCNTK3NZZk+WWVZ+1uBSp6V7lk6KA8S12VD2ouxWCR39meqSGks3SYdy 5udU5LzJjco9uUxxmWhZ13LT5VuXj67wXvH9SsxK3sr2VTqr1q8aWu2++uAaaE3imva1emvz1o6s 81l3ZD1pfdr6XzZYbSje8Gpj9Ma2PM28dXnDm3w21efL5YvzBzY7b67agtki3NK9de7Wsq0fC/gF 1wqtCksK32/jbbv2nfV3pd992p60vbvIvmj/DtwO0Y7+na47jxQrFq8oHt4VtKt5N3N3we5Xe5bs uVpiW1K1l7RXunewNLC0tUy/bEfZ+/KU8r4Kj4rGSo3KrZXT+/j7eve77W+o0qwqrHp3QHjgzkGf g83VhtUlh3CHcg49qYmq6fye/X1drXptYe2Hw6LDg0fCjnTUOdTVHdU4WlQP10vrx4/FH7v5g+cP rQ2shoONjMbC4+C49PjTHxN+7D8RcKL9JPtkw08GP1U20ZoKmqHm5c2TLSktg62xrT2n/E+1tzm3 Nf1s8fPh0zqnK84onyk6Szqbd/bTuRXnps5nnZ+4kHxhuH1J+/2LMRdvd4R2dF8KuHTlsvfli53u neeuuFw5fdXp6qlr7Gst1+2vN3fZdTX9YvdLU7d9d/MNhxutNx1vtvXM6znb69p74Zbnrcu3/W5f 71vQ19Mf2X9nIH5g8A7/ztjd9Lsv7uXcm7m/7gH2QcFDhYcljzQeVf9q8mvjoP3gmSHPoa7H4Y/v D/OGn/2W/dv7kbwn1Cclo9qjdWM2Y6fHvcdvPl34dORZ1rOZifzfFX+vfG78/Kc/3P7omoyZHHkh fvHpz20v1V4efmX7qn0qZOrR64zXM9MFb9TeHHnLftv5Lvrd6Ezue/z70g8mH9o+Bnx88Cnj06e/ AAOb8/zszueKAAAACXBIWXMAAA7DAAAOwwHHb6hkAAAHmklEQVRoBeVaa0wUVxQ+uzPLLqj1lbRW IkKKYgNq0WpRq0mjEWJRqVaLGmoCFdpCpYhGLP6hlvpCWl+NmGATFYH+qPhqq9W0wSqgLYhPFCxI Q2L7oyjCPthX7xkY2MedYRaGWbAnmcy959x77plvzp69c+5R2QnBICLLn7UAqg6D2aAJg8jyblPZ 7ubAatma/wVzVSVYblaB7U4l2OquAfgQGxlyodV4qQn+rAqYKZGgDp4M7MSpwASFgHr4KCIcuKRC T3/8+G8oLi7uVyvnzZsL4eHhomvYDQYw37gBphMFYC09D6AhP0ICrkpDprGkLQB614vAF0Iudnos aGa9DZqps0TXcxWePn0G6uvrXdkQFBQES5YsduPTGFKw5DzdaDRAeXkFTYdsvNDQUFFdpp8vgiF7 O9ibGjtARqB7SZbKIsDL5D8DdKsyuF+AFFUIeF9wMBqN8NVXX0NjY6PgckuXLsYfqHfJ2tQELR+l gD7+Y7A/+ktWY2xN10Gfsxz0x3aD3WSQVberMgR8z57cHgGPjY31Lujm6mp4tup9sJ694PoMsvYt l/OgbUc8WP9pklUvr4wH/PbtOzzL7Y4ejoAjceHFbYQCjParZdD23lpJK7FRq4Cd9jq3a2FCOsMU 2cFYa++A3dAGtkf3wPJ7oaguzut3xoHf5mPAvOgvOtYToaeAo26vgI4hRZ+xVfTZ2HdiQbt8BWgm TxEcxwYGd8nspkyw1NwAc9k5Es/pL8D+rBH0BPihX54Dlda3a25fGgUFJ0Cqh/PrcKAHBgZCYWEB z6PeGxoaYMuWTKps+/ZsQB1SCHcorWkbwV5Pj99MVBT4paYCG9wNqBS9Kq2O7FYiuMvyYAUYC3eA rYlsM10IgTd8mw1+H37hIvG8W1RUBBcvXhKc6BhSHAcp/kdqOn8BbGV/ONrQ1dZuTocX9u/3GPAu BZ0NduIUGJJxBNi5Sa4iro87G3N1GVUmlYmAnzp1RnC4EOA4QVHQ0cuNufuohmoz04n30UGiTuiB iZ7vF7cJNJGbqCNNJXQ7qINdmH0BHFUpCnr7b1eoYUXzSaKsgDtipIteC0xIlCOLa+Mfq+VBtRu/ J8aVK1d77eG8bmVB/+VXfl2nu9+6BKe+nB30eN2KT6kqzbc8CzFVVVVw4MBBqi5kioUUx0mKgm45 9p3j2lwbvVw9cqQbX04GExAMav+ZbiptDbfceEIMBHzXrhwhsWTAUQG3exHUJKMAt4k00kwTz8fQ 5vSGx86MhvaTzrsZ632S35FANTU1oukBqR7OL6WYp9v1en5NpzsTON6p318dle/QXqt+8uSp6Ny2 NvqzCU1SDHQhA54HPu7V8Q9WKikGunr0aKpN1oZHVL7cTLuhVW6VTvqOHz8OmNaVQsqBPop+sGCu rJJiZ5/HWK6dddPBhES68XrLwBCUn58PmIvpiRQDHQ3RpKxzs8e8/zDYmpvd+HIyzDfLqSkB5tXZ Hi8zYsRwwTmYgykpKRGU8wJFQfeZ+ya/rtO9LSfXqS9nx24ygukk/etTE+a+jRRbOywslBxS5ALe hQhTA7jbESNFQdeEvwaqoHFu9liOFoP+UJ4bv68MBFx/YCPVy9XkVIkJmCB5CQQ6PX0D6HQ6SEgQ /5jbu3cfiO14FAVd5esLug3rqQ9qyt4DrbtzwC4hJlIVuDB5wK33f3KRdHTxGE8qBQQEdAGOc8aM eQlSUpIFpyPgBw8Kf7kqCjpaqYtZCkz0QqrB5m8OQ0t8PFjq6qhyqcz2ikvQmhkNQoBj9hErB6TS 2LEvcx7uOH7OnNkQEfGGI8upjfEdD7pppDjoaMTQ7M+pYQZltuskEXX3LjY9ItvTZmgvJ2BnrQZj fhLYWxqo83HH4rsyhSrzlJmUlAhif6yFhUWA5xCupFgawHFhNdk+Dis8yp2PCh1mOI7Htu1JM5jL S7kjO67YiBzXIVkbH4DtIamNwQMLhjDw6pSh3JFUwwJAF/eZbKdGGN9TU9dDVtY2x2Wc2jt37uL+ fHEsT14BHRdn/P054Fsi3uJtod4tD+ug/YeTYP7+UM91L1QNHUzOwz/YJnsh0qRJk7hkl9CBBsb3 vLzD3MvhzfMa6GgAAi9E7RXlYDy0D2z3rnUWGwmN7JnvE5MF2vnLZPNw1xVjYmKgqqpasPwCa2lC Qy/BggXzualeiemuRrv2jRnpoE+MA9st56yg67ie+uy0WBiy9UfQLVrTb4CjDRg60tLoOXvexvz8 I11pggEJOm9ob++ayI0c2Hj47MlevLfr4TzcRiYkxIuqwOovTBN4NbyIWihRqB4/gxxQvAJMGPnY 8Q8ENkT6VlDiEpKHYfioqKgQLMnAcjss2Rg0oKtDZoA2IZXUwZCvWp08NSuS0fRgYHJyMilV2SL4 RYppYMmg68iDCn0MoKy/iF1Eio6WkaKjsP734CBSnUsjIT5tLO7bExPXQWnpZZqY43Gl0oJSBQTN 4yZ2r4IuQEqisTTaJy0ddO+uJOen9JRw96TB15Ls6Uo8mnrmdNCuJp69cCFgnuZ5pQEBOuZidHFr wGe2Z0X8g/WleD28WGprgZ0gPcU6WIF2tNvroDsa839pP5cfRwP95f0HN8yA0D1xOrwAAAAASUVO RK5CYII= --f46d044271569ac24a04c07f8517--