From commits-return-49504-archive-asf-public=cust-asf.ponee.io@subversion.apache.org Tue Oct 2 06:00:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9E692180660 for ; Tue, 2 Oct 2018 06:00:36 +0200 (CEST) Received: (qmail 92530 invoked by uid 500); 2 Oct 2018 04:00:35 -0000 Mailing-List: contact commits-help@subversion.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@subversion.apache.org Delivered-To: mailing list commits@subversion.apache.org Received: (qmail 92520 invoked by uid 99); 2 Oct 2018 04:00:35 -0000 Received: from Unknown (HELO svn01-us-west.apache.org) (209.188.14.144) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Oct 2018 04:00:35 +0000 Received: from svn01-us-west.apache.org (localhost [127.0.0.1]) by svn01-us-west.apache.org (ASF Mail Server at svn01-us-west.apache.org) with ESMTP id DC8913A00A1 for ; Tue, 2 Oct 2018 04:00:34 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1842585 - in /subversion/branches/1.10.x: ./ STATUS subversion/libsvn_fs_fs/recovery.c subversion/tests/cmdline/svnadmin_tests.py subversion/tests/cmdline/svntest/main.py subversion/tests/cmdline/svntest/wc.py Date: Tue, 02 Oct 2018 04:00:34 -0000 To: commits@subversion.apache.org From: svn-role@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20181002040034.DC8913A00A1@svn01-us-west.apache.org> Author: svn-role Date: Tue Oct 2 04:00:34 2018 New Revision: 1842585 URL: http://svn.apache.org/viewvc?rev=1842585&view=rev Log: Merge the r1838813 group from trunk: * r1838813, r1839703, r1839734, r1842262, r1842264 Let 'svnadmin recover' prune the rep-cache even if it is disabled. Justification: Can potentially lead to data loss. Votes: +1: julianfoad, rhuijben +1: danielsh (would prefer read_rep_cache() to check format numbers; see the "r1838813" thread on dev@) (without r1839703, r1839734) Modified: subversion/branches/1.10.x/ (props changed) subversion/branches/1.10.x/STATUS subversion/branches/1.10.x/subversion/libsvn_fs_fs/recovery.c subversion/branches/1.10.x/subversion/tests/cmdline/svnadmin_tests.py subversion/branches/1.10.x/subversion/tests/cmdline/svntest/main.py subversion/branches/1.10.x/subversion/tests/cmdline/svntest/wc.py Propchange: subversion/branches/1.10.x/ ------------------------------------------------------------------------------ --- svn:mergeinfo (original) +++ svn:mergeinfo Tue Oct 2 04:00:34 2018 @@ -102,4 +102,4 @@ /subversion/branches/verify-at-commit:1462039-1462408 /subversion/branches/verify-keep-going:1439280-1546110 /subversion/branches/wc-collate-path:1402685-1480384 -/subversion/trunk:1817837,1817856,1818577-1818578,1818584,1818651,1818662,1818727,1818801,1818803,1818807,1818868,1818871,1819036-1819037,1819043,1819049,1819052,1819093,1819146,1819162,1819444,1819556-1819557,1819603,1819804,1819911,1820044,1820046-1820047,1820518,1820627,1820718,1820778,1821183,1821224,1821621,1821678,1822401,1822587,1822591,1822996,1823202-1823203,1823211,1823327,1823791,1823966,1823989,1824033,1825024,1825045,1825215,1825266,1825306,1825709,1825711,1825721,1825736,1825778,1825783,1825787-1825788,1825979,1826720-1826721,1826747,1826811,1826814,1826877,1826907,1826971,1827105,1827114,1827191,1827562,1827574,1827670,1828613,1829012,1829015,1829241,1829260,1829344,1830083,1830882-1830883,1830885,1830900-1830901,1831110,1831112,1831540,1833465,1833621,1833836,1833842,1833864,1833866,1833895,1833897,1833899,1833901,1835760,1836306,1836762,1836802,1836963,1836976,1837037,1837790,1840991 +/subversion/trunk:1817837,1817856,1818577-1818578,1818584,1818651,1818662,1818727,1818801,1818803,1818807,1818868,1818871,1819036-1819037,1819043,1819049,1819052,1819093,1819146,1819162,1819444,1819556-1819557,1819603,1819804,1819911,1820044,1820046-1820047,1820518,1820627,1820718,1820778,1821183,1821224,1821621,1821678,1822401,1822587,1822591,1822996,1823202-1823203,1823211,1823327,1823791,1823966,1823989,1824033,1825024,1825045,1825215,1825266,1825306,1825709,1825711,1825721,1825736,1825778,1825783,1825787-1825788,1825979,1826720-1826721,1826747,1826811,1826814,1826877,1826907,1826971,1827105,1827114,1827191,1827562,1827574,1827670,1828613,1829012,1829015,1829241,1829260,1829344,1830083,1830882-1830883,1830885,1830900-1830901,1831110,1831112,1831540,1833465,1833621,1833836,1833842,1833864,1833866,1833895,1833897,1833899,1833901,1835760,1836306,1836762,1836802,1836963,1836976,1837037,1837790,1838813,1839703,1839734,1840991,1842262,1842264 Modified: subversion/branches/1.10.x/STATUS URL: http://svn.apache.org/viewvc/subversion/branches/1.10.x/STATUS?rev=1842585&r1=1842584&r2=1842585&view=diff ============================================================================== --- subversion/branches/1.10.x/STATUS (original) +++ subversion/branches/1.10.x/STATUS Tue Oct 2 04:00:34 2018 @@ -21,14 +21,6 @@ Veto-blocked changes: Approved changes: ================= - * r1838813, r1839703, r1839734, r1842262, r1842264 - Let 'svnadmin recover' prune the rep-cache even if it is disabled. - Justification: - Can potentially lead to data loss. - Votes: - +1: julianfoad, rhuijben - +1: danielsh (would prefer read_rep_cache() to check format numbers; see the "r1838813" thread on dev@) (without r1839703, r1839734) - * r1839662 For 'local missing' conflicts, scan for moves only if a YCA is known. Justification: Modified: subversion/branches/1.10.x/subversion/libsvn_fs_fs/recovery.c URL: http://svn.apache.org/viewvc/subversion/branches/1.10.x/subversion/libsvn_fs_fs/recovery.c?rev=1842585&r1=1842584&r2=1842585&view=diff ============================================================================== --- subversion/branches/1.10.x/subversion/libsvn_fs_fs/recovery.c (original) +++ subversion/branches/1.10.x/subversion/libsvn_fs_fs/recovery.c Tue Oct 2 04:00:34 2018 @@ -471,9 +471,15 @@ recover_body(void *baton, apr_pool_t *po } /* Prune younger-than-(newfound-youngest) revisions from the rep - cache if sharing is enabled taking care not to create the cache - if it does not exist. */ - if (ffd->rep_sharing_allowed) + cache, taking care not to create the cache if it does not exist. + + We do this whenever rep-cache.db exists, whether it's currently enabled + or not, to prevent a data loss that could result from having revisions + created after this 'recover' operation referring to rep-cache.db rows + that were created before the recover and that point to revisions younger- + than-(newfound-youngest). + */ + if (ffd->format >= SVN_FS_FS__MIN_REP_SHARING_FORMAT) { svn_boolean_t rep_cache_exists; Modified: subversion/branches/1.10.x/subversion/tests/cmdline/svnadmin_tests.py URL: http://svn.apache.org/viewvc/subversion/branches/1.10.x/subversion/tests/cmdline/svnadmin_tests.py?rev=1842585&r1=1842584&r2=1842585&view=diff ============================================================================== --- subversion/branches/1.10.x/subversion/tests/cmdline/svnadmin_tests.py (original) +++ subversion/branches/1.10.x/subversion/tests/cmdline/svnadmin_tests.py Tue Oct 2 04:00:34 2018 @@ -53,6 +53,24 @@ Wimp = svntest.testcase.Wimp_deco SkipDumpLoadCrossCheck = svntest.testcase.SkipDumpLoadCrossCheck_deco Item = svntest.wc.StateItem +def read_rep_cache(repo_dir): + """Return the rep-cache contents as a dict {hash: (rev, index, ...)}. + """ + db_path = os.path.join(repo_dir, 'db', 'rep-cache.db') + db1 = svntest.sqlite3.connect(db_path) + schema1 = db1.execute("pragma user_version").fetchone()[0] + # Can't test newer rep-cache schemas with an old built-in SQLite; see the + # documentation of STMT_CREATE_SCHEMA_V2 in ../../libsvn_fs_fs/rep-cache-db.sql + if schema1 >= 2 and svntest.sqlite3.sqlite_version_info < (3, 8, 2): + raise svntest.Failure("Can't read rep-cache schema %d using old " + "Python-SQLite version %s < (3,8,2)" % + (schema1, + svntest.sqlite3.sqlite_version_info)) + + content = { row[0]: row[1:] for row in + db1.execute("select * from rep_cache") } + return content + def check_hotcopy_bdb(src, dst): "Verify that the SRC BDB repository has been correctly copied to DST." ### TODO: This function should be extended to verify all hotcopied files, @@ -3823,6 +3841,63 @@ def load_issue4725(sbox): sbox2.build(create_wc=False, empty=True) load_and_verify_dumpstream(sbox2, None, [], None, False, dump, '-M100') +def check_recover_prunes_rep_cache(sbox, enable_rep_sharing): + """Check 'recover' prunes the rep-cache while enable-rep-sharing is + true/false. + """ + # Remember the initial rep cache content. + rep_cache_r1 = read_rep_cache(sbox.repo_dir) + #print '\n'.join([h + ": " + repr(ref) for h, ref in rep_cache_r1.items()]) + + # Commit one new rep and check the rep-cache is extended. + sbox.simple_append('iota', 'New line.\n') + sbox.simple_commit() + rep_cache_r2 = read_rep_cache(sbox.repo_dir) + if not (len(rep_cache_r2) == len(rep_cache_r1) + 1): + raise svntest.Failure + + fsfs_conf = svntest.main.get_fsfs_conf_file_path(sbox.repo_dir) + svntest.main.file_append(fsfs_conf, + # Add a newline in case the existing file doesn't + # end with one. + "\n" + "[rep-sharing]\n" + "enable-rep-sharing = %s\n" + % (('true' if enable_rep_sharing else 'false'),)) + + # Break r2 in such a way that 'recover' will discard it + head_rev_path = fsfs_file(sbox.repo_dir, 'revs', '2') + os.remove(head_rev_path) + current_path = os.path.join(sbox.repo_dir, 'db', 'current') + svntest.main.file_write(current_path, '1\n') + + # Recover back to r1. + svntest.actions.run_and_verify_svnadmin(None, [], + "recover", sbox.repo_dir) + svntest.actions.run_and_verify_svnlook(['1\n'], [], 'youngest', + sbox.repo_dir) + + # Check the rep-cache is pruned. + rep_cache_recovered = read_rep_cache(sbox.repo_dir) + if not (rep_cache_recovered == rep_cache_r1): + raise svntest.Failure + +@Issue(4077) +@SkipUnless(svntest.main.is_fs_type_fsfs_and_sqlite_can_read_our_db) +def recover_prunes_rep_cache_when_enabled(sbox): + "recover prunes rep cache when enabled" + sbox.build() + + check_recover_prunes_rep_cache(sbox, enable_rep_sharing=True) + +@Issue(4077) +@SkipUnless(svntest.main.is_fs_type_fsfs_and_sqlite_can_read_our_db) +def recover_prunes_rep_cache_when_disabled(sbox): + "recover prunes rep cache when disabled" + sbox.build() + + check_recover_prunes_rep_cache(sbox, enable_rep_sharing=False) + ######################################################################## # Run the tests @@ -3897,6 +3972,8 @@ test_list = [ None, dump_exclude_all_rev_changes, dump_invalid_filtering_option, load_issue4725, + recover_prunes_rep_cache_when_enabled, + recover_prunes_rep_cache_when_disabled, ] if __name__ == '__main__': Modified: subversion/branches/1.10.x/subversion/tests/cmdline/svntest/main.py URL: http://svn.apache.org/viewvc/subversion/branches/1.10.x/subversion/tests/cmdline/svntest/main.py?rev=1842585&r1=1842584&r2=1842585&view=diff ============================================================================== --- subversion/branches/1.10.x/subversion/tests/cmdline/svntest/main.py (original) +++ subversion/branches/1.10.x/subversion/tests/cmdline/svntest/main.py Tue Oct 2 04:00:34 2018 @@ -1633,6 +1633,14 @@ def server_has_atomic_revprop(): def server_has_reverse_get_file_revs(): return options.server_caps.has_reverse_get_file_revs +def python_sqlite_can_read_our_db(): + """Check if the Python builtin is capable enough to peek into wc.db""" + # Currently enough (1.7-1.9) + return svntest.sqlite3.sqlite_version_info >= (3, 6, 18) + +def is_fs_type_fsfs_and_sqlite_can_read_our_db(): + return is_fs_type_fsfs() and python_sqlite_can_read_our_db() + def is_plaintext_password_storage_disabled(): try: predicate = re.compile("^WARNING: Plaintext password storage is enabled!") Modified: subversion/branches/1.10.x/subversion/tests/cmdline/svntest/wc.py URL: http://svn.apache.org/viewvc/subversion/branches/1.10.x/subversion/tests/cmdline/svntest/wc.py?rev=1842585&r1=1842584&r2=1842585&view=diff ============================================================================== --- subversion/branches/1.10.x/subversion/tests/cmdline/svntest/wc.py (original) +++ subversion/branches/1.10.x/subversion/tests/cmdline/svntest/wc.py Tue Oct 2 04:00:34 2018 @@ -1092,8 +1092,7 @@ def svn_uri_quote(url): def python_sqlite_can_read_wc(): """Check if the Python builtin is capable enough to peek into wc.db""" - # Currently enough (1.7-1.9) - return svntest.sqlite3.sqlite_version_info >= (3, 6, 18) + return svntest.main.python_sqlite_can_read_our_db() def open_wc_db(local_path): """Open the SQLite DB for the WC path LOCAL_PATH.