subversion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From julianf...@apache.org
Subject svn commit: r1657307 - in /subversion/trunk: ./ build/ notes/ subversion/libsvn_fs_x/ subversion/tests/cmdline/ subversion/tests/cmdline/svntest/ subversion/tests/libsvn_fs_x/
Date Wed, 04 Feb 2015 16:32:47 GMT
Author: julianfoad
Date: Wed Feb  4 16:32:47 2015
New Revision: 1657307

URL: http://svn.apache.org/r1657307
Log:
Reintegrate the 'dump-load-cross-check' branch: implement optional dump/load
cross-checking in the test suite.

This is activated by passing 'DUMP_LOAD_CROSS_CHECK=1' to 'make check' and
friends, or by passing '--dump-load-cross-check' to the Python tests.

Remaining items mentioned in the BRANCH-README file were:

TODO:
  - Fix FAIL: svnmucc_tests.py 2: a no-op content-change is reported
    differently in different cases. (Fix it how? I'd prefer to make the
    output of a no-op change the same as a no-change.)

Ideas for improvement:
  - Improve the logic for finding repositories created by a test: detect
    when a test created a repository even if the sandbox is not marked as
    'built'; detect when a test created additional repositories.

  - Implement the same cross-checking for the C tests.

Modified:
    subversion/trunk/   (props changed)
    subversion/trunk/Makefile.in
    subversion/trunk/build/run_tests.py
    subversion/trunk/notes/   (props changed)
    subversion/trunk/subversion/libsvn_fs_x/   (props changed)
    subversion/trunk/subversion/tests/cmdline/authz_tests.py
    subversion/trunk/subversion/tests/cmdline/davautocheck.sh
    subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
    subversion/trunk/subversion/tests/cmdline/svnsync_authz_tests.py
    subversion/trunk/subversion/tests/cmdline/svntest/main.py
    subversion/trunk/subversion/tests/cmdline/svntest/sandbox.py
    subversion/trunk/subversion/tests/cmdline/svntest/testcase.py
    subversion/trunk/subversion/tests/cmdline/svntest/verify.py
    subversion/trunk/subversion/tests/libsvn_fs_x/   (props changed)

Propchange: subversion/trunk/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Feb  4 16:32:47 2015
@@ -11,6 +11,7 @@
 /subversion/branches/diff-optimizations-bytes:1037353-1067789
 /subversion/branches/dont-save-plaintext-passwords-by-default:870728-871118
 /subversion/branches/double-delete:870511-872970
+/subversion/branches/dump-load-cross-check:1654853-1657295
 /subversion/branches/ev2-export:1325914,1332738,1413107
 /subversion/branches/explore-wc:875486,875493,875497,875507,875511,875514,875559,875580-875581,875584,875587,875611,875627,875647,875667-875668,875711-875712,875733-875734,875736,875744-875748,875751,875758,875782,875795-875796,875830,875836,875838,875842,875852,875855,875864,875870,875873,875880,875885-875888,875890,875897-875898,875905,875907-875909,875935,875943-875944,875946,875979,875982-875983,875985-875986,875990,875997
 /subversion/branches/file-externals:871779-873302

Modified: subversion/trunk/Makefile.in
URL: http://svn.apache.org/viewvc/subversion/trunk/Makefile.in?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/Makefile.in (original)
+++ subversion/trunk/Makefile.in Wed Feb  4 16:32:47 2015
@@ -529,6 +529,9 @@ check: bin @TRANSFORM_LIBTOOL_SCRIPTS@ $
 	  if test "$(SKIP_C_TESTS)" != ""; then                              \
 	    flags="--skip-c-tests $$flags";                                  \
 	  fi;                                                                \
+	  if test "$(DUMP_LOAD_CROSS_CHECK)" != ""; then                     \
+	    flags="--dump-load-cross-check $$flags";                         \
+	  fi;                                                                \
 	  if test "$(FS_TYPE)" != ""; then                                   \
 	    flags="--fs-type $(FS_TYPE) $$flags";                            \
 	  fi;                                                                \

Modified: subversion/trunk/build/run_tests.py
URL: http://svn.apache.org/viewvc/subversion/trunk/build/run_tests.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/build/run_tests.py (original)
+++ subversion/trunk/build/run_tests.py Wed Feb  4 16:32:47 2015
@@ -130,7 +130,8 @@ class TestHarness:
                http_proxy=None, http_proxy_username=None,
                http_proxy_password=None, httpd_version=None,
                exclusive_wc_locks=None,
-               memcached_server=None, skip_c_tests=None):
+               memcached_server=None, skip_c_tests=None,
+               dump_load_cross_check=None):
     '''Construct a TestHarness instance.
 
     ABS_SRCDIR and ABS_BUILDDIR are the source and build directories.
@@ -193,6 +194,7 @@ class TestHarness:
     if not sys.stdout.isatty() or sys.platform == 'win32':
       TextColors.disable()
     self.skip_c_tests = (not not skip_c_tests)
+    self.dump_load_cross_check = (not not dump_load_cross_check)
 
     # Parse out the FSFS version number
     if self.fs_type is not None and self.fs_type.startswith('fsfs-v'):
@@ -549,6 +551,8 @@ class TestHarness:
       svntest.main.options.exclusive_wc_locks = self.exclusive_wc_locks
     if self.memcached_server is not None:
       svntest.main.options.memcached_server = self.memcached_server
+    if self.dump_load_cross_check is not None:
+      svntest.main.options.dump_load_cross_check = self.dump_load_cross_check
 
     svntest.main.options.srcdir = self.srcdir
 
@@ -712,6 +716,7 @@ def main():
     opts, args = my_getopt(sys.argv[1:], 'u:f:vc',
                            ['url=', 'fs-type=', 'verbose', 'cleanup',
                             'skip-c-tests', 'skip-C-tests',
+                            'dump-load-cross-check',
                             'http-library=', 'server-minor-version=',
                             'fsfs-packing', 'fsfs-sharding=',
                             'enable-sasl', 'parallel=', 'config-file=',
@@ -733,10 +738,10 @@ def main():
     parallel, config_file, log_to_stdout, list_tests, mode_filter, \
     milestone_filter, set_log_level, ssl_cert, http_proxy, \
     http_proxy_username, http_proxy_password, httpd_version, \
-    exclusive_wc_locks, memcached_server = \
+    exclusive_wc_locks, memcached_server, dump_load_cross_check = \
             None, None, None, None, None, None, None, None, None, None, \
             None, None, None, None, None, None, None, None, None, None, \
-            None, None, None, None,
+            None, None, None, None, None
   for opt, val in opts:
     if opt in ['-u', '--url']:
       base_url = val
@@ -756,6 +761,8 @@ def main():
       cleanup = 1
     elif opt in ['--skip-c-tests', '--skip-C-tests']:
       skip_c_tests = 1
+    elif opt in ['--dump-load-cross-check']:
+      dump_load_cross_check = 1
     elif opt in ['--enable-sasl']:
       enable_sasl = 1
     elif opt in ['--parallel']:
@@ -808,7 +815,8 @@ def main():
                    httpd_version=httpd_version,
                    exclusive_wc_locks=exclusive_wc_locks,
                    memcached_server=memcached_server,
-                   skip_c_tests=skip_c_tests)
+                   skip_c_tests=skip_c_tests,
+                   dump_load_cross_check=dump_load_cross_check)
 
   failed = th.run(args[2:])
   if failed:

Propchange: subversion/trunk/notes/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Feb  4 16:32:47 2015
@@ -3,6 +3,7 @@
 /subversion/branches/diff-callbacks3/notes:870059-870761
 /subversion/branches/dont-save-plaintext-passwords-by-default/notes:870728-871118
 /subversion/branches/double-delete/notes:870511-872970
+/subversion/branches/dump-load-cross-check/notes:1654853-1657295
 /subversion/branches/explore-wc/notes:875486,875493,875497,875507,875511,875514,875559,875580-875581,875584,875587,875611,875627,875647,875667-875668,875711-875712,875733-875734,875736,875744-875748,875751,875758,875782,875795-875796,875830,875836,875838,875842,875852,875855,875864,875870,875873,875880,875885-875888,875890,875897-875898,875905,875907-875909,875935,875943-875944,875946,875979,875982-875983,875985-875986,875990,875997
 /subversion/branches/file-externals/notes:871779-873302
 /subversion/branches/fs-rep-sharing/notes:869036-873803

Propchange: subversion/trunk/subversion/libsvn_fs_x/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Feb  4 16:32:47 2015
@@ -10,6 +10,7 @@
 /subversion/branches/diff-optimizations-bytes/subversion/libsvn_fs_x:1037353-1067789
 /subversion/branches/dont-save-plaintext-passwords-by-default/subversion/libsvn_fs_x:870728-871118
 /subversion/branches/double-delete/subversion/libsvn_fs_x:870511-872970
+/subversion/branches/dump-load-cross-check/subversion/libsvn_fs_x:1654853-1657295
 /subversion/branches/ev2-export/subversion/libsvn_fs_x:1325914,1332738,1413107
 /subversion/branches/explore-wc/subversion/libsvn_fs_x:875486,875493,875497,875507,875511,875514,875559,875580-875581,875584,875587,875611,875627,875647,875667-875668,875711-875712,875733-875734,875736,875744-875748,875751,875758,875782,875795-875796,875830,875836,875838,875842,875852,875855,875864,875870,875873,875880,875885-875888,875890,875897-875898,875905,875907-875909,875935,875943-875944,875946,875979,875982-875983,875985-875986,875990,875997
 /subversion/branches/file-externals/subversion/libsvn_fs_x:871779-873302

Modified: subversion/trunk/subversion/tests/cmdline/authz_tests.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/authz_tests.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/authz_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/authz_tests.py Wed Feb  4 16:32:47 2015
@@ -45,6 +45,7 @@ XFail = svntest.testcase.XFail_deco
 Issues = svntest.testcase.Issues_deco
 Issue = svntest.testcase.Issue_deco
 Wimp = svntest.testcase.Wimp_deco
+SkipDumpLoadCrossCheck = svntest.testcase.SkipDumpLoadCrossCheck_deco
 
 ######################################################################
 # Tests
@@ -123,6 +124,7 @@ def authz_open_directory(sbox):
                                         wc_dir)
 
 @Skip(svntest.main.is_ra_type_file)
+@SkipDumpLoadCrossCheck()
 def broken_authz_file(sbox):
   "broken authz files cause errors"
 

Modified: subversion/trunk/subversion/tests/cmdline/davautocheck.sh
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/davautocheck.sh?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/davautocheck.sh (original)
+++ subversion/trunk/subversion/tests/cmdline/davautocheck.sh Wed Feb  4 16:32:47 2015
@@ -401,6 +401,7 @@ fi
 say "Adding users for lock authentication"
 $HTPASSWD -bc $HTTPD_USERS jrandom   rayjandom
 $HTPASSWD -b  $HTTPD_USERS jconstant rayjandom
+$HTPASSWD -b  $HTTPD_USERS __dumpster__ __loadster__
 
 touch $HTTPD_MIME_TYPES
 

Modified: subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py Wed Feb  4 16:32:47 2015
@@ -1238,6 +1238,10 @@ def fsfs_recover_handle_missing_revs_or_
     ".*Revision 3 has a non-file where its revprops file should be.*"):
     raise svntest.Failure
 
+  # Restore the r3 revprops file, thus repairing the repository.
+  os.rmdir(revprop_3)
+  os.rename(revprop_was_3, revprop_3)
+
 
 #----------------------------------------------------------------------
 
@@ -2151,6 +2155,9 @@ def verify_keep_going(sbox):
                                    None, errput, None, "svnadmin: E165011:.*"):
     raise svntest.Failure
 
+  # Don't leave a corrupt repository
+  svntest.main.safe_rmtree(sbox.repo_dir, True)
+
 @SkipUnless(svntest.main.is_fs_type_fsfs)
 def verify_invalid_path_changes(sbox):
   "detect invalid changed path list entries"
@@ -2290,6 +2297,9 @@ def verify_invalid_path_changes(sbox):
                                    None, errput, None, "svnadmin: E165011:.*"):
     raise svntest.Failure
 
+  # Don't leave a corrupt repository
+  svntest.main.safe_rmtree(sbox.repo_dir, True)
+
 
 def verify_denormalized_names(sbox):
   "detect denormalized names and name collisions"
@@ -2669,6 +2679,9 @@ def verify_quickly(sbox):
                                    output, errput, exp_out, exp_err):
     raise svntest.Failure
 
+  # Don't leave a corrupt repository
+  svntest.main.safe_rmtree(sbox.repo_dir, True)
+
 
 @SkipUnless(svntest.main.is_fs_type_fsfs)
 @SkipUnless(svntest.main.fs_has_pack)

Modified: subversion/trunk/subversion/tests/cmdline/svnsync_authz_tests.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svnsync_authz_tests.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svnsync_authz_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svnsync_authz_tests.py Wed Feb  4 16:32:47 2015
@@ -37,6 +37,7 @@ import svntest
 from svntest.verify import SVNUnexpectedStdout, SVNUnexpectedStderr
 from svntest.verify import SVNExpectedStderr
 from svntest.main import write_restrictive_svnserve_conf
+from svntest.main import write_authz_file
 from svntest.main import server_has_partial_replay
 
 # Shared helpers
@@ -67,16 +68,14 @@ def basic_authz(sbox):
 
   run_init(dest_sbox.repo_url, sbox.repo_url)
 
-  args = tuple(s.authz_name() for s in [sbox, sbox, dest_sbox])
-  svntest.main.file_write(sbox.authz_file,
-                          "[%s:/]\n"
-                          "* = r\n"
-                          "\n"
-                          "[%s:/A/B]\n"
-                          "* = \n"
-                          "\n"
-                          "[%s:/]\n"
-                          "* = rw\n" % args)
+  src_authz = sbox.authz_name()
+  dst_authz = dest_sbox.authz_name()
+  write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':    '* = r',
+                       src_authz + ':/A/B': '* =',
+                       dst_authz + ':/':    '* = rw',
+                       })
 
   run_sync(dest_sbox.repo_url)
 
@@ -145,17 +144,14 @@ def copy_from_unreadable_dir(sbox):
 
   svntest.actions.enable_revprop_changes(dest_sbox.repo_dir)
 
-  args = tuple(s.authz_name() for s in [sbox, sbox, dest_sbox])
-  open(sbox.authz_file, 'w').write(
-             "[%s:/]\n"
-             "* = r\n"
-             "\n"
-             "[%s:/A/B]\n"
-             "* = \n"
-             "\n"
-             "[%s:/]\n"
-             "* = rw"
-             % args)
+  src_authz = sbox.authz_name()
+  dst_authz = dest_sbox.authz_name()
+  write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':    '* = r',
+                       src_authz + ':/A/B': '* =',
+                       dst_authz + ':/':    '* = rw',
+                       })
 
   run_init(dest_sbox.repo_url, sbox.repo_url)
 
@@ -259,17 +255,14 @@ def copy_with_mod_from_unreadable_dir(sb
 
   svntest.actions.enable_revprop_changes(dest_sbox.repo_dir)
 
-  args = tuple(s.authz_name() for s in [sbox, sbox, dest_sbox])
-  open(sbox.authz_file, 'w').write(
-             "[%s:/]\n"
-             "* = r\n"
-             "\n"
-             "[%s:/A/B]\n"
-             "* = \n"
-             "\n"
-             "[%s:/]\n"
-             "* = rw"
-             % args)
+  src_authz = sbox.authz_name()
+  dst_authz = dest_sbox.authz_name()
+  write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':    '* = r',
+                       src_authz + ':/A/B': '* =',
+                       dst_authz + ':/':    '* = rw',
+                       })
 
   run_init(dest_sbox.repo_url, sbox.repo_url)
 
@@ -351,17 +344,14 @@ def copy_with_mod_from_unreadable_dir_an
 
   svntest.actions.enable_revprop_changes(dest_sbox.repo_dir)
 
-  args = tuple(s.authz_name() for s in [sbox, sbox, dest_sbox])
-  open(sbox.authz_file, 'w').write(
-             "[%s:/]\n"
-             "* = r\n"
-             "\n"
-             "[%s:/A/B]\n"
-             "* = \n"
-             "\n"
-             "[%s:/]\n"
-             "* = rw"
-             % args)
+  src_authz = sbox.authz_name()
+  dst_authz = dest_sbox.authz_name()
+  write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':    '* = r',
+                       src_authz + ':/A/B': '* =',
+                       dst_authz + ':/':    '* = rw',
+                       })
 
   run_init(dest_sbox.repo_url, sbox.repo_url)
 
@@ -443,31 +433,24 @@ def specific_deny_authz(sbox):
   # For mod_dav_svn's parent path setup we need per-repos permissions in
   # the authz file...
   if sbox.repo_url.startswith('http'):
-    args = tuple(s.authz_name() for s in [sbox, sbox, sbox, dest_sbox])
-    svntest.main.file_write(sbox.authz_file,
-                            "[%s:/]\n"
-                            "* = r\n"
-                            "\n"
-                            "[%s:/A]\n"
-                            "* = \n"
-                            "\n"
-                            "[%s:/A_COPY/B/lambda]\n"
-                            "* = \n"
-                            "\n"
-                            "[%s:/]\n"
-                            "* = rw\n" % args)
+    src_authz = sbox.authz_name()
+    dst_authz = dest_sbox.authz_name()
+    write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':                '* = r',
+                       src_authz + ':/A':               '* =',
+                       src_authz + ':/A_COPY/B/lambda': '* =',
+                       dst_authz + ':/':                '* = rw',
+                       })
   # Otherwise we can just go with the permissions needed for the source
   # repository.
   else:
-    svntest.main.file_write(sbox.authz_file,
-                            "[/]\n"
-                            "* = r\n"
-                            "\n"
-                            "[/A]\n"
-                            "* = \n"
-                            "\n"
-                            "[/A_COPY/B/lambda]\n"
-                            "* = \n")
+    write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       '/':                '* = r',
+                       '/A':               '* =',
+                       '/A_COPY/B/lambda': '* =',
+                       })
 
   run_sync(dest_sbox.repo_url)
 
@@ -498,14 +481,13 @@ def copy_delete_unreadable_child(sbox):
   svntest.actions.enable_revprop_changes(dest_sbox.repo_dir)
 
   # Lock down the source.
-  authz = sbox.authz_name()
   write_restrictive_svnserve_conf(sbox.repo_dir, anon_access='read')
-  svntest.main.file_write(sbox.authz_file,
-      "[%s:/]\n"
-      "* = r\n"
-      "[%s:/A]\n"
-      "* =  \n"
-      % (authz, authz))
+  src_authz = sbox.authz_name()
+  write_authz_file(sbox, None,
+                   prefixed_rules = {
+                       src_authz + ':/':  '* = r',
+                       src_authz + ':/A': '* =',
+                       })
 
   dest_url = svntest.main.file_scheme_prefix \
                     + urllib.pathname2url(os.path.abspath(dest_sbox.repo_dir))

Modified: subversion/trunk/subversion/tests/cmdline/svntest/main.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svntest/main.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svntest/main.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svntest/main.py Wed Feb  4 16:32:47 2015
@@ -132,6 +132,10 @@ else:
 wc_author = 'jrandom'
 wc_passwd = 'rayjandom'
 
+# Username and password used by svnrdump in dump/load cross-checks
+crosscheck_username = '__dumpster__'
+crosscheck_password = '__loadster__'
+
 # Username and password used by the working copies for "second user"
 # scenarios
 wc_author2 = 'jconstant' # use the same password as wc_author
@@ -693,14 +697,25 @@ def _with_config_dir(args):
   else:
     return args + ('--config-dir', default_config_dir)
 
+class svnrdump_crosscheck_authentication:
+  pass
+
 def _with_auth(args):
   assert '--password' not in args
-  args = args + ('--password', wc_passwd,
+  if svnrdump_crosscheck_authentication in args:
+    args = filter(lambda x: x is not svnrdump_crosscheck_authentication, args)
+    auth_username = crosscheck_username
+    auth_password = crosscheck_password
+  else:
+    auth_username = wc_author
+    auth_password = wc_passwd
+
+  args = args + ('--password', auth_password,
                  '--no-auth-cache' )
   if '--username' in args:
     return args
   else:
-    return args + ('--username', wc_author )
+    return args + ('--username', auth_username )
 
 # For running subversion and returning the output
 def run_svn(error_expected, *varargs):
@@ -934,8 +949,13 @@ def _post_create_repos(path, minor_versi
     # This actually creates TWO [users] sections in the file (one of them is
     # uncommented in `svnadmin create`'s template), so we exercise the .ini
     # files reading code's handling of duplicates, too. :-)
-    file_append(os.path.join(path, "conf", "passwd"),
-                "[users]\njrandom = rayjandom\njconstant = rayjandom\n");
+    users = ("[users]\n"
+             "jrandom = rayjandom\n"
+             "jconstant = rayjandom\n")
+    if tests_verify_dump_load_cross_check():
+      # Insert a user for the dump/load cross-check.
+      users += (crosscheck_username + " = " + crosscheck_password + "\n")
+    file_append(os.path.join(path, "conf", "passwd"), users)
 
   if options.fs_type is None or options.fs_type == 'fsfs':
     # fsfs.conf file
@@ -1201,7 +1221,7 @@ def write_restrictive_svnserve_conf_with
 # parallel execution at the bottom like so
 #   if __name__ == '__main__':
 #     svntest.main.run_tests(test_list, serial_only = True)
-def write_authz_file(sbox, rules, sections=None):
+def write_authz_file(sbox, rules, sections=None, prefixed_rules=None):
   """Write an authz file to SBOX, appropriate for the RA method used,
 with authorizations rules RULES mapping paths to strings containing
 the rules. You can add sections SECTIONS (ex. groups, aliases...) with
@@ -1217,15 +1237,32 @@ an appropriate list of mappings.
   repo_name = os.path.basename(repo_name)
 
   if sbox.repo_url.startswith("http"):
-    prefix = repo_name + ":"
+    default_prefix = repo_name + ":"
   else:
-    prefix = ""
+    default_prefix = ""
+
   if sections:
     for p, r in sections.items():
       fp.write("[%s]\n%s\n" % (p, r))
 
-  for p, r in rules.items():
-    fp.write("[%s%s]\n%s\n" % (prefix, p, r))
+  if not prefixed_rules:
+    prefixed_rules = dict()
+
+  if rules:
+    for p, r in rules.items():
+      prefixed_rules[default_prefix + p] = r
+
+  for p, r in prefixed_rules.items():
+    fp.write("[%s]\n%s\n" % (p, r))
+    if tests_verify_dump_load_cross_check():
+      # Insert an ACE that lets the dump/load cross-check bypass
+      # authz restrictions.
+      fp.write(crosscheck_username + " = rw\n")
+
+  if tests_verify_dump_load_cross_check() and '/' not in prefixed_rules:
+    # We need a repository-root ACE for the dump/load cross-check
+    fp.write("[/]\n" + crosscheck_username + " = rw\n")
+
   fp.close()
 
 # See the warning about parallel test execution in write_authz_file
@@ -1379,6 +1416,9 @@ def make_log_msg():
 def tests_use_prepacakaged_repository():
   return options.fsfs_version is not None
 
+def tests_verify_dump_load_cross_check():
+  return options.dump_load_cross_check
+
 def is_ra_type_dav():
   return options.test_area_url.startswith('http')
 
@@ -1568,6 +1608,8 @@ class TestSpawningThread(threading.Threa
       args.append('--fsfs-packing')
     if options.fsfs_version:
       args.append('--fsfs-version=' + str(options.fsfs_version))
+    if options.dump_load_cross_check:
+      args.append('--dump-load-cross-check')
 
     result, stdout_lines, stderr_lines = spawn_process(command, 0, False, None,
                                                        *args)
@@ -1892,6 +1934,11 @@ def _create_parser():
                     help='Default shard size (for fsfs)')
   parser.add_option('--fsfs-version', type='int', action='store',
                     help='FSFS version (fsfs)')
+  parser.add_option('--dump-load-cross-check', action='store_true',
+                    help="After every test, run a series of dump and load " +
+                         "tests with svnadmin, svnrdump and svndumpfilter " +
+                         " on the testcase repositories to cross-check " +
+                         " dump file compatibility.")
   parser.add_option('--config-file', action='store',
                     help="Configuration file for tests.")
   parser.add_option('--set-log-level', action='callback', type='str',

Modified: subversion/trunk/subversion/tests/cmdline/svntest/sandbox.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svntest/sandbox.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svntest/sandbox.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svntest/sandbox.py Wed Feb  4 16:32:47 2015
@@ -26,6 +26,7 @@ import shutil
 import copy
 import urllib
 import logging
+import re
 
 import svntest
 
@@ -46,6 +47,8 @@ class Sandbox:
     # This flag is set to True by build() and returned by is_built()
     self._is_built = False
 
+    self.was_cwd = os.getcwd()
+
   def _set_name(self, name, read_only=False):
     """A convenience method for renaming a sandbox, useful when
     working with multiple repositories in the same unit test."""
@@ -381,6 +384,116 @@ class Sandbox:
     youngest = int(output[0])
     return youngest
 
+  def verify_repo(self):
+    """
+    """
+    svnrdump_headers_missing = re.compile(
+        "Text-content-sha1: .*|Text-copy-source-md5: .*|"
+        "Text-copy-source-sha1: .*|Text-delta-base-sha1: .*"
+    )
+    svnrdump_headers_always = re.compile(
+        "Prop-delta: .*"
+    )
+
+    dumpfile_a_n = svntest.actions.run_and_verify_dump(self.repo_dir,
+                                                       deltas=False)
+    dumpfile_a_d = svntest.actions.run_and_verify_dump(self.repo_dir,
+                                                       deltas=True)
+    dumpfile_r_d = svntest.actions.run_and_verify_svnrdump(
+      None, svntest.verify.AnyOutput, [], 0, 'dump', '-q', self.repo_url,
+      svntest.main.svnrdump_crosscheck_authentication)
+
+    # Compare the two deltas dumpfiles, ignoring expected differences
+    dumpfile_a_d_cmp = [l for l in dumpfile_a_d
+                       if not svnrdump_headers_missing.match(l)
+                                and not svnrdump_headers_always.match(l)]
+    dumpfile_r_d_cmp = [l for l in dumpfile_r_d
+                       if not svnrdump_headers_always.match(l)]
+    # Ignore differences in number of blank lines between node records,
+    # as svnrdump puts 3 whereas svnadmin puts 2 after a replace-with-copy.
+    svntest.verify.compare_dump_files(None, None,
+                                      dumpfile_a_d_cmp,
+                                      dumpfile_r_d_cmp,
+                                      ignore_number_of_blank_lines=True)
+
+    # Try loading the dump files.
+    # For extra points, load each with the other tool:
+    #   svnadmin dump | svnrdump load
+    #   svnrdump dump | svnadmin load
+    repo_dir_a_n, repo_url_a_n = self.add_repo_path('load_a_n')
+    svntest.main.create_repos(repo_dir_a_n)
+    svntest.actions.enable_revprop_changes(repo_dir_a_n)
+    svntest.actions.run_and_verify_svnrdump(
+      dumpfile_a_n, svntest.verify.AnyOutput, [], 0, 'load', repo_url_a_n,
+      svntest.main.svnrdump_crosscheck_authentication)
+
+    repo_dir_a_d, repo_url_a_d = self.add_repo_path('load_a_d')
+    svntest.main.create_repos(repo_dir_a_d)
+    svntest.actions.enable_revprop_changes(repo_dir_a_d)
+    svntest.actions.run_and_verify_svnrdump(
+      dumpfile_a_d, svntest.verify.AnyOutput, [], 0, 'load', repo_url_a_d,
+      svntest.main.svnrdump_crosscheck_authentication)
+
+    repo_dir_r_d, repo_url_r_d = self.add_repo_path('load_r_d')
+    svntest.main.create_repos(repo_dir_r_d)
+    svntest.actions.run_and_verify_load(repo_dir_r_d, dumpfile_r_d)
+
+    # Dump the loaded repositories in the same way; expect exact equality
+    reloaded_dumpfile_a_n = svntest.actions.run_and_verify_dump(repo_dir_a_n)
+    reloaded_dumpfile_a_d = svntest.actions.run_and_verify_dump(repo_dir_a_d)
+    reloaded_dumpfile_r_d = svntest.actions.run_and_verify_dump(repo_dir_r_d)
+    svntest.verify.compare_dump_files(None, None,
+                                      reloaded_dumpfile_a_n,
+                                      reloaded_dumpfile_a_d,
+                                      ignore_uuid=True)
+    svntest.verify.compare_dump_files(None, None,
+                                      reloaded_dumpfile_a_d,
+                                      reloaded_dumpfile_r_d,
+                                      ignore_uuid=True)
+
+    # Run each dump through svndumpfilter and check for no further change.
+    for dumpfile in [dumpfile_a_n,
+                     dumpfile_a_d,
+                     dumpfile_r_d
+                     ]:
+      ### No buffer size seems to work for update_tests-2. So skip that test?
+      ### (Its dumpfile size is ~360 KB non-delta, ~180 KB delta.)
+      if len(''.join(dumpfile)) > 100000:
+        continue
+
+      exit_code, dumpfile2, errput = svntest.main.run_command_stdin(
+        svntest.main.svndumpfilter_binary, None, -1, True,
+        dumpfile, '--quiet', 'include', '/')
+      assert not exit_code and not errput
+      # Ignore empty prop sections in the input file during comparison, as
+      # svndumpfilter strips them.
+      # Ignore differences in number of blank lines between node records,
+      # as svndumpfilter puts 3 instead of 2 after an add or delete record.
+      svntest.verify.compare_dump_files(None, None, dumpfile, dumpfile2,
+                                        expect_content_length_always=True,
+                                        ignore_empty_prop_sections=True,
+                                        ignore_number_of_blank_lines=True)
+
+  def verify(self, skip_cross_check=False):
+    """Do additional testing that should hold for any sandbox, such as
+       verifying that the repository can be dumped.
+    """
+    if (not skip_cross_check
+        and svntest.main.tests_verify_dump_load_cross_check()):
+      if self.is_built() and not self.read_only:
+        # verify that we can in fact dump the repo
+        # (except for the few tests that deliberately corrupt the repo)
+        os.chdir(self.was_cwd)
+        if os.path.exists(self.repo_dir):
+          logger.info("VERIFY: running dump/load cross-check")
+          self.verify_repo()
+      else:
+        logger.info("VERIFY: WARNING: skipping dump/load cross-check:"
+                    " is-built=%s, read-only=%s"
+                    % (self.is_built() and "true" or "false",
+                       self.read_only and "true" or "false"))
+    pass
+
 def is_url(target):
   return (target.startswith('^/')
           or target.startswith('file://')

Modified: subversion/trunk/subversion/tests/cmdline/svntest/testcase.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svntest/testcase.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svntest/testcase.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svntest/testcase.py Wed Feb  4 16:32:47 2015
@@ -28,7 +28,8 @@ import os, types, sys
 import svntest
 
 # if somebody does a "from testcase import *", they only get these names
-__all__ = ['_XFail', '_Wimp', '_Skip', '_SkipUnless']
+__all__ = ['_XFail', '_Wimp', '_Skip', '_SkipUnless',
+           '_SkipDumpLoadCrossCheck']
 
 RESULT_OK = 'ok'
 RESULT_FAIL = 'fail'
@@ -135,7 +136,7 @@ class FunctionTestCase(TestCase):
   is derived from the file name in which FUNC was defined)
   """
 
-  def __init__(self, func, issues=None):
+  def __init__(self, func, issues=None, skip_cross_check=False):
     # it better be a function that accepts an sbox parameter and has a
     # docstring on it.
     assert isinstance(func, types.FunctionType)
@@ -161,6 +162,7 @@ class FunctionTestCase(TestCase):
 
     TestCase.__init__(self, doc=doc, issues=issues)
     self.func = func
+    self.skip_cross_check = skip_cross_check
 
   def get_function_name(self):
     return self.func.func_name
@@ -173,7 +175,9 @@ class FunctionTestCase(TestCase):
     return os.path.splitext(os.path.basename(filename))[0]
 
   def run(self, sandbox):
-    return self.func(sandbox)
+    result = self.func(sandbox)
+    sandbox.verify(skip_cross_check = self.skip_cross_check)
+    return result
 
 
 class _XFail(TestCase):
@@ -261,11 +265,22 @@ class _SkipUnless(_Skip):
     _Skip.__init__(self, test_case, lambda c=cond_func: not c())
 
 
-def create_test_case(func, issues=None):
+class _SkipDumpLoadCrossCheck(TestCase):
+  """A test that will skip the post-test dump/load cross-check."""
+
+  def __init__(self, test_case, cond_func=lambda: True, wip=None,
+               issues=None):
+    TestCase.__init__(self,
+                      create_test_case(test_case, skip_cross_check=True),
+                      cond_func, wip=wip, issues=issues)
+
+
+def create_test_case(func, issues=None, skip_cross_check=False):
   if isinstance(func, TestCase):
     return func
   else:
-    return FunctionTestCase(func, issues=issues)
+    return FunctionTestCase(func, issues=issues,
+                            skip_cross_check=skip_cross_check)
 
 
 # Various decorators to make declaring tests as such simpler
@@ -322,5 +337,15 @@ def Issues_deco(*issues):
 
   return _second
 
+def SkipDumpLoadCrossCheck_deco(cond_func = lambda: True):
+  def _second(func):
+    if isinstance(func, TestCase):
+      return _SkipDumpLoadCrossCheck(func, cond_func, issues=func.issues)
+    else:
+      return _SkipDumpLoadCrossCheck(func, cond_func)
+
+  return _second
+
+
 # Create a singular alias, for linguistic correctness
 Issue_deco = Issues_deco

Modified: subversion/trunk/subversion/tests/cmdline/svntest/verify.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svntest/verify.py?rev=1657307&r1=1657306&r2=1657307&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svntest/verify.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svntest/verify.py Wed Feb  4 16:32:47 2015
@@ -474,8 +474,10 @@ class DumpParser:
     if not m:
       if required:
         raise SVNDumpParseError("expected '%s' at line %d\n%s"
+                                "\nPrevious lines:\n%s"
                                 % (regex, self.current,
-                                   self.lines[self.current]))
+                                   self.lines[self.current],
+                                   ''.join(self.lines[max(0,self.current - 10):self.current])))
       else:
         return None
     self.current += 1
@@ -507,6 +509,10 @@ class DumpParser:
       headers.append((key, val))
     return headers
 
+
+  def parse_boolean(self, header, required):
+    return self.parse_line(header + ': (false|true)$', required)
+
   def parse_format(self):
     return self.parse_line('SVN-fs-dump-format-version: ([0-9]+)$')
 
@@ -516,6 +522,9 @@ class DumpParser:
   def parse_revision(self):
     return self.parse_line('Revision-number: ([0-9]+)$')
 
+  def parse_prop_delta(self):
+    return self.parse_line('Prop-delta: (false|true)$', required=False)
+
   def parse_prop_length(self, required=True):
     return self.parse_line('Prop-content-length: ([0-9]+)$', required)
 
@@ -523,10 +532,7 @@ class DumpParser:
     return self.parse_line('Content-length: ([0-9]+)$', required)
 
   def parse_path(self):
-    path = self.parse_line('Node-path: (.+)$', required=False)
-    if not path and self.lines[self.current] == 'Node-path: \n':
-      self.current += 1
-      path = ''
+    path = self.parse_line('Node-path: (.*)$', required=False)
     return path
 
   def parse_kind(self):
@@ -557,6 +563,15 @@ class DumpParser:
   def parse_text_sha1(self):
     return self.parse_line('Text-content-sha1: ([0-9a-z]+)$', required=False)
 
+  def parse_text_delta(self):
+    return self.parse_line('Text-delta: (false|true)$', required=False)
+
+  def parse_text_delta_base_md5(self):
+    return self.parse_line('Text-delta-base-md5: ([0-9a-f]+)$', required=False)
+
+  def parse_text_delta_base_sha1(self):
+    return self.parse_line('Text-delta-base-sha1: ([0-9a-f]+)$', required=False)
+
   def parse_text_length(self):
     return self.parse_line('Text-content-length: ([0-9]+)$', required=False)
 
@@ -586,8 +601,14 @@ class DumpParser:
 
         return key
 
-      key = read_key_or_value(curprop)
-      value = read_key_or_value(curprop)
+      if props[curprop[0]].startswith('K'):
+        key = read_key_or_value(curprop)
+        value = read_key_or_value(curprop)
+      elif props[curprop[0]].startswith('D'):
+        key = read_key_or_value(curprop)
+        value = None
+      else:
+        raise
       prophash[key] = value
 
     return prophash
@@ -664,7 +685,7 @@ class DumpParser:
       if self.current >= len(self.lines):
         break
       path = self.parse_path()
-      if not path and not path is '':
+      if path is None:
         break
       if not nodes.get(path):
         nodes[path] = {}
@@ -702,7 +723,11 @@ class DumpParser:
     self.parse_all_revisions()
     return self.parsed
 
-def compare_dump_files(message, label, expected, actual):
+def compare_dump_files(message, label, expected, actual,
+                       ignore_uuid=False,
+                       expect_content_length_always=False,
+                       ignore_empty_prop_sections=False,
+                       ignore_number_of_blank_lines=False):
   """Parse two dump files EXPECTED and ACTUAL, both of which are lists
   of lines as returned by run_and_verify_dump, and check that the same
   revisions, nodes, properties, etc. are present in both dumps.
@@ -711,8 +736,39 @@ def compare_dump_files(message, label, e
   parsed_expected = DumpParser(expected).parse()
   parsed_actual = DumpParser(actual).parse()
 
+  if ignore_uuid:
+    parsed_expected['uuid'] = '<ignored>'
+    parsed_actual['uuid'] = '<ignored>'
+
+  for parsed in [parsed_expected, parsed_actual]:
+    for rev_name, rev_record in parsed.items():
+      #print "Found %s" % (rev_name,)
+      if 'nodes' in rev_record:
+        #print "Found %s.%s" % (rev_name, 'nodes')
+        for path_name, path_record in rev_record['nodes'].items():
+          #print "Found %s.%s.%s" % (rev_name, 'nodes', path_name)
+          for action_name, action_record in path_record.items():
+            #print "Found %s.%s.%s.%s" % (rev_name, 'nodes', path_name, action_name)
+
+            if expect_content_length_always:
+              if action_record.get('content_length') == None:
+                #print 'Adding: %s.%s.%s.%s.%s' % (rev_name, 'nodes', path_name, action_name,
'content_length=0')
+                action_record['content_length'] = '0'
+            if ignore_empty_prop_sections:
+              if action_record.get('prop_length') == '10':
+                #print 'Removing: %s.%s.%s.%s.%s' % (rev_name, 'nodes', path_name, action_name,
'prop_length')
+                action_record['prop_length'] = None
+                del action_record['props']
+                old_content_length = int(action_record['content_length'])
+                action_record['content_length'] = str(old_content_length - 10)
+            if ignore_number_of_blank_lines:
+              action_record['blanks'] = 0
+
   if parsed_expected != parsed_actual:
-    raise svntest.Failure('\n' + '\n'.join(ndiff(
+    print 'DIFF of raw dumpfiles (including expected differences)'
+    print ''.join(ndiff(expected, actual))
+    raise svntest.Failure('DIFF of parsed dumpfiles (ignoring expected differences)\n'
+                          + '\n'.join(ndiff(
           pprint.pformat(parsed_expected).splitlines(),
           pprint.pformat(parsed_actual).splitlines())))
 

Propchange: subversion/trunk/subversion/tests/libsvn_fs_x/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Wed Feb  4 16:32:47 2015
@@ -10,6 +10,7 @@
 /subversion/branches/diff-optimizations-bytes/subversion/tests/libsvn_fs_x:1037353-1067789
 /subversion/branches/dont-save-plaintext-passwords-by-default/subversion/tests/libsvn_fs_x:870728-871118
 /subversion/branches/double-delete/subversion/tests/libsvn_fs_x:870511-872970
+/subversion/branches/dump-load-cross-check/subversion/tests/libsvn_fs_x:1654853-1657295
 /subversion/branches/ev2-export/subversion/tests/libsvn_fs_x:1325914,1332738,1413107
 /subversion/branches/explore-wc/subversion/tests/libsvn_fs_x:875486,875493,875497,875507,875511,875514,875559,875580-875581,875584,875587,875611,875627,875647,875667-875668,875711-875712,875733-875734,875736,875744-875748,875751,875758,875782,875795-875796,875830,875836,875838,875842,875852,875855,875864,875870,875873,875880,875885-875888,875890,875897-875898,875905,875907-875909,875935,875943-875944,875946,875979,875982-875983,875985-875986,875990,875997
 /subversion/branches/file-externals/subversion/tests/libsvn_fs_x:871779-873302



Mime
View raw message