subversion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stef...@apache.org
Subject svn commit: r1597212 - in /subversion/trunk/subversion: include/private/ libsvn_fs_x/ libsvn_subr/ tests/cmdline/
Date Fri, 23 May 2014 22:56:23 GMT
Author: stefan2
Date: Fri May 23 22:56:22 2014
New Revision: 1597212

URL: http://svn.apache.org/r1597212
Log:
Sync'ing FSX with FSFS:
Port the low-level checksum support including verification and tests.

This is not a simple merge but mainly a "guided reimplementation",
taking the FSFS code as a blueprint but applying similar changes in
more places (as we have more item types in FSX) and accounting for
all the low-level differences.

As utility code, add private svn_checksum__* APIs to wrap streams
with a convenient checksum calculator.

Record the following revisions as affectively having been merged
from FSFS to FSX: r1573371,1588812,1588899,1588906.

* subversion/libsvn_fs_x
  (): Updated mergeinfo.

* subversion/include/private/svn_subr_private.h
  (svn_checksum__wrap_write_stream,
   svn_checksum__wrap_write_stream_fnv1a_32x4): Declare new private API;
                                only the second one is currently in use.

* subversion/libsvn_subr/checksum.c
  (stream_baton_t,
   write_handler,
   close_handler,
   wrap_write_stream): Implement generic checksumming stream wrapper.
  (svn_checksum__wrap_write_stream,
   close_handler_fnv1a_32x4,
   svn_checksum__wrap_write_stream_fnv1a_32x4): Implement new private
                                                APIs on top of it.

* subversion/libsvn_fs_x/index.h
  (svn_fs_x__p2l_entry_t): Attach a checksum to every item.

* subversion/libsvn_fs_x/index.c
  (svn_fs_x__p2l_index_create): Store the checksums.
  (read_entry): Read them again.
  (p2l_index_lookup): Explicitly init the checksum for the dynamically
                      created "past-end-of-file" dummy entry.

* subversion/libsvn_fs_x/fs_x.c
  (write_revision_zero): Update index P2L file template for r0.

* subversion/libsvn_fs_x/transaction.c
  (rep_write_baton): Hold the new low-level checksum.
  (rep_write_get_baton): Wrap the raw rep stream with a checksum calculator.
  (rep_write_contents_close): Store the checksum in the proto index entry.
  (write_container_delta_rep): Wrap the raw rep stream with a checksum
                               calculator and tell it to write directly
                               to the proto index entry.
  (write_final_rev): Calculate checksums for noderevs as well as ...
  (write_final_changed_path_info): ... for changed paths lists.

* subversion/libsvn_fs_x/pack.c
  (auto_pad_block): Explicitly initialize checksums for unused ranges.
  (write_nodes_container,
   write_reps_container,
   write_changes_container): Calculate and store the low-level checksums.

* subversion/libsvn_fs_x/verify.c
  (STREAM_THRESHOLD,
   expect_buffer_nul,
   read_all_nul,
   expected_checksum,
   expected_buffered_checksum,
   expected_streamed_checksum,
   compare_p2l_to_rev): New low-level checksum checking code taken 
                        over from FSFS.
  (verify_index_consistency): Execute the new verification step as well.

* subversion/tests/cmdline/svnadmin_tests.py
  (verify_quickly): Enable for FSX as well.

Modified:
    subversion/trunk/subversion/include/private/svn_subr_private.h
    subversion/trunk/subversion/libsvn_fs_x/   (props changed)
    subversion/trunk/subversion/libsvn_fs_x/fs_x.c
    subversion/trunk/subversion/libsvn_fs_x/index.c
    subversion/trunk/subversion/libsvn_fs_x/index.h
    subversion/trunk/subversion/libsvn_fs_x/pack.c
    subversion/trunk/subversion/libsvn_fs_x/transaction.c
    subversion/trunk/subversion/libsvn_fs_x/verify.c
    subversion/trunk/subversion/libsvn_subr/checksum.c
    subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py

Modified: subversion/trunk/subversion/include/private/svn_subr_private.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/include/private/svn_subr_private.h?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/include/private/svn_subr_private.h (original)
+++ subversion/trunk/subversion/include/private/svn_subr_private.h Fri May 23 22:56:22 2014
@@ -270,6 +270,32 @@ svn_checksum__from_digest_fnv1a_32x4(con
 
 
 /**
+ * Return a stream that calculates a checksum of type @a kind over all
+ * data written to the @a inner_stream.  When the returned stream gets
+ * closed, write the checksum to @a *checksum.
+ * Allocate the result in @a pool.
+ *
+ * @note The stream returned only supports #svn_stream_write and
+ * #svn_stream_close.
+ */
+svn_stream_t *
+svn_checksum__wrap_write_stream(svn_checksum_t **checksum,
+                                svn_stream_t *inner_stream,
+                                svn_checksum_kind_t kind,
+                                apr_pool_t *pool);
+
+/**
+ * Return a stream that calculates a 32 bit modified FNV-1a checksum
+ * over all data written to the @a inner_stream and writes the digest
+ * to @a *digest when the returned stream gets closed.
+ * Allocate the stream in @a pool.
+ */
+svn_stream_t *
+svn_checksum__wrap_write_stream_fnv1a_32x4(apr_uint32_t *digest,
+                                           svn_stream_t *inner_stream,
+                                           apr_pool_t *pool);
+
+/**
  * Return a 32 bit FNV-1a checksum for the first @a len bytes in @a input.
  * The representation is in Big Endian.
  *

Propchange: subversion/trunk/subversion/libsvn_fs_x/
------------------------------------------------------------------------------
  Merged /subversion/trunk/subversion/libsvn_fs_fs:r1573371,1588812,1588899,1588906

Modified: subversion/trunk/subversion/libsvn_fs_x/fs_x.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/fs_x.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/fs_x.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/fs_x.c Fri May 23 22:56:22 2014
@@ -829,14 +829,16 @@ write_revision_zero(svn_fs_t *fs)
   path = svn_fs_x__path_p2l_index(fs, 0, fs->pool);
   SVN_ERR(svn_io_file_create_binary
               (path,
-              "\0"                /* start rev */
-              "\x80\x80\4\1\x11"  /* 64k pages, 1 page using 17 bytes */
+              "\0\x7b"            /* start rev, rev file size */
+              "\x80\x80\4\1\x21"  /* 64k pages, 1 page using 33 bytes */
               "\0"                /* offset entry 0 page 1 */
-              "\x1d\x11\0\6"      /* len, type + 16 * count, (rev, 2*item)* */
-              "\x5d\x15\0\4"
-              "\1\x16\0\2"
-              "\x85\xff\3\0",     /* last entry fills up 64k page */
-              23,
+                                  /* len, type & count, checksum,
+                                     (rev, 2*item)* */
+              "\x1d\x11\x8e\xef\xf2\xd6\x01\0\6"
+              "\x5d\x15\xb6\xea\x97\xe0\x0f\0\4"
+              "\1\x16\x9d\x9e\xa9\x94\x0f\0\2"
+              "\x85\xff\3\0\0",   /* last entry fills up 64k page */
+              40,
               fs->pool));
   SVN_ERR(svn_io_set_file_read_only(path, FALSE, fs->pool));
 

Modified: subversion/trunk/subversion/libsvn_fs_x/index.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/index.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/index.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/index.c Fri May 23 22:56:22 2014
@@ -1925,6 +1925,9 @@ svn_fs_x__p2l_index_create(svn_fs_t *fs,
       SVN_ERR(svn_spillbuf__write(buffer, (const char *)encoded,
                                   encode_uint(encoded, entry.type + entry.item_count * 16),
                                   iter_pool));
+      SVN_ERR(svn_spillbuf__write(buffer, (const char *)encoded,
+                                  encode_uint(encoded, entry.fnv1_checksum),
+                                  iter_pool));
 
       /* container contents (only one for non-container items) */
       for (sub_item = 0; sub_item < entry.item_count; ++sub_item)
@@ -2227,6 +2230,8 @@ read_entry(packed_number_stream_t *strea
   SVN_ERR(packed_stream_get(&value, stream));
   entry.type = (int)value % 16;
   entry.item_count = (apr_uint32_t)(value / 16);
+  SVN_ERR(packed_stream_get(&value, stream));
+  entry.fnv1_checksum = (apr_uint32_t)value;
 
   if (entry.item_count == 0)
     {
@@ -2672,6 +2677,7 @@ p2l_index_lookup(apr_array_header_t *ent
               entry->offset = entry_end;
               entry->size = block_end - entry_end;
               entry->type = SVN_FS_X__ITEM_TYPE_UNUSED;
+              entry->fnv1_checksum = 0;
               entry->item_count = 0;
               entry->items = NULL;
             }

Modified: subversion/trunk/subversion/libsvn_fs_x/index.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/index.h?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/index.h (original)
+++ subversion/trunk/subversion/libsvn_fs_x/index.h Fri May 23 22:56:22 2014
@@ -67,6 +67,9 @@ typedef struct svn_fs_x__p2l_entry_t
   /* type of the item (see SVN_FS_X__ITEM_TYPE_*) defines */
   unsigned type;
 
+  /* modified FNV-1a checksum.  0 if unknown checksum */
+  apr_uint32_t fnv1_checksum;
+
   /* Number of items in this block / container.  Their list can be found
    * in *ITEMS.  0 for unused sections.  1 for non-container items,
    * > 1 for containers. */

Modified: subversion/trunk/subversion/libsvn_fs_x/pack.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/pack.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/pack.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/pack.c Fri May 23 22:56:22 2014
@@ -914,6 +914,7 @@ auto_pad_block(pack_context_t *context,
       null_entry.offset = context->pack_offset;
       null_entry.size = padding;
       null_entry.type = SVN_FS_X__ITEM_TYPE_UNUSED;
+      null_entry.fnv1_checksum = 0;
       null_entry.item_count = 0;
       null_entry.items = NULL;
 
@@ -1082,17 +1083,20 @@ write_nodes_container(pack_context_t *co
     return SVN_NO_ERROR;
 
   /* serialize container */
-  pack_stream = svn_stream_from_aprfile2(context->pack_file, TRUE,
-                                         scratch_pool);
-
+  container_entry = apr_palloc(context->info_pool, sizeof(*container_entry));
+  pack_stream = svn_checksum__wrap_write_stream_fnv1a_32x4
+                                (&container_entry->fnv1_checksum,
+                                 svn_stream_from_aprfile2(context->pack_file,
+                                                          TRUE, scratch_pool),
+                                 scratch_pool);
   SVN_ERR(svn_fs_x__write_noderevs_container(pack_stream, *container,
-                                              scratch_pool));
+                                             scratch_pool));
+  SVN_ERR(svn_stream_close(pack_stream));
   SVN_ERR(svn_io_file_seek(context->pack_file, APR_CUR, &offset,
                            scratch_pool));
 
   /* replace first noderev item in ENTRIES with the container
-    and set all others to NULL */
-  container_entry = apr_palloc(context->info_pool, sizeof(*container_entry));
+     and set all others to NULL */
   container_entry->offset = context->pack_offset;
   container_entry->size = offset - container_entry->offset;
   container_entry->type = SVN_FS_X__ITEM_TYPE_NODEREVS_CONT;
@@ -1233,9 +1237,14 @@ write_reps_container(pack_context_t *con
   svn_fs_x__p2l_entry_t container_entry;
 
   svn_stream_t *pack_stream
-    = svn_stream_from_aprfile2(context->pack_file, TRUE, pool);
+    = svn_checksum__wrap_write_stream_fnv1a_32x4
+                                (&container_entry.fnv1_checksum,
+                                 svn_stream_from_aprfile2(context->pack_file,
+                                                          TRUE, pool),
+                                 pool);
 
   SVN_ERR(svn_fs_x__write_reps_container(pack_stream, container, pool));
+  SVN_ERR(svn_stream_close(pack_stream));
   SVN_ERR(svn_io_file_seek(context->pack_file, SEEK_CUR, &offset, pool));
 
   container_entry.offset = context->pack_offset;
@@ -1539,11 +1548,16 @@ write_changes_container(pack_context_t *
   svn_fs_x__p2l_entry_t container_entry;
 
   svn_stream_t *pack_stream
-    = svn_stream_from_aprfile2(context->pack_file, TRUE, pool);
+    = svn_checksum__wrap_write_stream_fnv1a_32x4
+                                (&container_entry.fnv1_checksum,
+                                 svn_stream_from_aprfile2(context->pack_file,
+                                                          TRUE, pool),
+                                 pool);
 
   SVN_ERR(svn_fs_x__write_changes_container(pack_stream,
                                              container,
                                              pool));
+  SVN_ERR(svn_stream_close(pack_stream));
   SVN_ERR(svn_io_file_seek(context->pack_file, SEEK_CUR, &offset, pool));
 
   container_entry.offset = context->pack_offset;

Modified: subversion/trunk/subversion/libsvn_fs_x/transaction.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/transaction.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/transaction.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/transaction.c Fri May 23 22:56:22 2014
@@ -1898,6 +1898,9 @@ struct rep_write_baton
   svn_checksum_ctx_t *md5_checksum_ctx;
   svn_checksum_ctx_t *sha1_checksum_ctx;
 
+  /* Receives the low-level checksum when closing REP_STREAM. */
+  apr_uint32_t fnv1a_checksum;
+
   apr_pool_t *pool;
 
   apr_pool_t *parent_pool;
@@ -2135,7 +2138,10 @@ rep_write_get_baton(struct rep_write_bat
                                  b->pool));
 
   b->file = file;
-  b->rep_stream = svn_stream_from_aprfile2(file, TRUE, b->pool);
+  b->rep_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+                              &b->fnv1a_checksum,
+                              svn_stream_from_aprfile2(file, TRUE, b->pool),
+                              b->pool);
 
   SVN_ERR(svn_fs_x__get_file_offset(&b->rep_offset, file, b->pool));
 
@@ -2368,6 +2374,7 @@ rep_write_contents_close(void *baton)
     {
       svn_fs_x__p2l_entry_t entry;
       svn_fs_x__id_part_t noderev_id;
+      svn_checksum_t *checksum;
       noderev_id.change_set = SVN_FS_X__INVALID_CHANGE_SET;
       noderev_id.number = rep->id.number;
 
@@ -2377,6 +2384,7 @@ rep_write_contents_close(void *baton)
       entry.type = SVN_FS_X__ITEM_TYPE_FILE_REP;
       entry.item_count = 1;
       entry.items = &noderev_id;
+      entry.fnv1_checksum = b->fnv1a_checksum;
 
       SVN_ERR(store_sha1_rep_mapping(b->fs, b->noderev, b->pool));
       SVN_ERR(store_p2l_index_entry(b->fs, txn_id, &entry, b->pool));
@@ -2595,6 +2603,7 @@ write_container_delta_rep(representation
   svn_stream_t *stream;
   representation_t *base_rep;
   representation_t *old_rep;
+  svn_fs_x__p2l_entry_t entry;
   svn_stream_t *source;
   svn_fs_x__rep_header_t header = { 0 };
 
@@ -2626,7 +2635,10 @@ write_container_delta_rep(representation
       header.type = svn_fs_x__rep_self_delta;
     }
 
-  file_stream = svn_stream_from_aprfile2(file, TRUE, pool);
+  file_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+                                  &entry.fnv1_checksum,
+                                  svn_stream_from_aprfile2(file, TRUE, pool),
+                                  pool);
   SVN_ERR(svn_fs_x__write_rep_header(&header, file_stream, pool));
   SVN_ERR(svn_fs_x__get_file_offset(&delta_start, file, pool));
 
@@ -2670,7 +2682,6 @@ write_container_delta_rep(representation
     }
   else
     {
-      svn_fs_x__p2l_entry_t entry;
       svn_fs_x__id_part_t noderev_id;
 
       /* Write out our cosmetic end marker. */
@@ -2820,6 +2831,7 @@ write_final_rev(const svn_fs_id_t **new_
   svn_fs_x__txn_id_t txn_id = svn_fs_x__id_txn_id(id);
   svn_fs_x__p2l_entry_t entry;
   svn_fs_x__change_set_t change_set = svn_fs_x__change_set_by_rev(rev);
+  svn_stream_t *file_stream;
 
   *new_id_p = NULL;
 
@@ -2959,8 +2971,12 @@ write_final_rev(const svn_fs_id_t **new_
   if (at_root)
     SVN_ERR(validate_root_noderev(fs, noderev, rev, pool));
 
-  SVN_ERR(svn_fs_x__write_noderev(svn_stream_from_aprfile2(file, TRUE, pool),
-                                  noderev, ffd->format, pool));
+  file_stream = svn_checksum__wrap_write_stream_fnv1a_32x4(
+                                  &entry.fnv1_checksum,
+                                  svn_stream_from_aprfile2(file, TRUE, pool),
+                                  pool);
+  SVN_ERR(svn_fs_x__write_noderev(file_stream, noderev, ffd->format, pool));
+  SVN_ERR(svn_stream_close(file_stream));
 
   /* reference the root noderev from the log-to-phys index */
   noderev_id.change_set = SVN_FS_X__INVALID_CHANGE_SET;
@@ -2995,6 +3011,7 @@ write_final_changed_path_info(apr_off_t 
                               apr_pool_t *pool)
 {
   apr_off_t offset;
+  svn_stream_t *stream;
   apr_hash_index_t *hi;
   svn_fs_x__p2l_entry_t entry;
   svn_fs_x__id_part_t rev_item
@@ -3013,8 +3030,12 @@ write_final_changed_path_info(apr_off_t 
         change->copyfrom_rev = new_rev - 1;
     }
 
-  SVN_ERR(svn_fs_x__write_changes(svn_stream_from_aprfile2(file, TRUE, pool),
-                                  fs, changed_paths, TRUE, pool));
+  /* write to target file & calculate checksum */
+  stream = svn_checksum__wrap_write_stream_fnv1a_32x4(&entry.fnv1_checksum,
+                             svn_stream_from_aprfile2(file, TRUE, pool),
+                             pool);
+  SVN_ERR(svn_fs_x__write_changes(stream, fs, changed_paths, TRUE, pool));
+  SVN_ERR(svn_stream_close(stream));
 
   *offset_p = offset;
 

Modified: subversion/trunk/subversion/libsvn_fs_x/verify.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_x/verify.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_x/verify.c (original)
+++ subversion/trunk/subversion/libsvn_fs_x/verify.c Fri May 23 22:56:22 2014
@@ -22,6 +22,7 @@
 
 #include "verify.h"
 #include "fs_x.h"
+#include "private/svn_subr_private.h"
 
 #include "cached_data.h"
 #include "rep-cache.h"
@@ -314,8 +315,258 @@ compare_p2l_to_l2p_index(svn_fs_t *fs,
             }
         }
 
+      if (cancel_func)
+        SVN_ERR(cancel_func(cancel_baton));
+    }
+
+  svn_pool_destroy(iterpool);
+
+  return SVN_NO_ERROR;
+}
+
+/* Items smaller than this can be read at once into a buffer and directly
+ * be checksummed.  Larger items require stream processing.
+ * Must be a multiple of 8. */
+#define STREAM_THRESHOLD 4096
+
+/* Verify that the next SIZE bytes read from FILE are NUL.
+ * SIZE must not exceed STREAM_THRESHOLD.  Use POOL for allocations.
+ */
+static svn_error_t *
+expect_buffer_nul(apr_file_t *file,
+                  apr_off_t size,
+                  apr_pool_t *pool)
+{
+  union
+  {
+    unsigned char buffer[STREAM_THRESHOLD];
+    apr_uint64_t chunks[STREAM_THRESHOLD / sizeof(apr_uint64_t)];
+  } data;
+
+  apr_size_t i;
+  SVN_ERR_ASSERT(size <= STREAM_THRESHOLD);
+
+  /* read the whole data block; error out on failure */
+  data.chunks[(size - 1)/ sizeof(apr_uint64_t)] = 0;
+  SVN_ERR(svn_io_file_read_full2(file, data.buffer, size, NULL, NULL, pool));
+
+  /* chunky check */
+  for (i = 0; i < size / sizeof(apr_uint64_t); ++i)
+    if (data.chunks[i] != 0)
+      break;
+
+  /* byte-wise check upon mismatch or at the end of the block */
+  for (i *= sizeof(apr_uint64_t); i < size; ++i)
+    if (data.buffer[i] != 0)
+      {
+        const char *file_name;
+        apr_off_t offset;
+
+        SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+        SVN_ERR(svn_fs_x__get_file_offset(&offset, file, pool));
+        offset -= size - i;
+
+        return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
+                                 _("Empty section in file %s contains "
+                                   "non-NUL data at offset %s"),
+                                 file_name, apr_off_t_toa(pool, offset));
+      }
+
+  return SVN_NO_ERROR;
+}
+
+/* Verify that the next SIZE bytes read from FILE are NUL.
+ * Use POOL for allocations.
+ */
+static svn_error_t *
+read_all_nul(apr_file_t *file,
+             apr_off_t size,
+             apr_pool_t *pool)
+{
+  for (; size >= STREAM_THRESHOLD; size -= STREAM_THRESHOLD)
+    SVN_ERR(expect_buffer_nul(file, STREAM_THRESHOLD, pool));
+
+  if (size)
+    SVN_ERR(expect_buffer_nul(file, size, pool));
+
+  return SVN_NO_ERROR;
+}
+
+/* Compare the ACTUAL checksum with the one expected by ENTRY.
+ * Return an error in case of mismatch.  Use the name of FILE
+ * in error message.  Allocate data in POOL.
+ */
+static svn_error_t *
+expected_checksum(apr_file_t *file,
+                  svn_fs_x__p2l_entry_t *entry,
+                  apr_uint32_t actual,
+                  apr_pool_t *pool)
+{
+  if (actual != entry->fnv1_checksum)
+    {
+      const char *file_name;
+
+      SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+      SVN_ERR(svn_io_file_name_get(&file_name, file, pool));
+      return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
+                               _("Checksum mismatch item at offset %s of "
+                                 "length %s bytes in file %s"),
+                               apr_off_t_toa(pool, entry->offset),
+                               apr_off_t_toa(pool, entry->size), file_name);
+    }
+
+  return SVN_NO_ERROR;
+}
+
+/* Verify that the FNV checksum over the next ENTRY->SIZE bytes read
+ * from FILE will match ENTRY's expected checksum.  SIZE must not
+ * exceed STREAM_THRESHOLD.  Use POOL for allocations.
+ */
+static svn_error_t *
+expected_buffered_checksum(apr_file_t *file,
+                           svn_fs_x__p2l_entry_t *entry,
+                           apr_pool_t *pool)
+{
+  unsigned char buffer[STREAM_THRESHOLD];
+  SVN_ERR_ASSERT(entry->size <= STREAM_THRESHOLD);
+
+  SVN_ERR(svn_io_file_read_full2(file, buffer, (apr_size_t)entry->size,
+                                 NULL, NULL, pool));
+  SVN_ERR(expected_checksum(file, entry,
+                            svn__fnv1a_32x4(buffer, (apr_size_t)entry->size),
+                            pool));
+
+  return SVN_NO_ERROR;
+}
+
+/* Verify that the FNV checksum over the next ENTRY->SIZE bytes read from
+ * FILE will match ENTRY's expected checksum.  Use POOL for allocations.
+ */
+static svn_error_t *
+expected_streamed_checksum(apr_file_t *file,
+                           svn_fs_x__p2l_entry_t *entry,
+                           apr_pool_t *pool)
+{
+  unsigned char buffer[STREAM_THRESHOLD];
+  svn_checksum_t *checksum;
+  svn_checksum_ctx_t *context
+    = svn_checksum_ctx_create(svn_checksum_fnv1a_32x4, pool);
+  apr_off_t size = entry->size;
+
+  while (size > 0)
+    {
+      apr_size_t to_read = size > sizeof(buffer)
+                         ? sizeof(buffer)
+                         : (apr_size_t)size;
+      SVN_ERR(svn_io_file_read_full2(file, buffer, to_read, NULL, NULL,
+                                     pool));
+      SVN_ERR(svn_checksum_update(context, buffer, to_read));
+      size -= to_read;
+    }
+
+  SVN_ERR(svn_checksum_final(&checksum, context, pool));
+  SVN_ERR(expected_checksum(file, entry,
+                            ntohl(*(const apr_uint32_t *)checksum->digest),
+                            pool));
+
+  return SVN_NO_ERROR;
+}
+
+/* Verify that for all phys-to-log index entries for revisions START to
+ * START + COUNT-1 in FS match the actual pack / rev file contents.
+ * If given, invoke CANCEL_FUNC with CANCEL_BATON at regular intervals.
+ * Use POOL for allocations.
+ *
+ * Please note that we can only check on pack / rev file granularity and
+ * must only be called for a single rev / pack file.
+ */
+static svn_error_t *
+compare_p2l_to_rev(svn_fs_t *fs,
+                   svn_revnum_t start,
+                   svn_revnum_t count,
+                   svn_cancel_func_t cancel_func,
+                   void *cancel_baton,
+                   apr_pool_t *pool)
+{
+  fs_x_data_t *ffd = fs->fsap_data;
+  apr_pool_t *iterpool = svn_pool_create(pool);
+  apr_off_t max_offset;
+  apr_off_t offset = 0;
+  apr_file_t *rev_file;
+
+  /* open the pack / rev file that is covered by the p2l index */
+  SVN_ERR(svn_fs_x__open_pack_or_rev_file(&rev_file, fs, start, pool));
+
+  /* check file size vs. range covered by index */
+  SVN_ERR(svn_io_file_seek(rev_file, APR_END, &offset, pool));
+  SVN_ERR(svn_fs_x__p2l_get_max_offset(&max_offset, fs, start, pool));
+
+  if (offset != max_offset)
+    return svn_error_createf(SVN_ERR_FS_ITEM_INDEX_INCONSISTENT, NULL,
+                             _("File size of %s for revision r%ld does "
+                               "not match p2l index size of %s"),
+                             apr_off_t_toa(pool, offset), start,
+                             apr_off_t_toa(pool, max_offset));
+
+  SVN_ERR(svn_io_file_aligned_seek(rev_file, ffd->block_size, NULL, 0, pool));
+
+  /* for all offsets in the file, get the P2L index entries and check
+     them against the L2P index */
+  for (offset = 0; offset < max_offset; )
+    {
+      apr_array_header_t *entries;
+      int i;
+
       svn_pool_clear(iterpool);
 
+      /* get all entries for the current block */
+      SVN_ERR(svn_fs_x__p2l_index_lookup(&entries, fs, start, offset,
+                                         ffd->p2l_page_size, iterpool));
+
+      /* process all entries (and later continue with the next block) */
+      for (i = 0; i < entries->nelts; ++i)
+        {
+          svn_fs_x__p2l_entry_t *entry
+            = &APR_ARRAY_IDX(entries, i, svn_fs_x__p2l_entry_t);
+
+          /* skip bits we previously checked */
+          if (i == 0 && entry->offset < offset)
+            continue;
+
+          /* skip zero-sized entries */
+          if (entry->size == 0)
+            continue;
+
+          /* p2l index must cover all rev / pack file offsets exactly once */
+          if (entry->offset != offset)
+            return svn_error_createf(SVN_ERR_FS_ITEM_INDEX_INCONSISTENT,
+                                     NULL,
+                                     _("p2l index entry for revision r%ld"
+                                       " is non-contiguous between offsets "
+                                       " %s and %s"),
+                                     start,
+                                     apr_off_t_toa(pool, offset),
+                                     apr_off_t_toa(pool, entry->offset));
+
+          /* empty sections must contain NUL bytes only */
+          if (entry->type == SVN_FS_X__ITEM_TYPE_UNUSED)
+            {
+              /* skip filler entry at the end of the p2l index */
+              if (entry->offset != max_offset)
+                SVN_ERR(read_all_nul(rev_file, entry->size, pool));
+            }
+          else if (entry->fnv1_checksum)
+            {
+              if (entry->size < STREAM_THRESHOLD)
+                SVN_ERR(expected_buffered_checksum(rev_file, entry, pool));
+              else
+                SVN_ERR(expected_streamed_checksum(rev_file, entry, pool));
+            }
+
+          /* advance offset */
+          offset += entry->size;
+        }
+
       if (cancel_func)
         SVN_ERR(cancel_func(cancel_baton));
     }
@@ -360,6 +611,10 @@ verify_index_consistency(svn_fs_t *fs,
       SVN_ERR(compare_p2l_to_l2p_index(fs, pack_start, pack_end - pack_start,
                                        cancel_func, cancel_baton, iterpool));
 
+      /* verify in-index checksums and types vs. actual rev / pack files */
+      SVN_ERR(compare_p2l_to_rev(fs, pack_start, pack_end - pack_start,
+                                 cancel_func, cancel_baton, iterpool));
+
       svn_pool_clear(iterpool);
     }
 

Modified: subversion/trunk/subversion/libsvn_subr/checksum.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/checksum.c?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_subr/checksum.c (original)
+++ subversion/trunk/subversion/libsvn_subr/checksum.c Fri May 23 22:56:22 2014
@@ -661,3 +661,140 @@ svn_checksum_is_empty_checksum(svn_check
         SVN_ERR_MALFUNCTION_NO_RETURN();
     }
 }
+
+/* Checksum calculating stream wrappers.
+ */
+
+/* Baton used by write_handler and close_handler to calculate the checksum
+ * and return the result to the stream creator.  It accommodates the data
+ * needed by svn_checksum__wrap_write_stream_fnv1a_32x4 as well as
+ * svn_checksum__wrap_write_stream.
+ */
+typedef struct stream_baton_t
+{
+  /* Stream we are wrapping. Forward write() and close() operations to it. */
+  svn_stream_t *inner_stream;
+
+  /* Build the checksum data in here. */
+  svn_checksum_ctx_t *context;
+
+  /* Write the final checksum here. May be NULL. */
+  svn_checksum_t **checksum;
+
+  /* Copy the digest of the final checksum. May be NULL. */
+  unsigned char *digest;
+
+  /* Allocate the resulting checksum here. */
+  apr_pool_t *pool;
+} stream_baton_t;
+
+/* Implement svn_write_fn_t.
+ * Update checksum and pass data on to inner stream.
+ */
+static svn_error_t *
+write_handler(void *baton,
+              const char *data,
+              apr_size_t *len)
+{
+  stream_baton_t *b = baton;
+
+  SVN_ERR(svn_checksum_update(b->context, data, *len));
+  SVN_ERR(svn_stream_write(b->inner_stream, data, len));
+
+  return SVN_NO_ERROR;
+}
+
+/* Implement svn_close_fn_t.
+ * Finalize checksum calculation and write results. Close inner stream.
+ */
+static svn_error_t *
+close_handler(void *baton)
+{
+  stream_baton_t *b = baton;
+  svn_checksum_t *local_checksum;
+
+  /* Ensure we can always write to *B->CHECKSUM. */
+  if (!b->checksum)
+    b->checksum = &local_checksum;
+
+  /* Get the final checksum. */
+  SVN_ERR(svn_checksum_final(b->checksum, b->context, b->pool));
+
+  /* Extract digest, if wanted. */
+  if (b->digest)
+    {
+      apr_size_t digest_size = DIGESTSIZE((*b->checksum)->kind);
+      memcpy(b->digest, (*b->checksum)->digest, digest_size);
+    }
+
+  /* Done here.  Now, close the underlying stream as well. */
+  return svn_error_trace(svn_stream_close(b->inner_stream));
+}
+
+/* Common constructor function for svn_checksum__wrap_write_stream and
+ * svn_checksum__wrap_write_stream_fnv1a_32x4, taking the superset of their
+ * respecting parameters.
+ *
+ * In the current usage, either CHECKSUM or DIGEST will be NULL but this
+ * function does not enforce any such restriction.  Also, the caller must
+ * make sure that DIGEST refers to a buffer of sufficient length.
+ */
+svn_stream_t *
+wrap_write_stream(svn_checksum_t **checksum,
+                  unsigned char *digest,
+                  svn_stream_t *inner_stream,
+                  svn_checksum_kind_t kind,
+                  apr_pool_t *pool)
+{
+  svn_stream_t *outer_stream;
+
+  stream_baton_t *baton = apr_pcalloc(pool, sizeof(*baton));
+  baton->inner_stream = inner_stream;
+  baton->context = svn_checksum_ctx_create(kind, pool);
+  baton->checksum = checksum;
+  baton->digest = digest;
+  baton->pool = pool;
+
+  outer_stream = svn_stream_create(baton, pool);
+  svn_stream_set_write(outer_stream, write_handler);
+  svn_stream_set_close(outer_stream, close_handler);
+
+  return outer_stream;
+}
+
+svn_stream_t *
+svn_checksum__wrap_write_stream(svn_checksum_t **checksum,
+                                svn_stream_t *inner_stream,
+                                svn_checksum_kind_t kind,
+                                apr_pool_t *pool)
+{
+  return wrap_write_stream(checksum, NULL, inner_stream, kind, pool);
+}
+
+/* Implement svn_close_fn_t.
+ * For FNV-1a-like checksums, we want the checksum as 32 bit integer instead
+ * of a big endian 4 byte sequence.  This simply wraps close_handler adding
+ * the digest conversion.
+ */
+static svn_error_t *
+close_handler_fnv1a_32x4(void *baton)
+{
+  stream_baton_t *b = baton;
+  SVN_ERR(close_handler(baton));
+
+  *(apr_uint32_t *)b->digest = ntohl(*(apr_uint32_t *)b->digest);
+  return SVN_NO_ERROR;
+}
+
+svn_stream_t *
+svn_checksum__wrap_write_stream_fnv1a_32x4(apr_uint32_t *digest,
+                                           svn_stream_t *inner_stream,
+                                           apr_pool_t *pool)
+{
+  svn_stream_t *result
+    = wrap_write_stream(NULL, (unsigned char *)digest, inner_stream,
+                        svn_checksum_fnv1a_32x4, pool);
+  svn_stream_set_close(result, close_handler_fnv1a_32x4);
+
+  return result;
+}

Modified: subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py?rev=1597212&r1=1597211&r2=1597212&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py (original)
+++ subversion/trunk/subversion/tests/cmdline/svnadmin_tests.py Fri May 23 22:56:22 2014
@@ -2478,7 +2478,7 @@ def verify_metadata_only(sbox):
       'STDOUT', [], output)
 
 
-@SkipUnless(svntest.main.is_fs_type_fsfs)
+@Skip(svntest.main.is_fs_type_bdb)
 def verify_quickly(sbox):
   "verify quickly using metadata"
 



Mime
View raw message