subversion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From s...@apache.org
Subject svn commit: r1131120 - in /subversion/trunk: INSTALL Makefile.in build.conf configure.ac subversion/libsvn_client/add.c subversion/libsvn_subr/io.c
Date Fri, 03 Jun 2011 17:54:12 GMT
Author: stsp
Date: Fri Jun  3 17:54:12 2011
New Revision: 1131120

URL: http://svn.apache.org/viewvc?rev=1131120&view=rev
Log:
Add optional support for libmagic and use it to determine mime-types
of binary files.

libmagic is a BSD-licensed library for recognizing file types based
on magic numbers. It is shipped with many Linux distributions and
other UNIX-like systems as part of the file(1) utility.
libmagic also seems to have Windows support but I don't know how to
modify the Windows build system to compile with libmagic (help from
windows devs to fill this gap would be appreciated!).

The existing auto-props and mime-types-file mechanisms take precedence
over libmagic during mime-type detection. But libmagic has the advantage
that no configuration tweaks are needed and that it does not rely on the
filename extension to determine the mime-type.

Using libmagic also works around problems where Subversion's own heuristic
misdetects a binary file as text. I've seen this happen with some PDF
documents, for instance.

* subversion/libsvn_subr/io.c
  (svn_io_detect_mimetype2): If support for libmagic was compiled in
   and no mime-type was found in the provided mime-type map, try to
   get the mime-type from libmagic. If that fails fall back to the existing
   heuristic. Do not attempt to determine mime-types for empty files because
   that can result in mime-types such as "application/x-empty" which
   Subversion would interpret as binary.

* subversion/libsvn_client/add.c
  (svn_client__get_auto_props): Ignore "text/*" mimetypes returned from
   libmagic. This preserves existing default behavior of only setting the
   svn:mime-type property on binary files.

* Makefile.in: Expand libmagic library and include definitions.

* configure.ac: Detect libmagic. Provide an --with-libmagic switch to
   point at installations in non-default paths or disable libmagic support
   (--with-libmagic=no).

* INSTALL: Document libmagic dependency.

* build.conf:
  (libsvn_subr): Add optional libmagic dependency.
  (magic): New -- definition for optional libmagic dependency.

Modified:
    subversion/trunk/INSTALL
    subversion/trunk/Makefile.in
    subversion/trunk/build.conf
    subversion/trunk/configure.ac
    subversion/trunk/subversion/libsvn_client/add.c
    subversion/trunk/subversion/libsvn_subr/io.c

Modified: subversion/trunk/INSTALL
URL: http://svn.apache.org/viewvc/subversion/trunk/INSTALL?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/INSTALL (original)
+++ subversion/trunk/INSTALL Fri Jun  3 17:54:12 2011
@@ -146,6 +146,13 @@ I.    INTRODUCTION
          Subversion contains optional support for storing passwords in
          KWallet (KDE 4) or GNOME Keyring.
 
+      * libmagic
+
+         If the libmagic library is detected at compile time,
+         it will be used to determine mime-types of binary files
+         which are added to version control. Note that mime-types
+         configured via auto-props or the mime-types-file option
+         take precedence.
 
   C. Dependencies in Detail
 
@@ -565,6 +572,21 @@ I.    INTRODUCTION
 
       For more information on CSVN, see subversion/bindings/ctypes-python/README.
 
+      21. libmagic (OPTIONAL)
+
+      Subversion's configure script attempts to find libmagic automatically.
+      If it is installed in a non-standard location, then use:
+
+        --with-libmagic=/path/to/libmagic/prefix
+
+      The files include/magic.h and lib/libmagic.so.1.0 (or similar)
+      are expected beneath this prefix directory. If they cannot be
+      found Subversion will be compiled without support for libmagic.
+
+      If libmagic is installed but support for it should not be compiled
+      in, then use:
+      
+        --with-libmagic=no
 
   D. Documentation
 

Modified: subversion/trunk/Makefile.in
URL: http://svn.apache.org/viewvc/subversion/trunk/Makefile.in?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/Makefile.in (original)
+++ subversion/trunk/Makefile.in Fri Jun  3 17:54:12 2011
@@ -50,6 +50,7 @@ SVN_DB_LIBS = @SVN_DB_LIBS@
 SVN_GNOME_KEYRING_LIBS = @SVN_GNOME_KEYRING_LIBS@
 SVN_GSSAPI_LIBS = @SVN_GSSAPI_LIBS@
 SVN_KWALLET_LIBS = @SVN_KWALLET_LIBS@
+SVN_MAGIC_LIBS = @SVN_MAGIC_LIBS@
 SVN_SASL_LIBS = @SVN_SASL_LIBS@
 SVN_SERF_LIBS = @SVN_SERF_LIBS@
 SVN_SQLITE_LIBS = @SVN_SQLITE_LIBS@
@@ -121,9 +122,9 @@ LT_CXX_LIBADD = @LT_CXX_LIBADD@
 INCLUDES = -I$(top_srcdir)/subversion/include -I$(top_builddir)/subversion \
            @SVN_APR_INCLUDES@ @SVN_APRUTIL_INCLUDES@ @SVN_APR_MEMCACHE_INCLUDES@ \
            @SVN_DB_INCLUDES@ @SVN_GNOME_KEYRING_INCLUDES@ @SVN_GSSAPI_INCLUDES@ \
-           @SVN_KWALLET_INCLUDES@ @SVN_NEON_INCLUDES@ @SVN_SASL_INCLUDES@ \
-           @SVN_SERF_INCLUDES@ @SVN_SQLITE_INCLUDES@ @SVN_XML_INCLUDES@ \
-           @SVN_ZLIB_INCLUDES@
+           @SVN_KWALLET_INCLUDES@ @SVN_MAGIC_INCLUDES@ @SVN_NEON_INCLUDES@ \
+           @SVN_SASL_INCLUDES@ @SVN_SERF_INCLUDES@ @SVN_SQLITE_INCLUDES@ \
+           @SVN_XML_INCLUDES@ @SVN_ZLIB_INCLUDES@
 
 APACHE_INCLUDES = @APACHE_INCLUDES@
 APACHE_LIBEXECDIR = $(DESTDIR)@APACHE_LIBEXECDIR@

Modified: subversion/trunk/build.conf
URL: http://svn.apache.org/viewvc/subversion/trunk/build.conf?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/build.conf (original)
+++ subversion/trunk/build.conf Fri Jun  3 17:54:12 2011
@@ -315,7 +315,7 @@ description = Subversion General Utility
 type = lib
 install = fsmod-lib
 path = subversion/libsvn_subr
-libs = aprutil apriconv apr xml zlib apr_memcache sqlite
+libs = aprutil apriconv apr xml zlib apr_memcache sqlite magic
 msvc-libs = advapi32.lib shfolder.lib ole32.lib crypt32.lib version.lib
 msvc-export = 
         svn_auth.h svn_base64.h svn_cache_config.h svn_checksum.h svn_cmdline.h
@@ -1055,6 +1055,10 @@ external-lib = $(SVN_GSSAPI_LIBS)
 type = lib
 external-lib = $(SVN_KWALLET_LIBS)
 
+[magic]
+type = lib
+external-lib = $(SVN_MAGIC_LIBS)
+
 [sasl]
 type = lib
 external-lib = $(SVN_SASL_LIBS)

Modified: subversion/trunk/configure.ac
URL: http://svn.apache.org/viewvc/subversion/trunk/configure.ac?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/configure.ac (original)
+++ subversion/trunk/configure.ac Fri Jun  3 17:54:12 2011
@@ -686,6 +686,46 @@ if test $USE_NLS = "yes"; then
 fi
 AC_SUBST(MSGFMTFLAGS)
 
+dnl libmagic -------------------
+
+libmagic_found=no
+
+AC_ARG_WITH(libmagic,AS_HELP_STRING([--with-libmagic=PREFIX],
+                                [libmagic filetype detection library]),
+[
+  if test "$withval" = "yes" ; then
+    AC_MSG_ERROR([--with-libmagic requires an argument.])
+  else
+    libmagic_prefix=$withval
+    save_cppflags="$CPPFLAGS"
+    CPPFLAGS="$CPPFLAGS -I$libmagic_prefix/include"
+    AC_CHECK_HEADERS(magic.h,[
+      save_ldflags="$LDFLAGS"
+      LDFLAGS="-L$libmagic_prefix/lib"
+      AC_CHECK_LIB(magic, magic_open, [libmagic_found="yes"])
+      LDFLAGS="$save_ldflags"
+    ])
+    CPPFLAGS="$save_cppflags"
+  fi
+],
+[
+  AC_CHECK_HEADER(magic.h, [
+    AC_CHECK_LIB(magic, magic_open, [libmagic_found="builtin"])
+  ])
+])
+
+if test "$libmagic_found" != "no"; then
+  AC_DEFINE([HAVE_LIBMAGIC], [1], [Defined if libmagic support is enabled])
+  SVN_MAGIC_LIBS="-lmagic"
+fi
+
+if test "$libmagic_found" = "yes"; then
+  SVN_MAGIC_INCLUDES="-I$libmagic_prefix/include"
+  LDFLAGS="$LDFLAGS `SVN_REMOVE_STANDARD_LIB_DIRS(-L$libmagic_prefix/lib)`"
+fi
+
+AC_SUBST(SVN_MAGIC_INCLUDES)
+AC_SUBST(SVN_MAGIC_LIBS)
 
 dnl KWallet -------------------
 SVN_LIB_KWALLET

Modified: subversion/trunk/subversion/libsvn_client/add.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_client/add.c?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_client/add.c (original)
+++ subversion/trunk/subversion/libsvn_client/add.c Fri Jun  3 17:54:12 2011
@@ -253,6 +253,40 @@ svn_client__get_auto_props(apr_hash_t **
     {
       SVN_ERR(svn_io_detect_mimetype2(&autoprops.mimetype, path,
                                       ctx->mimetypes_map, pool));
+
+#ifdef HAVE_LIBMAGIC
+      /* We want to set an svn:mime-type property by default only on binary
+       * files. So don't set an svn:mime-type property on text files unless
+       * their mime-type appears in the map. This preserves behaviour
+       * of Subversion releases that did not include libmagic support.
+       * In those releases svn_io_detect_mimetype2() returned
+       * "application/octet-stream" or NULL unless the type was in the map. */
+      if (autoprops.mimetype && strncmp(autoprops.mimetype, "text/", 5) == 0)
+        {
+          if (ctx->mimetypes_map)
+            {
+              svn_boolean_t type_is_in_map = FALSE;
+              apr_hash_index_t *hi;
+
+              for (hi = apr_hash_first(pool, ctx->mimetypes_map);
+                   hi;
+                   hi = apr_hash_next(hi))
+                {
+                  const char *type_from_map = svn__apr_hash_index_val(hi);
+                  if (strcmp(type_from_map, autoprops.mimetype) == 0)
+                    {
+                      type_is_in_map = TRUE;
+                      break;
+                    }
+                }
+              if (!type_is_in_map)
+                autoprops.mimetype = NULL;
+            }
+          else
+            autoprops.mimetype = NULL;
+        }
+#endif
+
       if (autoprops.mimetype)
         apr_hash_set(autoprops.properties, SVN_PROP_MIME_TYPE,
                      strlen(SVN_PROP_MIME_TYPE),

Modified: subversion/trunk/subversion/libsvn_subr/io.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/io.c?rev=1131120&r1=1131119&r2=1131120&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_subr/io.c (original)
+++ subversion/trunk/subversion/libsvn_subr/io.c Fri Jun  3 17:54:12 2011
@@ -66,6 +66,10 @@
 #include "private/svn_atomic.h"
 #include "private/svn_io_private.h"
 
+#ifdef HAVE_LIBMAGIC
+#include <magic.h>
+#endif
+
 #define SVN_SLEEP_ENV_VAR "SVN_I_LOVE_CORRUPTED_WORKING_COPIES_SO_DISABLE_SLEEP_FOR_TIMESTAMPS"
 
 /*
@@ -2906,6 +2910,10 @@ svn_io_detect_mimetype2(const char **mim
   svn_error_t *err;
   unsigned char block[1024];
   apr_size_t amt_read = sizeof(block);
+  apr_finfo_t finfo;
+#ifdef HAVE_LIBMAGIC
+  magic_t magic;
+#endif
 
   /* Default return value is NULL. */
   *mimetype = NULL;
@@ -2933,6 +2941,40 @@ svn_io_detect_mimetype2(const char **mim
                              _("Can't detect MIME type of non-file '%s'"),
                              svn_dirent_local_style(file, pool));
 
+
+  /* Check if the file is empty and do not set a mime-type if it is.
+   * This also avoids spurious mime-types like "application/x-empty"
+   * from libmagic.
+   * ### merge this with the kind check above to save a stat() call */
+  SVN_ERR(svn_io_stat(&finfo, file, APR_FINFO_SIZE, pool));
+  if (finfo.size == 0)
+    return SVN_NO_ERROR;
+
+#ifdef HAVE_LIBMAGIC
+  /* Try to determine the mime-type with libmagic. */
+  magic = magic_open(MAGIC_MIME_TYPE | MAGIC_ERROR);
+  if (magic)
+    {
+      /* This loads the default magic database.
+       * Point the MAGIC environment variable at your favourite .mgc
+       * file to load a non-default database. */
+      if (magic_load(magic, NULL) != -1)
+        {
+          const char *magic_mime_type;
+
+          magic_mime_type = magic_file(magic, file);
+          if (magic_mime_type != NULL)
+            *mimetype = apr_pstrdup(pool, magic_mime_type);
+        }
+
+      /* This deallocates memory pointed to by magic_mime_type. */
+      magic_close(magic);
+
+      if (*mimetype)
+        return SVN_NO_ERROR;
+    }
+#endif
+
   SVN_ERR(svn_io_file_open(&fh, file, APR_READ, 0, pool));
 
   /* Read a block of data from FILE. */



Mime
View raw message