subversion-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From br...@apache.org
Subject svn commit: r1809081 [2/5] - in /subversion/upstream/utf8proc: ./ bench/ data/ pgsql/ ruby/ test/
Date Wed, 20 Sep 2017 20:55:40 GMT
Added: subversion/upstream/utf8proc/NEWS.md
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/NEWS.md?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/NEWS.md (added)
+++ subversion/upstream/utf8proc/NEWS.md Wed Sep 20 20:55:39 2017
@@ -0,0 +1,303 @@
+# utf8proc release history #
+
+## Version 2.1 ##
+
+2016-12-16:
+
+- New functions `utf8proc_map_custom` and `utf8proc_decompose_custom`
+  to allow user-supplied transformations of codepoints, in conjunction
+  with other transformations ([#89]).
+
+- New function `utf8proc_normalize_utf32` to apply normalizations
+  directly to UTF-32 data (not just UTF-8) ([#88]).
+
+- Fixed stack overflow that could occur due to incorrect definition
+  of `UINT16_MAX` with some compilers ([#84]).
+
+- Fixed conflict with `stdbool.h` in Visual Studio ([#90]).
+
+- Updated font metrics to use Unifont 9.0.04.
+
+## Version 2.0.2 ##
+
+2016-07-27:
+
+- Move `-Wmissing-prototypes` warning flag from `Makefile` to `.travis.yml`
+  since MSVC does not understand this flag and it is occasionally useful to
+  build using MSVC through the `Makefile` ([#79]).
+
+- Use a different variable name for a nested loop in `bench/bench.c`, and
+  declare it in a C89 way rather than inside the `for` to avoid "error:
+  'for' loop initial declarations are only allowed in C99 mode" ([#80]).
+
+## Version 2.0.1 ##
+
+2016-07-13:
+
+- Bug fix in `utf8proc_grapheme_break_stateful` ([#77]).
+
+- Tests now use versioned Unicode files, so they will no longer
+  break when a new version of Unicode is released ([#78]).
+
+## Version 2.0 ##
+
+2016-07-13:
+
+- Updated for Unicode 9.0 ([#70]).
+
+- New `utf8proc_grapheme_break_stateful` to handle the complicated
+  grapheme-breaking rules in Unicode 9.  The old `utf8proc_grapheme_break`
+  is still provided, but may incorrectly identify grapheme breaks
+  in some Unicode-9 sequences.
+
+- Smaller Unicode tables ([#62], [#68]).  This required changes
+  in the `utf8proc_property_t` structure, which breaks backward
+  compatibility if you access this `struct` directly.  The
+  functions in the API remain backward-compatible, however.
+
+- Buffer overrun fix ([#66]).
+
+## Version 1.3.1 ##
+
+2015-11-02:
+
+- Do not export symbol for internal function `unsafe_encode_char()` ([#55]).
+
+- Install relative symbolic links for shared libraries ([#58]).
+
+- Enable and fix compiler warnings ([#55], [#58]).
+
+- Add missing files to `make clean` ([#58]).
+
+## Version 1.3 ##
+
+2015-07-06:
+
+- Updated for Unicode 8.0 ([#45]).
+
+- New `utf8proc_tolower` and `utf8proc_toupper` functions, portable
+  replacements for `towlower` and `towupper` in the C library ([#40]).
+
+- Don't treat Unicode "non-characters" as invalid, and improved
+  validity checking in general ([#35]).
+
+- Prefix all typedefs with `utf8proc_`, e.g. `utf8proc_int32_t`,
+  to avoid collisions with other libraries ([#32]).
+
+- Rename `DLLEXPORT` to `UTF8PROC_DLLEXPORT` to prevent collisions.
+
+- Fix build breakage in the benchmark routines.
+
+- More fine-grained Makefile variables (`PICFLAG` etcetera), so that
+  compilation flags can be selectively overridden, and in particular
+  so that `CFLAGS` can be changed without accidentally eliminating
+  necessary flags like `-fPIC` and `-std=c99` ([#43]).
+
+- Updated character-width tables based on Unifont 8.0.01 ([#51]) and
+  the Unicode 8 character categories ([#47]).
+
+## Version 1.2 ##
+
+2015-03-28:
+
+- Updated for Unicode 7.0 ([#6]).
+
+- New function `utf8proc_grapheme_break(c1,c2)` that returns whether
+  there is a grapheme break between `c1` and `c2` ([#20]).
+
+- New function `utf8proc_charwidth(c)` that returns the number of
+  column-positions that should be required for `c`; essentially a
+  portable replacment for `wcwidth(c)` ([#27]).
+
+- New function `utf8proc_category(c)` that returns the Unicode
+  category of `c` (as one of the constants `UTF8PROC_CATEGORY_xx`).
+  Also, a function `utf8proc_category_string(c)` that returns the Unicode
+  category of `c` as a two-character string.
+
+- `cmake` script `CMakeLists.txt`, in addition to `Makefile`, for
+  easier compilation on Windows ([#28]).
+
+- Various `Makefile` improvements: a `make check` target to perform
+  tests ([#13]), `make install`, a rule to automate updating the Unicode
+  tables, etcetera.
+
+- The shared library is now versioned (e.g. has a soname on GNU/Linux) ([#24]).
+
+- C++/MSVC compatibility ([#17]).
+
+- Most `#defined` constants are now `enums` ([#29]).
+
+- New preprocessor constants `UTF8PROC_VERSION_MAJOR`,
+  `UTF8PROC_VERSION_MINOR`, and `UTF8PROC_VERSION_PATCH` for compile-time
+  detection of the API version.
+
+- Doxygen-formatted documentation ([#29]).
+
+- The Ruby and PostgreSQL plugins have been removed due to lack of testing ([#22]).
+
+## Version 1.1.6 ##
+
+2013-11-27:
+
+- PostgreSQL 9.2 and 9.3 compatibility (lowercase `c` language name)
+
+## Version 1.1.5 ##
+
+2009-08-20:
+
+- Use `RSTRING_PTR()` and `RSTRING_LEN()` instead of `RSTRING()->ptr` and
+  `RSTRING()->len` for ruby1.9 compatibility (and `#define` them, if not
+  existent)
+
+2009-10-02:
+
+- Patches for compatibility with Microsoft Visual Studio
+
+2009-10-08:
+
+- Fixes to make utf8proc usable in C++ programs
+
+2009-10-16:
+
+## Version 1.1.4 ##
+
+2009-06-14:
+
+- replaced C++ style comments for compatibility reasons
+- added typecasts to suppress compiler warnings
+- removed redundant source files for ruby-gemfile generation
+
+2009-08-19:
+
+- Changed copyright notice for Public Software Group e. V.
+- Minor changes in the `README` file
+
+## Version 1.1.3 ##
+
+2008-10-04:
+
+- Added a function `utf8proc_version` returning a string containing the version
+  number of the library.
+- Included a target `libutf8proc.dylib` for MacOSX.
+
+2009-05-01:
+- PostgreSQL 8.3 compatibility (use of `SET_VARSIZE` macro)
+
+## Version 1.1.2 ##
+
+2007-07-25:
+
+- Fixed a serious bug in the data file generator, which caused characters
+  being treated incorrectly, when stripping default ignorable characters or
+  calculating grapheme cluster boundaries.
+
+## Version 1.1.1 ##
+
+2007-06-25:
+
+- Added a new PostgreSQL function `unistrip`, which behaves like `unifold`,
+  but also removes all character marks (e.g. accents).
+
+2007-07-22:
+
+- Changed license from BSD to MIT style.
+- Added a new function `utf8proc_codepoint_valid` to the C library.
+- Changed compiler flags in `Makefile` from `-g -O0` to `-O2`
+- The ruby script, which was used to build the `utf8proc_data.c` file, is now
+  included in the distribution.
+
+## Version 1.0.3 ##
+
+2007-03-16:
+
+- Fixed a bug in the ruby library, which caused an error, when splitting an
+  empty string at grapheme cluster boundaries (method `String#utf8chars`).
+
+## Version 1.0.2 ##
+
+2006-09-21:
+
+- included a check in `Integer#utf8`, which raises an exception, if the given
+  code-point is invalid because of being too high (this was missing yet)
+
+2006-12-26:
+
+- added support for PostgreSQL version 8.2
+
+## Version 1.0.1 ##
+
+2006-09-20:
+
+- included a gem file for the ruby version of the library
+
+Release of version 1.0.1
+
+## Version 1.0 ##
+
+2006-09-17:
+
+- added the `LUMP` option, which lumps certain characters together (see `lump.md`) (also used for the PostgreSQL `unifold` function)
+- added the `STRIPMARK` option, which strips marking characters (or marks of composed characters)
+- deprecated ruby method `String#char_ary` in favour of `String#utf8chars`
+
+## Version 0.3 ##
+
+2006-07-18:
+
+- changed normalization from NFC to NFKC for postgresql unifold function
+
+2006-08-04:
+
+- added support to mark the beginning of a grapheme cluster with 0xFF (option: `CHARBOUND`)
+- added the ruby method `String#chars`, which is returning an array of UTF-8 encoded grapheme clusters
+- added `NLF2LF` transformation in postgresql `unifold` function
+- added the `DECOMPOSE` option, if you neither use `COMPOSE` or `DECOMPOSE`, no normalization will be performed (different from previous versions)
+- using integer constants rather than C-strings for character properties
+- fixed (hopefully) a problem with the ruby library on Mac OS X, which occurred when compiler optimization was switched on
+
+## Version 0.2 ##
+
+2006-06-05:
+
+- changed behaviour of PostgreSQL function to return NULL in case of invalid input, rather than raising an exceptional condition
+- improved efficiency of PostgreSQL function (no transformation to C string is done)
+
+2006-06-20:
+
+- added -fpic compiler flag in Makefile
+- fixed bug in the C code for the ruby library (usage of non-existent function)
+
+## Version 0.1 ##
+
+2006-06-02: initial release of version 0.1
+
+[#6]: https://github.com/JuliaLang/utf8proc/issues/6
+[#13]: https://github.com/JuliaLang/utf8proc/issues/13
+[#17]: https://github.com/JuliaLang/utf8proc/issues/17
+[#20]: https://github.com/JuliaLang/utf8proc/issues/20
+[#22]: https://github.com/JuliaLang/utf8proc/issues/22
+[#24]: https://github.com/JuliaLang/utf8proc/issues/24
+[#27]: https://github.com/JuliaLang/utf8proc/issues/27
+[#28]: https://github.com/JuliaLang/utf8proc/issues/28
+[#29]: https://github.com/JuliaLang/utf8proc/issues/29
+[#32]: https://github.com/JuliaLang/utf8proc/issues/32
+[#35]: https://github.com/JuliaLang/utf8proc/issues/35
+[#40]: https://github.com/JuliaLang/utf8proc/issues/40
+[#43]: https://github.com/JuliaLang/utf8proc/issues/43
+[#45]: https://github.com/JuliaLang/utf8proc/issues/45
+[#47]: https://github.com/JuliaLang/utf8proc/issues/47
+[#51]: https://github.com/JuliaLang/utf8proc/issues/51
+[#55]: https://github.com/JuliaLang/utf8proc/issues/55
+[#58]: https://github.com/JuliaLang/utf8proc/issues/58
+[#62]: https://github.com/JuliaLang/utf8proc/issues/62
+[#66]: https://github.com/JuliaLang/utf8proc/issues/66
+[#68]: https://github.com/JuliaLang/utf8proc/issues/68
+[#70]: https://github.com/JuliaLang/utf8proc/issues/70
+[#77]: https://github.com/JuliaLang/utf8proc/issues/77
+[#78]: https://github.com/JuliaLang/utf8proc/issues/78
+[#79]: https://github.com/JuliaLang/utf8proc/issues/79
+[#80]: https://github.com/JuliaLang/utf8proc/issues/80
+[#84]: https://github.com/JuliaLang/utf8proc/pull/84
+[#88]: https://github.com/JuliaLang/utf8proc/pull/88
+[#89]: https://github.com/JuliaLang/utf8proc/pull/89
+[#90]: https://github.com/JuliaLang/utf8proc/issues/90

Propchange: subversion/upstream/utf8proc/NEWS.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/README.md
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/README.md?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/README.md (added)
+++ subversion/upstream/utf8proc/README.md Wed Sep 20 20:55:39 2017
@@ -0,0 +1,69 @@
+# utf8proc
+[![Travis CI Status](https://travis-ci.org/JuliaLang/utf8proc.png)](https://travis-ci.org/JuliaLang/utf8proc)
+[![AppVeyor Status](https://ci.appveyor.com/api/projects/status/aou20lfkyhj8xbwq/branch/master?svg=true)](https://ci.appveyor.com/project/tkelman/utf8proc/branch/master)
+
+
+[utf8proc](http://julialang.org/utf8proc/) is a small, clean C
+library that provides Unicode normalization, case-folding, and other
+operations for data in the [UTF-8
+encoding](http://en.wikipedia.org/wiki/UTF-8).  It was [initially
+developed](http://www.public-software-group.org/utf8proc) by Jan
+Behrens and the rest of the [Public Software
+Group](http://www.public-software-group.org/), who deserve *nearly all
+of the credit* for this package.  With the blessing of the Public
+Software Group, the [Julia developers](http://julialang.org/) have
+taken over development of utf8proc, since the original developers have
+moved to other projects.
+
+(utf8proc is used for basic Unicode
+support in the [Julia language](http://julialang.org/), and the Julia
+developers became involved because they wanted to add Unicode 7 support and other features.)
+
+(The original utf8proc package also includes Ruby and PostgreSQL plug-ins.
+We removed those from utf8proc in order to focus exclusively on the C
+library for the time being, but plan to add them back in or release them as separate packages.)
+
+The utf8proc package is licensed under the
+free/open-source [MIT "expat"
+license](http://opensource.org/licenses/MIT) (plus certain Unicode
+data governed by the similarly permissive [Unicode data
+license](http://www.unicode.org/copyright.html#Exhibit1)); please see
+the included `LICENSE.md` file for more detailed information.
+
+## Quick Start
+
+For compilation of the C library run `make`.
+
+## General Information
+
+The C library is found in this directory after successful compilation
+and is named `libutf8proc.a` (for the static library) and
+`libutf8proc.so` (for the dynamic library).
+
+The Unicode version supported is 9.0.0.
+
+For Unicode normalizations, the following options are used:
+
+* Normalization Form C:  `STABLE`, `COMPOSE`
+* Normalization Form D:  `STABLE`, `DECOMPOSE`
+* Normalization Form KC: `STABLE`, `COMPOSE`, `COMPAT`
+* Normalization Form KD: `STABLE`, `DECOMPOSE`, `COMPAT`
+
+## C Library
+
+The documentation for the C library is found in the `utf8proc.h` header file.
+`utf8proc_map` is function you will most likely be using for mapping UTF-8
+strings, unless you want to allocate memory yourself.
+
+## To Do
+
+See the Github [issues list](https://github.com/JuliaLang/utf8proc/issues).
+
+## Contact
+
+Bug reports, feature requests, and other queries can be filed at
+the [utf8proc issues page on Github](https://github.com/JuliaLang/utf8proc/issues).
+
+## See also
+
+An independent Lua translation of this library, [lua-mojibake](https://github.com/differentprogramming/lua-mojibake), is also available.

Propchange: subversion/upstream/utf8proc/README.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/appveyor.yml
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/appveyor.yml?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/appveyor.yml (added)
+++ subversion/upstream/utf8proc/appveyor.yml Wed Sep 20 20:55:39 2017
@@ -0,0 +1,42 @@
+branches:
+  only:
+    - master
+    - /release-.*/
+
+notifications:
+  - provider: Email
+    on_build_success: false
+    on_build_failure: false
+    on_build_status_changed: false
+
+build_script:
+  - ps: if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod `
+      https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | `
+      Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { `
+      throw "There are newer queued builds for this pull request, failing early." }
+  - mkdir msvc_static
+  - cd msvc_static
+  - cmake ..
+  - cmake --build .
+  - mkdir ..\msvc_shared
+  - cd ..\msvc_shared
+  - cmake .. -DBUILD_SHARED_LIBS=ON
+  - cmake --build .
+  - set PATH=C:\MinGW\bin;%PATH%
+  - C:\MinGW\msys\1.0\bin\sh --login -c "
+      cd /c/projects/utf8proc &&
+      mkdir mingw_static &&
+      cd mingw_static &&
+      cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -G'MSYS Makefiles' &&
+      make &&
+      mkdir ../mingw_shared &&
+      cd ../mingw_shared &&
+      cmake .. -DCMAKE_VERBOSE_MAKEFILE=ON -DBUILD_SHARED_LIBS=ON -G'MSYS Makefiles' &&
+      make
+      "
+
+on_finish:
+  # Uncomment the following line for interactive debugging, which
+  # will print login data for a temporary remote session after the
+  # build. This requires an RDP version 6 client, e.g., FreeRDP.
+  #- ps: $blockRdp = $true; iex ((new-object net.webclient).DownloadString('https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-rdp.ps1'))

Propchange: subversion/upstream/utf8proc/appveyor.yml
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/Makefile
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/Makefile?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/Makefile (added)
+++ subversion/upstream/utf8proc/bench/Makefile Wed Sep 20 20:55:39 2017
@@ -0,0 +1,39 @@
+CURL=curl
+
+CC = cc
+CFLAGS = -O2 -std=c99 -pedantic -Wall
+
+all: bench
+
+LIBUTF8PROC = ../utf8proc.o
+
+bench: bench.o util.o $(LIBUTF8PROC)
+	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ bench.o util.o $(LIBUTF8PROC)
+
+DATAURL = https://raw.githubusercontent.com/duerst/eprun/master/benchmark
+DATAFILES = Deutsch_.txt Japanese_.txt Korean_.txt Vietnamese_.txt
+
+$(DATAFILES):
+	$(CURL) -O $(DATAURL)/$@
+
+bench.out: $(DATAFILES) bench
+	./bench -nfkc $(DATAFILES) > $@
+
+# you may need make CPPFLAGS=... LDFLAGS=... to help it find ICU
+icu: icu.o util.o
+	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ icu.o util.o -licuuc
+
+icu.out: $(DATAFILES) icu
+	./icu $(DATAFILES) > $@
+
+unistring: unistring.o util.o
+	$(CC) $(CFLAGS) $(LDFLAGS) -o $@ unistring.o util.o -lunistring
+
+unistring.out: $(DATAFILES) unistring
+	./unistring $(DATAFILES) > $@
+
+.c.o:
+	$(CC) $(CPPFLAGS) -I.. $(CFLAGS) -c -o $@ $<
+
+clean:
+	rm -rf *.o *.txt bench *.out icu unistring

Propchange: subversion/upstream/utf8proc/bench/Makefile
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/bench.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/bench.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/bench.c (added)
+++ subversion/upstream/utf8proc/bench/bench.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,56 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include "utf8proc.h"
+#include "util.h"
+
+int main(int argc, char **argv)
+{
+	 int i, j;
+	 int options = 0;
+	 
+	 for (i = 1; i < argc; ++i) {
+		  if (!strcmp(argv[i], "-nfkc")) {
+			   options |= UTF8PROC_STABLE|UTF8PROC_COMPOSE|UTF8PROC_COMPAT;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfkd")) {
+			   options |= UTF8PROC_STABLE|UTF8PROC_DECOMPOSE|UTF8PROC_COMPAT;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfc")) {
+			   options |= UTF8PROC_STABLE|UTF8PROC_COMPOSE;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfd")) {
+			   options |= UTF8PROC_STABLE|UTF8PROC_DECOMPOSE;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-casefold")) {
+			   options |= UTF8PROC_CASEFOLD;
+			   continue;
+		  }
+		  if (argv[i][0] == '-') {
+			   fprintf(stderr, "unrecognized option: %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+
+		  size_t len;
+		  uint8_t *src = readfile(argv[i], &len);
+		  if (!src) {
+			   fprintf(stderr, "error reading %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+		  uint8_t *dest;
+		  mytime start = gettime();
+		  for (j = 0; j < 100; ++j) {
+			   utf8proc_map(src, len, &dest, options);
+			   free(dest);
+		  }
+		  printf("%s: %g\n", argv[i], elapsed(gettime(), start) / 100);
+		  free(src);
+	 }
+
+	 return EXIT_SUCCESS;
+}

Propchange: subversion/upstream/utf8proc/bench/bench.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/icu.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/icu.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/icu.c (added)
+++ subversion/upstream/utf8proc/bench/icu.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,61 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+/* ICU4C */
+#include <unicode/utypes.h>
+#include <unicode/ustring.h>
+#include <unicode/ucnv.h>
+#include <unicode/unorm2.h>
+
+#include "util.h"
+
+int main(int argc, char **argv)
+{
+	 int i;
+
+	 UErrorCode err;
+	 UConverter *uc = ucnv_open("UTF8", &err);
+	 if (U_FAILURE(err)) return EXIT_FAILURE;
+
+	 const UNormalizer2 *NFKC = unorm2_getNFKCInstance(&err);
+	 if (U_FAILURE(err)) return EXIT_FAILURE;
+	 
+	 for (i = 1; i < argc; ++i) {
+		  if (argv[i][0] == '-') {
+			   fprintf(stderr, "unrecognized option: %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+
+		  size_t len;
+		  uint8_t *src = readfile(argv[i], &len);
+		  if (!src) {
+			   fprintf(stderr, "error reading %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+
+		  /* convert UTF8 data to ICU's UTF16 */
+		  UChar *usrc = (UChar*) malloc(2*len * sizeof(UChar));
+		  ucnv_toUChars(uc, usrc, 2*len, (char*) src, len, &err);
+		  if (U_FAILURE(err)) return EXIT_FAILURE;
+		  size_t ulen = u_strlen(usrc);
+
+		  /* ICU's insane normalization API requires you to
+			 know the size of the destination buffer in advance,
+			 or alternatively to repeatly try normalizing and
+			 double the buffer size until it succeeds.  Here, I just
+			 allocate a huge destination buffer to avoid the issue. */
+		  UChar *udest = (UChar*) malloc(10*ulen * sizeof(UChar));
+
+		  mytime start = gettime();
+		  for (int i = 0; i < 100; ++i) {
+			   unorm2_normalize(NFKC, usrc, ulen, udest, 10*ulen, &err);
+			   if (U_FAILURE(err)) return EXIT_FAILURE;
+		  }
+		  printf("%s: %g\n", argv[i], elapsed(gettime(), start) / 100);
+		  free(udest);
+		  free(usrc);
+		  free(src);
+	 }
+
+	 return EXIT_SUCCESS;
+}

Propchange: subversion/upstream/utf8proc/bench/icu.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/unistring.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/unistring.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/unistring.c (added)
+++ subversion/upstream/utf8proc/bench/unistring.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,60 @@
+/* comparitive benchmark of GNU libunistring */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+/* libunistring */
+#include <unistr.h>
+#include <uninorm.h>
+
+#include "util.h"
+
+int main(int argc, char **argv)
+{
+	 int i;
+	 uninorm_t nf = UNINORM_NFKC;
+
+	 for (i = 1; i < argc; ++i) {
+		  if (!strcmp(argv[i], "-nfkc")) {
+			   nf = UNINORM_NFKC;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfkd")) {
+			   nf = UNINORM_NFKD;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfc")) {
+			   nf = UNINORM_NFC;
+			   continue;
+		  }
+		  if (!strcmp(argv[i], "-nfd")) {
+			   nf = UNINORM_NFD;
+			   continue;
+		  }
+		  if (argv[i][0] == '-') {
+			   fprintf(stderr, "unrecognized option: %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+
+		  size_t len;
+		  uint8_t *src = readfile(argv[i], &len);
+		  if (!src) {
+			   fprintf(stderr, "error reading %s\n", argv[i]);
+			   return EXIT_FAILURE;
+		  }
+
+		  size_t destlen;
+		  uint8_t *dest;
+		  mytime start = gettime();
+		  for (int i = 0; i < 100; ++i) {
+			   dest = u8_normalize(nf, src, len, NULL, &destlen);
+			   if (!dest) return EXIT_FAILURE;
+			   free(dest);
+		  }
+		  printf("%s: %g\n", argv[i], elapsed(gettime(), start) / 100);
+		  free(src);
+	 }
+
+	 return EXIT_SUCCESS;
+}

Propchange: subversion/upstream/utf8proc/bench/unistring.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/util.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/util.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/util.c (added)
+++ subversion/upstream/utf8proc/bench/util.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,39 @@
+#include <stdio.h>
+#include <stdlib.h>
+#include <sys/stat.h>
+
+#include "util.h"
+
+/* read file named FILENAME into an array of *len bytes,
+   returning NULL on error */
+uint8_t *readfile(const char *filename, size_t *len)
+{
+	 *len = 0;
+	 struct stat st;
+	 if (0 != stat(filename, &st)) return NULL;
+	 *len = st.st_size;
+	 FILE *f = fopen(filename, "r");
+	 if (!f) return NULL;
+	 uint8_t *s = (uint8_t *) malloc(sizeof(uint8_t) * *len);
+	 if (!s) return NULL;
+	 if (fread(s, 1, *len, f) != *len) {
+		  free(s);
+		  s = NULL;
+	 }
+	 fclose(f);
+	 return s;
+}
+
+mytime gettime(void) {
+	 mytime t;
+	 gettimeofday(&t, NULL);
+	 return t;
+}
+
+/* time difference in seconds */
+double elapsed(mytime t1, mytime t0)
+{
+     return (double)(t1.tv_sec - t0.tv_sec) +
+          (double)(t1.tv_usec - t0.tv_usec) * 1.0E-6;
+}
+

Propchange: subversion/upstream/utf8proc/bench/util.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/bench/util.h
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/bench/util.h?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/bench/util.h (added)
+++ subversion/upstream/utf8proc/bench/util.h Wed Sep 20 20:55:39 2017
@@ -0,0 +1,22 @@
+#ifndef UTIL_H
+#define UTIL_H 1
+
+#include <inttypes.h>
+#include <sys/time.h>
+#include <time.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+uint8_t *readfile(const char *filename, size_t *len);
+
+typedef struct timeval mytime;
+mytime gettime(void);
+double elapsed(mytime t1, mytime t0);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* UTIL_H */

Propchange: subversion/upstream/utf8proc/bench/util.h
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/data/Makefile
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/data/Makefile?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/data/Makefile (added)
+++ subversion/upstream/utf8proc/data/Makefile Wed Sep 20 20:55:39 2017
@@ -0,0 +1,66 @@
+# Unicode data generation rules.  Except for the test data files, most
+# users will not use these Makefile rules, which are primarily to re-generate
+# unicode_data.c when we get a new Unicode version or charwidth data; they
+# require ruby, fontforge, and julia to be installed.
+
+# programs
+CURL=curl
+RUBY=ruby
+PERL=perl
+MAKE=make
+JULIA=julia
+FONTFORGE=fontforge
+CURLFLAGS = --retry 5 --location
+
+.PHONY: clean
+
+.DELETE_ON_ERROR:
+
+utf8proc_data.c.new: data_generator.rb UnicodeData.txt GraphemeBreakProperty.txt DerivedCoreProperties.txt CompositionExclusions.txt CaseFolding.txt CharWidths.txt
+	$(RUBY) data_generator.rb < UnicodeData.txt > $@
+
+# GNU Unifont version for font metric calculations:
+UNIFONT_VERSION=9.0.04
+
+unifont.ttf:
+	$(CURL) $(CURLFLAGS) -o $@ $(URLCACHE)https://mirrors.kernel.org/gnu/unifont/unifont-$(UNIFONT_VERSION)/unifont-$(UNIFONT_VERSION).ttf
+
+unifont_upper.ttf:
+	$(CURL) $(CURLFLAGS) -o $@ $(URLCACHE)https://mirrors.kernel.org/gnu/unifont/unifont-$(UNIFONT_VERSION)/unifont_upper-$(UNIFONT_VERSION).ttf
+
+%.sfd: %.ttf
+	$(FONTFORGE) -lang=ff -c "Open(\"$<\");Save(\"$@\");Quit(0);"
+
+CharWidths.txt: charwidths.jl unifont.sfd unifont_upper.sfd EastAsianWidth.txt
+	$(JULIA) charwidths.jl > $@
+
+# Unicode data version
+UNICODE_VERSION=9.0.0
+
+UnicodeData.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/UnicodeData.txt
+
+EastAsianWidth.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/EastAsianWidth.txt
+
+GraphemeBreakProperty.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/auxiliary/GraphemeBreakProperty.txt
+
+DerivedCoreProperties.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/DerivedCoreProperties.txt
+
+CompositionExclusions.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/CompositionExclusions.txt
+
+CaseFolding.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/CaseFolding.txt
+
+NormalizationTest.txt:
+	$(CURL) $(CURLFLAGS) -o $@ -O $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/NormalizationTest.txt
+
+GraphemeBreakTest.txt:
+	$(CURL) $(CURLFLAGS) $(URLCACHE)http://www.unicode.org/Public/$(UNICODE_VERSION)/ucd/auxiliary/GraphemeBreakTest.txt | $(PERL) -pe 's,÷,/,g;s,×,+,g' > $@
+
+clean:
+	rm -f UnicodeData.txt EastAsianWidth.txt GraphemeBreakProperty.txt DerivedCoreProperties.txt CompositionExclusions.txt CaseFolding.txt NormalizationTest.txt GraphemeBreakTest.txt CharWidths.txt unifont*.ttf unifont*.sfd
+	rm -f utf8proc_data.c.new

Propchange: subversion/upstream/utf8proc/data/Makefile
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/data/charwidths.jl
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/data/charwidths.jl?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/data/charwidths.jl (added)
+++ subversion/upstream/utf8proc/data/charwidths.jl Wed Sep 20 20:55:39 2017
@@ -0,0 +1,190 @@
+# Following work by @jiahao, we compute character widths using a combination of
+#   * advance widths from GNU Unifont (advance width 512 = 1 en)
+#   * UAX 11: East Asian Width
+#   * a few exceptions as needed
+# Adapted from http://nbviewer.ipython.org/gist/jiahao/07e8b08bf6d8671e9734
+#
+# Requires Julia (obviously) and FontForge.
+
+#############################################################################
+# Julia 0.3/0.4 compatibility (taken from Compat package)
+if VERSION < v"0.4.0-dev+1387"
+    typealias AbstractString String
+end
+if VERSION < v"0.4.0-dev+1419"
+    const UInt32 = Uint32
+end
+if VERSION < v"0.4.0-dev+3874"
+    Base.parse{T<:Integer}(::Type{T}, s::AbstractString) = parseint(T, s)
+end
+
+CharWidths = Dict{Int,Int}()
+
+#############################################################################
+# Use ../libutf8proc for category codes, rather than the one in Julia,
+# to minimize bootstrapping complexity when a new version of Unicode comes out.
+catcode(c) = ccall((:utf8proc_category,"../libutf8proc"), Cint, (Int32,), c)
+
+# use Base.UTF8proc module to get category codes constants, since
+# we won't change these in utf8proc.
+import Base.UTF8proc
+
+#############################################################################
+# Use a default width of 1 for all character categories that are
+# letter/symbol/number-like.  This can be overriden by Unifont or UAX 11
+# below, but provides a useful nonzero fallback for new codepoints when
+# a new Unicode version has been released but Unifont hasn't been updated yet.
+
+zerowidth = Set{Int}() # categories that may contain zero-width chars
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_CN)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_MN)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_MC)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_ME)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_SK)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_ZS)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_ZL)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_ZP)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_CC)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_CF)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_CS)
+push!(zerowidth, UTF8proc.UTF8PROC_CATEGORY_CO)
+for c in 0x0000:0x110000
+    if catcode(c) ∉ zerowidth
+        CharWidths[c] = 1
+    end
+end
+
+#############################################################################
+# Widths from GNU Unifont
+
+#Read sfdfile for character widths
+function parsesfd(filename::AbstractString, CharWidths::Dict{Int,Int}=Dict{Int,Int}())
+    state=:seekchar
+    lineno = 0
+    codepoint = width = nothing
+    for line in readlines(open(filename))
+        lineno += 1
+        if state==:seekchar         #StartChar: nonmarkingreturn
+            if contains(line, "StartChar: ")
+                codepoint = nothing
+                width = nothing
+                state = :readdata
+            end
+        elseif state==:readdata #Encoding: 65538 -1 2, Width: 1024
+            contains(line, "Encoding:") && (codepoint = parse(Int, split(line)[3]))
+            contains(line, "Width:") && (width = parse(Int, split(line)[2]))
+            if codepoint!=nothing && width!=nothing && codepoint >= 0
+                w=div(width, 512) # 512 units to the en
+                if w > 0
+                    # only add nonzero widths, since (1) the default is zero
+                    # and (2) this circumvents some apparent bugs in Unifont
+                    # (https://savannah.gnu.org/bugs/index.php?45395)
+                    CharWidths[codepoint] = w
+                end
+                state = :seekchar
+            end
+        end
+    end
+    CharWidths
+end
+CharWidths=parsesfd("unifont.sfd", CharWidths)
+CharWidths=parsesfd("unifont_upper.sfd", CharWidths)
+
+#############################################################################
+# Widths from UAX #11: East Asian Width
+#   .. these take precedence over the Unifont width for all codepoints
+#      listed explicitly as wide/full/narrow/half-width
+
+for line in readlines(open("EastAsianWidth.txt"))
+    #Strip comments
+    line[1] == '#' && continue
+    precomment = split(line, '#')[1]
+    #Parse code point range and width code
+    tokens = split(precomment, ';')
+    length(tokens) >= 2 || continue
+    charrange = tokens[1]
+    width = strip(tokens[2])
+    #Parse code point range into Julia UnitRange
+    rangetokens = split(charrange, "..")
+    charstart = parse(UInt32, "0x"*rangetokens[1])
+    charend = parse(UInt32, "0x"*rangetokens[length(rangetokens)>1 ? 2 : 1])
+
+    #Assign widths
+    for c in charstart:charend
+        if width=="W" || width=="F" # wide or full
+            CharWidths[c]=2
+        elseif width=="Na"|| width=="H" # narrow or half
+            CharWidths[c]=1
+        end
+    end
+end
+
+#############################################################################
+# A few exceptions to the above cases, found by manual comparison
+# to other wcwidth functions and similar checks.
+
+for c in keys(CharWidths)
+    cat = catcode(c)
+
+    # make sure format control character (category Cf) have width 0,
+    # except for the Arabic characters 0x06xx (see unicode std 6.2, sec. 8.2)
+    if cat==UTF8proc.UTF8PROC_CATEGORY_CF && c ∉ [0x0601,0x0602,0x0603,0x06dd]
+        CharWidths[c]=0
+    end
+
+    # Unifont has nonzero width for a number of non-spacing combining
+    # characters, e.g. (in 7.0.06): f84,17b4,17b5,180b,180d,2d7f, and
+    # the variation selectors
+    if cat==UTF8proc.UTF8PROC_CATEGORY_MN
+        CharWidths[c]=0
+    end
+
+    # We also assign width of zero to unassigned and private-use
+    # codepoints (Unifont includes ConScript Unicode Registry PUA fonts,
+    # but since these are nonstandard it seems questionable to recognize them).
+    if cat==UTF8proc.UTF8PROC_CATEGORY_CO || cat==UTF8proc.UTF8PROC_CATEGORY_CN
+        CharWidths[c]=0
+    end
+
+    # for some reason, Unifont has width-2 glyphs for ASCII control chars
+    if cat==UTF8proc.UTF8PROC_CATEGORY_CC
+        CharWidths[c]=0
+    end
+end
+
+#By definition, should have zero width (on the same line)
+#0x002028 '
' category: Zl name: LINE SEPARATOR/
+#0x002029 '
' category: Zp name: PARAGRAPH SEPARATOR/
+CharWidths[0x2028]=0
+CharWidths[0x2029]=0
+
+#By definition, should be narrow = width of 1 en space
+#0x00202f ' ' category: Zs name: NARROW NO-BREAK SPACE/
+CharWidths[0x202f]=1
+
+#By definition, should be wide = width of 1 em space
+#0x002001 ' ' category: Zs name: EM QUAD/
+#0x002003 ' ' category: Zs name: EM SPACE/
+CharWidths[0x2001]=2
+CharWidths[0x2003]=2
+
+#############################################################################
+# Output (to a file or pipe) for processing by data_generator.rb
+# ... don't bother to output zero widths since that will be the default.
+
+firstc = 0x000000
+lastv = 0
+uhex(c) = uppercase(hex(c,4))
+for c in 0x0000:0x110000
+    v = get(CharWidths, c, 0)
+    if v != lastv || c == 0x110000
+        v < 4 || error("invalid charwidth $v for $c")
+        if firstc+1 < c
+            println(uhex(firstc), "..", uhex(c-1), "; ", lastv)
+        else
+            println(uhex(firstc), "; ", lastv)
+        end
+        firstc = c
+        lastv = v
+    end
+end

Propchange: subversion/upstream/utf8proc/data/charwidths.jl
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/data/data_generator.rb
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/data/data_generator.rb?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/data/data_generator.rb (added)
+++ subversion/upstream/utf8proc/data/data_generator.rb Wed Sep 20 20:55:39 2017
@@ -0,0 +1,411 @@
+#!/usr/bin/env ruby
+
+#  This file was used to generate the 'unicode_data.c' file by parsing the
+#  Unicode data file 'UnicodeData.txt' of the Unicode Character Database.
+#  It is included for informational purposes only and not intended for
+#  production use.
+
+
+#  Copyright (c) 2009 Public Software Group e. V., Berlin, Germany
+#
+#  Permission is hereby granted, free of charge, to any person obtaining a
+#  copy of this software and associated documentation files (the "Software"),
+#  to deal in the Software without restriction, including without limitation
+#  the rights to use, copy, modify, merge, publish, distribute, sublicense,
+#  and/or sell copies of the Software, and to permit persons to whom the
+#  Software is furnished to do so, subject to the following conditions:
+#
+#  The above copyright notice and this permission notice shall be included in
+#  all copies or substantial portions of the Software.
+#
+#  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+#  FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+#  DEALINGS IN THE SOFTWARE.
+
+
+#  This file contains derived data from a modified version of the
+#  Unicode data files. The following license applies to that data:
+#
+#  COPYRIGHT AND PERMISSION NOTICE
+#
+#  Copyright (c) 1991-2007 Unicode, Inc. All rights reserved. Distributed
+#  under the Terms of Use in http://www.unicode.org/copyright.html.
+#
+#  Permission is hereby granted, free of charge, to any person obtaining a
+#  copy of the Unicode data files and any associated documentation (the "Data
+#  Files") or Unicode software and any associated documentation (the
+#  "Software") to deal in the Data Files or Software without restriction,
+#  including without limitation the rights to use, copy, modify, merge,
+#  publish, distribute, and/or sell copies of the Data Files or Software, and
+#  to permit persons to whom the Data Files or Software are furnished to do
+#  so, provided that (a) the above copyright notice(s) and this permission
+#  notice appear with all copies of the Data Files or Software, (b) both the
+#  above copyright notice(s) and this permission notice appear in associated
+#  documentation, and (c) there is clear notice in each modified Data File or
+#  in the Software as well as in the documentation associated with the Data
+#  File(s) or Software that the data or software has been modified.
+#
+#  THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
+#  KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+#  MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF
+#  THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS
+#  INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR
+#  CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF
+#  USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
+#  TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+#  PERFORMANCE OF THE DATA FILES OR SOFTWARE.
+#
+#  Except as contained in this notice, the name of a copyright holder shall
+#  not be used in advertising or otherwise to promote the sale, use or other
+#  dealings in these Data Files or Software without prior written
+#  authorization of the copyright holder.
+
+
+$ignorable_list = File.read("DerivedCoreProperties.txt")[/# Derived Property: Default_Ignorable_Code_Point.*?# Total code points:/m]
+$ignorable = []
+$ignorable_list.each_line do |entry|
+  if entry =~ /^([0-9A-F]+)\.\.([0-9A-F]+)/
+    $1.hex.upto($2.hex) { |e2| $ignorable << e2 }
+  elsif entry =~ /^[0-9A-F]+/
+    $ignorable << $&.hex
+  end
+end
+
+$grapheme_boundclass_list = File.read("GraphemeBreakProperty.txt")
+$grapheme_boundclass = Hash.new("UTF8PROC_BOUNDCLASS_OTHER")
+$grapheme_boundclass_list.each_line do |entry|
+  if entry =~ /^([0-9A-F]+)\.\.([0-9A-F]+)\s*;\s*([A-Za-z_]+)/
+    $1.hex.upto($2.hex) { |e2| $grapheme_boundclass[e2] = "UTF8PROC_BOUNDCLASS_" + $3.upcase }
+  elsif entry =~ /^([0-9A-F]+)\s*;\s*([A-Za-z_]+)/
+    $grapheme_boundclass[$1.hex] = "UTF8PROC_BOUNDCLASS_" + $2.upcase
+  end
+end
+
+$charwidth_list = File.read("CharWidths.txt")
+$charwidth = Hash.new(0)
+$charwidth_list.each_line do |entry|
+  if entry =~ /^([0-9A-F]+)\.\.([0-9A-F]+)\s*;\s*([0-9]+)/
+    $1.hex.upto($2.hex) { |e2| $charwidth[e2] = $3.to_i }
+  elsif entry =~ /^([0-9A-F]+)\s*;\s*([0-9]+)/
+    $charwidth[$1.hex] = $2.to_i
+  end
+end
+
+$exclusions = File.read("CompositionExclusions.txt")[/# \(1\) Script Specifics.*?# Total code points:/m]
+$exclusions = $exclusions.chomp.split("\n").collect { |e| e.hex }
+
+$excl_version = File.read("CompositionExclusions.txt")[/# \(2\) Post Composition Version precomposed characters.*?# Total code points:/m]
+$excl_version = $excl_version.chomp.split("\n").collect { |e| e.hex }
+
+$case_folding_string = File.open("CaseFolding.txt", :encoding => 'utf-8').read
+$case_folding = {}
+$case_folding_string.chomp.split("\n").each do |line|
+  next unless line =~ /([0-9A-F]+); [CFS]; ([0-9A-F ]+);/i
+  $case_folding[$1.hex] = $2.split(" ").collect { |e| e.hex }
+end
+
+$int_array = []
+$int_array_indicies = {}
+
+def str2c(string, prefix)
+  return "0" if string.nil?
+  return "UTF8PROC_#{prefix}_#{string.upcase}"
+end
+def pushary(array)
+  idx = $int_array_indicies[array]
+  unless idx
+    $int_array_indicies[array] = $int_array.length
+    idx = $int_array.length
+    array.each { |entry| $int_array << entry }
+  end
+  return idx
+end
+def cpary2utf16encoded(array)
+  return array.flat_map { |cp|
+      if (cp <= 0xFFFF)
+        raise "utf-16 code: #{cp}" if cp & 0b1111100000000000 == 0b1101100000000000
+        cp
+      else
+        temp = cp - 0x10000
+        [(temp >> 10) | 0b1101100000000000, (temp & 0b0000001111111111) | 0b1101110000000000]
+      end
+    }
+end
+def cpary2c(array)
+  return "UINT16_MAX" if array.nil? || array.length == 0
+  lencode = array.length - 1 #no sequence has len 0, so we encode len 1 as 0, len 2 as 1, ... 
+  array = cpary2utf16encoded(array)
+  if lencode >= 7 #we have only 3 bits for the length (which is already cutting it close. might need to change it to 2 bits in future Unicode versions)
+    array = [lencode] + array 
+    lencode = 7
+  end  
+  idx = pushary(array) 
+  raise "Array index out of bound" if idx > 0x1FFF
+  return "#{idx | (lencode << 13)}"
+end
+def singlecpmap(cp)
+  return "UINT16_MAX" if cp == nil
+  idx = pushary(cpary2utf16encoded([cp]))
+  raise "Array index out of bound" if idx > 0xFFFF
+  return "#{idx}"
+end
+
+class UnicodeChar
+  attr_accessor :code, :name, :category, :combining_class, :bidi_class,
+                :decomp_type, :decomp_mapping,
+                :bidi_mirrored,
+                :uppercase_mapping, :lowercase_mapping, :titlecase_mapping,
+                #caches:
+                :c_entry_index, :c_decomp_mapping, :c_case_folding
+  def initialize(line)
+    raise "Could not parse input." unless line =~ /^
+      ([0-9A-F]+);        # code
+      ([^;]+);            # name
+      ([A-Z]+);           # general category
+      ([0-9]+);           # canonical combining class
+      ([A-Z]+);           # bidi class
+      (<([A-Z]*)>)?       # decomposition type
+      ((\ ?[0-9A-F]+)*);  # decompomposition mapping
+      ([0-9]*);           # decimal digit
+      ([0-9]*);           # digit
+      ([^;]*);            # numeric
+      ([YN]*);            # bidi mirrored
+      ([^;]*);            # unicode 1.0 name
+      ([^;]*);            # iso comment
+      ([0-9A-F]*);        # simple uppercase mapping
+      ([0-9A-F]*);        # simple lowercase mapping
+      ([0-9A-F]*)$/ix     # simple titlecase mapping
+    @code              = $1.hex
+    @name              = $2
+    @category          = $3
+    @combining_class   = Integer($4)
+    @bidi_class        = $5
+    @decomp_type       = $7
+    @decomp_mapping    = ($8=='') ? nil :
+                         $8.split.collect { |element| element.hex }
+    @bidi_mirrored     = ($13=='Y') ? true : false
+    @uppercase_mapping = ($16=='') ? nil : $16.hex
+    @lowercase_mapping = ($17=='') ? nil : $17.hex
+    @titlecase_mapping = ($18=='') ? nil : $18.hex
+  end
+  def case_folding
+    $case_folding[code]
+  end
+  def c_entry(comb_indicies)
+    "  " <<
+    "{#{str2c category, 'CATEGORY'}, #{combining_class}, " <<
+    "#{str2c bidi_class, 'BIDI_CLASS'}, " <<
+    "#{str2c decomp_type, 'DECOMP_TYPE'}, " <<
+    "#{c_decomp_mapping}, " <<
+    "#{c_case_folding}, " <<
+    "#{singlecpmap uppercase_mapping }, " <<
+    "#{singlecpmap lowercase_mapping }, " <<
+    "#{singlecpmap titlecase_mapping }, " <<
+    "#{comb_indicies[code] ? comb_indicies[code]: 'UINT16_MAX'}, " <<
+    "#{bidi_mirrored}, " <<
+    "#{$exclusions.include?(code) or $excl_version.include?(code)}, " <<
+    "#{$ignorable.include?(code)}, " <<
+    "#{%W[Zl Zp Cc Cf].include?(category) and not [0x200C, 0x200D].include?(category)}, " <<
+    "#{$charwidth[code]}, 0, " <<
+    "#{$grapheme_boundclass[code]}},\n"
+  end
+end
+
+chars = []
+char_hash = {}
+
+while gets
+  if $_ =~ /^([0-9A-F]+);<[^;>,]+, First>;/i
+    first = $1.hex
+    gets
+    char = UnicodeChar.new($_)
+    raise "No last character of sequence found." unless
+      $_ =~ /^([0-9A-F]+);<([^;>,]+), Last>;/i
+    last = $1.hex
+    name = "<#{$2}>"
+    for i in first..last
+      char_clone = char.clone
+      char_clone.code = i
+      char_clone.name = name
+      char_hash[char_clone.code] = char_clone
+      chars << char_clone
+    end
+  else
+    char = UnicodeChar.new($_)
+    char_hash[char.code] = char
+    chars << char
+  end
+end
+
+comb1st_indicies = {}
+comb2nd_indicies = {}
+comb2nd_indicies_sorted_keys = []
+comb2nd_indicies_nonbasic = {}
+comb_array = []
+
+chars.each do |char|
+  if !char.nil? and char.decomp_type.nil? and char.decomp_mapping and
+      char.decomp_mapping.length == 2 and !char_hash[char.decomp_mapping[0]].nil? and
+      char_hash[char.decomp_mapping[0]].combining_class == 0 and
+      not $exclusions.include?(char.code)
+
+    dm0 = char.decomp_mapping[0]
+    dm1 = char.decomp_mapping[1]
+    unless comb1st_indicies[dm0]
+      comb1st_indicies[dm0] = comb1st_indicies.keys.length
+    end
+    unless comb2nd_indicies[dm1]
+      comb2nd_indicies_sorted_keys << dm1
+      comb2nd_indicies[dm1] = comb2nd_indicies.keys.length 
+    end
+    comb_array[comb1st_indicies[dm0]] ||= []
+    raise "Duplicate canonical mapping: #{char.code} #{dm0} #{dm1}" if comb_array[comb1st_indicies[dm0]][comb2nd_indicies[dm1]]
+    comb_array[comb1st_indicies[dm0]][comb2nd_indicies[dm1]] = char.code
+    
+    comb2nd_indicies_nonbasic[dm1] = true if char.code > 0xFFFF
+  end
+  char.c_decomp_mapping = cpary2c(char.decomp_mapping)
+  char.c_case_folding = cpary2c(char.case_folding)
+end 
+
+comb_indicies = {}
+cumoffset = 0
+comb1st_indicies_lastoffsets = []
+comb1st_indicies_firstoffsets = []
+comb1st_indicies.each do |dm0, index|
+  first = nil
+  last = nil
+  offset = 0
+  comb2nd_indicies_sorted_keys.each_with_index do |dm1, b|
+    if comb_array[index][b] 
+      first = offset unless first
+      last = offset
+      last += 1 if comb2nd_indicies_nonbasic[dm1]
+    end
+    offset += 1
+    offset += 1 if comb2nd_indicies_nonbasic[dm1]
+  end
+  comb1st_indicies_firstoffsets[index] = first
+  comb1st_indicies_lastoffsets[index] = last
+  raise "double index" if comb_indicies[dm0]
+  comb_indicies[dm0] = cumoffset
+  cumoffset += last - first + 1 + 2
+end
+
+offset = 0
+comb2nd_indicies_sorted_keys.each do |dm1|
+  raise "double index" if comb_indicies[dm1]
+  comb_indicies[dm1] = 0x8000 | (comb2nd_indicies[dm1] + offset)
+  raise "too large comb index" if  comb2nd_indicies[dm1] + offset > 0x4000
+  if comb2nd_indicies_nonbasic[dm1]
+    comb_indicies[dm1] = comb_indicies[dm1] | 0x4000
+    offset += 1
+  end
+end
+
+properties_indicies = {}
+properties = []
+chars.each do |char|
+  c_entry = char.c_entry(comb_indicies)
+  char.c_entry_index = properties_indicies[c_entry]
+  unless char.c_entry_index
+    properties_indicies[c_entry] = properties.length
+    char.c_entry_index = properties.length
+    properties << c_entry
+  end
+end
+
+stage1 = []
+stage2 = []
+for code in 0...0x110000
+  next unless code % 0x100 == 0
+  stage2_entry = []
+  for code2 in code...(code+0x100)
+    if char_hash[code2]
+      stage2_entry << (char_hash[code2].c_entry_index + 1)
+    else
+      stage2_entry << 0
+    end
+  end
+  old_index = stage2.index(stage2_entry)
+  if old_index
+    stage1 << (old_index * 0x100)
+  else
+    stage1 << (stage2.length * 0x100)
+    stage2 << stage2_entry
+  end
+end
+
+$stdout << "const utf8proc_uint16_t utf8proc_sequences[] = {\n  "
+i = 0
+$int_array.each do |entry|
+  i += 1
+  if i == 8
+    i = 0
+    $stdout << "\n  "
+  end
+  $stdout << entry << ", "
+end
+$stdout << "};\n\n"
+
+$stdout << "const utf8proc_uint16_t utf8proc_stage1table[] = {\n  "
+i = 0
+stage1.each do |entry|
+  i += 1
+  if i == 8
+    i = 0
+    $stdout << "\n  "
+  end
+  $stdout << entry << ", "
+end
+$stdout << "};\n\n"
+
+$stdout << "const utf8proc_uint16_t utf8proc_stage2table[] = {\n  "
+i = 0
+stage2.flatten.each do |entry|
+  i += 1
+  if i == 8
+    i = 0
+    $stdout << "\n  "
+  end
+  $stdout << entry << ", "
+end
+$stdout << "};\n\n"
+
+$stdout << "const utf8proc_property_t utf8proc_properties[] = {\n"
+$stdout << "  {0, 0, 0, 0, UINT16_MAX, UINT16_MAX, UINT16_MAX, UINT16_MAX, UINT16_MAX, UINT16_MAX,  false,false,false,false, 0, 0, UTF8PROC_BOUNDCLASS_OTHER},\n"
+properties.each { |line|
+  $stdout << line
+}
+$stdout << "};\n\n"
+
+
+
+$stdout << "const utf8proc_uint16_t utf8proc_combinations[] = {\n  "
+i = 0
+comb1st_indicies.keys.each_index do |a|
+  offset = 0
+  $stdout << comb1st_indicies_firstoffsets[a] << ", " << comb1st_indicies_lastoffsets[a] << ", "
+  comb2nd_indicies_sorted_keys.each_with_index do |dm1, b|
+    break if offset > comb1st_indicies_lastoffsets[a] 
+    if offset >= comb1st_indicies_firstoffsets[a]
+      i += 1
+      if i == 8
+        i = 0
+        $stdout << "\n  "
+      end
+      v = comb_array[a][b] ? comb_array[a][b] : 0
+      $stdout << (( v & 0xFFFF0000 ) >> 16) << ", " if comb2nd_indicies_nonbasic[dm1]
+      $stdout << (v & 0xFFFF) << ", "
+    end
+    offset += 1
+    offset += 1 if comb2nd_indicies_nonbasic[dm1]    
+  end
+  $stdout  << "\n"
+end
+$stdout << "};\n\n"
+

Propchange: subversion/upstream/utf8proc/data/data_generator.rb
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/lump.md
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/lump.md?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/lump.md (added)
+++ subversion/upstream/utf8proc/lump.md Wed Sep 20 20:55:39 2017
@@ -0,0 +1,27 @@
+```
+U+0020      <-- all space characters (general category Zs)
+U+0027  '   <-- left/right single quotation mark U+2018..2019,
+                modifier letter apostrophe U+02BC,
+                modifier letter vertical line U+02C8
+U+002D  -   <-- all dash characters (general category Pd),
+                minus U+2212
+U+002F  /   <-- fraction slash U+2044,
+                division slash U+2215
+U+003A  :   <-- ratio U+2236
+U+003C  <   <-- single left-pointing angle quotation mark U+2039,
+                left-pointing angle bracket U+2329,
+                left angle bracket U+3008
+U+003E  >   <-- single right-pointing angle quotation mark U+203A,
+                right-pointing angle bracket U+232A,
+                right angle bracket U+3009
+U+005C  \   <-- set minus U+2216
+U+005E  ^   <-- modifier letter up arrowhead U+02C4,
+                modifier letter circumflex accent U+02C6,
+                caret U+2038,
+                up arrowhead U+2303
+U+005F  _   <-- all connector characters (general category Pc),
+                modifier letter low macron U+02CD
+U+0060  `   <-- modifier letter grave accent U+02CB
+U+007C  |   <-- divides U+2223
+U+007E  ~   <-- tilde operator U+223C
+```

Propchange: subversion/upstream/utf8proc/lump.md
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/case.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/case.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/case.c (added)
+++ subversion/upstream/utf8proc/test/case.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,50 @@
+#include "tests.h"
+#include <wctype.h>
+
+int main(int argc, char **argv)
+{
+     int error = 0, better = 0;
+     utf8proc_int32_t c;
+
+     (void) argc; /* unused */
+     (void) argv; /* unused */
+
+     /* some simple sanity tests of the character widths */
+     for (c = 0; c <= 0x110000; ++c) {
+          utf8proc_int32_t l = utf8proc_tolower(c);
+          utf8proc_int32_t u = utf8proc_toupper(c);
+
+          check(l == c || utf8proc_codepoint_valid(l), "invalid tolower");
+          check(u == c || utf8proc_codepoint_valid(u), "invalid toupper");
+
+          if (sizeof(wint_t) > 2 || c < (1<<16)) {
+               wint_t l0 = towlower(c), u0 = towupper(c);
+               
+               /* OS unicode tables may be out of date.  But if they
+                  do have a lower/uppercase mapping, hopefully it
+                  is correct? */
+               if (l0 != c && l0 != l) {
+                    fprintf(stderr, "MISMATCH %x != towlower(%x) == %x\n",
+                            l, c, l0);
+                    ++error;
+               }
+               else if (l0 != l) { /* often true for out-of-date OS unicode */
+                    ++better;
+                    /* printf("%x != towlower(%x) == %x\n", l, c, l0); */
+               }
+               if (u0 != c && u0 != u) {
+                    fprintf(stderr, "MISMATCH %x != towupper(%x) == %x\n",
+                            u, c, u0);
+                    ++error;
+               }
+               else if (u0 != u) { /* often true for out-of-date OS unicode */
+                    ++better;
+                    /* printf("%x != towupper(%x) == %x\n", u, c, u0); */
+               }
+          }
+     }
+     check(!error, "utf8proc case conversion FAILED %d tests.", error);
+     printf("More up-to-date than OS unicode tables for %d tests.\n", better);
+     printf("utf8proc case conversion tests SUCCEEDED.\n");
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/case.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/charwidth.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/charwidth.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/charwidth.c (added)
+++ subversion/upstream/utf8proc/test/charwidth.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,71 @@
+#include "tests.h"
+#include <ctype.h>
+#include <wchar.h>
+
+static int my_isprint(int c) {
+     int cat = utf8proc_get_property(c)->category;
+     return (UTF8PROC_CATEGORY_LU <= cat && cat <= UTF8PROC_CATEGORY_ZS) ||
+          (c == 0x0601 || c == 0x0602 || c == 0x0603 || c == 0x06dd);
+}
+
+int main(int argc, char **argv)
+{
+     int c, error = 0, updates = 0;
+
+     (void) argc; /* unused */
+     (void) argv; /* unused */
+
+     /* some simple sanity tests of the character widths */
+     for (c = 0; c <= 0x110000; ++c) {
+          int cat = utf8proc_get_property(c)->category;
+          int w = utf8proc_charwidth(c);
+          if ((cat == UTF8PROC_CATEGORY_MN || cat == UTF8PROC_CATEGORY_ME) &&
+              w > 0) {
+               fprintf(stderr, "nonzero width %d for combining char %x\n", w, c);
+               error = 1;
+          }
+          if (w == 0 &&
+			  ((cat >= UTF8PROC_CATEGORY_LU && cat <= UTF8PROC_CATEGORY_LO) ||
+			   (cat >= UTF8PROC_CATEGORY_ND && cat <= UTF8PROC_CATEGORY_SC) ||
+			   (cat >= UTF8PROC_CATEGORY_SO && cat <= UTF8PROC_CATEGORY_ZS))) {
+               fprintf(stderr, "zero width for symbol-like char %x\n", c);
+               error = 1;
+          }
+          if (c <= 127 && ((!isprint(c) && w > 0) ||
+                           (isprint(c) && wcwidth(c) != w))) {
+               fprintf(stderr, "wcwidth %d mismatch %d for %s ASCII %x\n",
+                       wcwidth(c), w, 
+                       isprint(c) ? "printable" : "non-printable", c);
+               error = 1;
+          }
+          if (!my_isprint(c) && w > 0) {
+               fprintf(stderr, "non-printing %x had width %d\n", c, w);
+               error = 1;
+          }
+     }
+     check(!error, "utf8proc_charwidth FAILED tests.");
+
+     /* print some other information by compariing with system wcwidth */
+     printf("Mismatches with system wcwidth (not necessarily errors):\n");
+     for (c = 0; c <= 0x110000; ++c) {
+          int w = utf8proc_charwidth(c);
+          int wc = wcwidth(c);
+          if (sizeof(wchar_t) == 2 && c >= (1<<16)) continue;
+          /* lots of these errors for out-of-date system unicode tables */
+          if (wc == -1 && my_isprint(c) && w > 0) {
+			   updates += 1;
+#if 0
+               printf("  wcwidth(%x) = -1 for printable char\n", c);
+#endif
+		  }
+          if (wc == -1 && !my_isprint(c) && w > 0)
+               printf("  wcwidth(%x) = -1 for non-printable width-%d char\n", c, w);
+          if (wc >= 0 && wc != w)
+               printf("  wcwidth(%x) = %d != charwidth %d\n", c, wc, w);
+     }
+	 printf("   ... (positive widths for %d chars unknown to wcwidth) ...\n",
+			updates);
+     printf("Character-width tests SUCCEEDED.\n");
+
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/charwidth.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/custom.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/custom.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/custom.c (added)
+++ subversion/upstream/utf8proc/test/custom.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,27 @@
+#include "tests.h"
+
+static int thunk_test = 1;
+
+static utf8proc_int32_t custom(utf8proc_int32_t codepoint, void *thunk)
+{
+    check(((int *) thunk) == &thunk_test, "unexpected thunk passed");
+    if (codepoint == 'a')
+        return 'b';
+    if (codepoint == 'S')
+        return 0x00df; /* ß */
+    return codepoint;
+}
+
+int main(void)
+{
+    utf8proc_uint8_t input[] = {0x41,0x61,0x53,0x62,0xef,0xbd,0x81,0x00}; /* "AaSb\uff41" */
+    utf8proc_uint8_t correct[] = {0x61,0x62,0x73,0x73,0x62,0x61,0x00}; /* "abssba" */
+    utf8proc_uint8_t *output;
+    utf8proc_map_custom(input, 0, &output, UTF8PROC_CASEFOLD | UTF8PROC_COMPOSE | UTF8PROC_COMPAT | UTF8PROC_NULLTERM,
+                        custom, &thunk_test);
+    printf("mapped \"%s\" -> \"%s\"\n", (char*)input, (char*)output);
+    check(strlen((char*) output) == 6, "incorrect output length");
+    check(!memcmp(correct, output, 7), "incorrect output data");
+    free(output);
+    return 0;
+}

Propchange: subversion/upstream/utf8proc/test/custom.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/graphemetest.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/graphemetest.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/graphemetest.c (added)
+++ subversion/upstream/utf8proc/test/graphemetest.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,74 @@
+#include "tests.h"
+
+int main(int argc, char **argv)
+{
+    char *buf = NULL;
+    size_t bufsize = 0;
+    FILE *f = argc > 1 ? fopen(argv[1], "r") : NULL;
+    utf8proc_uint8_t src[1024];
+    int len;
+    
+    check(f != NULL, "error opening GraphemeBreakTest.txt");
+    while (getline(&buf, &bufsize, f) > 0) {
+        size_t bi = 0, si = 0;
+        lineno += 1;
+        
+        if (lineno % 100 == 0)
+            printf("checking line %zd...\n", lineno);
+        
+        if (buf[0] == '#') continue;
+        
+        while (buf[bi]) {
+            bi = skipspaces(buf, bi);
+            if (buf[bi] == '/') { /* grapheme break */
+                src[si++] = '/';
+                bi++;
+            }
+            else if (buf[bi] == '+') { /* no break */
+                bi++;
+            }
+            else if (buf[bi] == '#') { /* start of comments */
+                break;
+            }
+	    else { /* hex-encoded codepoint */
+                len = encode((char*) (src + si), buf + bi) - 1;
+                while (src[si]) ++si; /* advance to NUL termination */
+                bi += len;
+            }
+        }
+        if (si && src[si-1] == '/')
+            --si; /* no break after final grapheme */
+        src[si] = 0; /* NUL-terminate */
+        
+        if (si) {
+            utf8proc_uint8_t utf8[1024]; /* copy src without 0xff grapheme separators */
+            size_t i = 0, j = 0;
+            utf8proc_ssize_t glen;
+            utf8proc_uint8_t *g; /* utf8proc_map grapheme results */
+            while (i < si) {
+                if (src[i] != '/')
+                    utf8[j++] = src[i++];
+                else
+                    i++;
+            }
+            glen = utf8proc_map(utf8, j, &g, UTF8PROC_CHARBOUND);
+            if (glen == UTF8PROC_ERROR_INVALIDUTF8) {
+                 /* the test file contains surrogate codepoints, which are only for UTF-16 */
+                 printf("line %zd: ignoring invalid UTF-8 codepoints\n", lineno);
+            }
+            else {
+                 check(glen >= 0, "utf8proc_map error = %s",
+                       utf8proc_errmsg(glen));
+                 for (i = 0; i <= glen; ++i)
+                      if (g[i] == 0xff)
+                          g[i] = '/'; /* easier-to-read output (/ is not in test strings) */
+                 check(!strcmp((char*)g, (char*)src),
+                       "grapheme mismatch: \"%s\" instead of \"%s\"", (char*)g, (char*)src);
+            }
+            free(g);
+        }
+    }
+    fclose(f);
+    printf("Passed tests after %zd lines!\n", lineno);
+    return 0;
+}

Propchange: subversion/upstream/utf8proc/test/graphemetest.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/iterate.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/iterate.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/iterate.c (added)
+++ subversion/upstream/utf8proc/test/iterate.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,166 @@
+#include "tests.h"
+#include <ctype.h>
+#include <wchar.h>
+
+static  int     tests;
+static  int     error;
+
+#define CHECKVALID(pos, val, len) buf[pos] = val; testbytes(buf,len,len,__LINE__)
+#define CHECKINVALID(pos, val, len) buf[pos] = val; testbytes(buf,len,UTF8PROC_ERROR_INVALIDUTF8,__LINE__)
+
+static void testbytes(unsigned char *buf, int len, utf8proc_ssize_t retval, int line)
+{
+    utf8proc_int32_t out[16];
+    utf8proc_ssize_t ret;
+
+    /* Make a copy to ensure that memory is left uninitialized after "len"
+     * bytes. This way, Valgrind can detect overreads.
+     */
+    unsigned char tmp[16];
+    memcpy(tmp, buf, len);
+
+    tests++;
+    if ((ret = utf8proc_iterate(tmp, len, out)) != retval) {
+        fprintf(stderr, "Failed (%d):", line);
+        for (int i = 0; i < len ; i++) {
+            fprintf(stderr, " 0x%02x", tmp[i]);
+        }
+        fprintf(stderr, " -> %zd\n", ret);
+        error++;
+    }
+}
+
+int main(int argc, char **argv)
+{
+    uint32_t byt;
+    unsigned char buf[16];
+
+    tests = error = 0;
+
+    // Check valid sequences that were considered valid erroneously before
+    buf[0] = 0xef;
+    buf[1] = 0xb7;
+    for (byt = 0x90; byt < 0xa0; byt++) {
+        CHECKVALID(2, byt, 3);
+    }
+    // Check 0xfffe and 0xffff
+    buf[1] = 0xbf;
+    CHECKVALID(2, 0xbe, 3);
+    CHECKVALID(2, 0xbf, 3);
+    // Check 0x??fffe & 0x??ffff
+    for (byt = 0x1fffe; byt < 0x110000; byt += 0x10000) {
+        buf[0] = 0xf0 | (byt >> 18);
+        buf[1] = 0x80 | ((byt >> 12) & 0x3f);
+        CHECKVALID(3, 0xbe, 4);
+        CHECKVALID(3, 0xbf, 4);
+    }
+    
+    // Continuation byte not after lead
+    for (byt = 0x80; byt < 0xc0; byt++) {
+        CHECKINVALID(0, byt, 1);
+    }
+
+    // Continuation byte not after lead
+    for (byt = 0x80; byt < 0xc0; byt++) {
+        CHECKINVALID(0, byt, 1);
+    }
+
+    // Test lead bytes
+    for (byt = 0xc0; byt <= 0xff; byt++) {
+	// Single lead byte at end of string
+        CHECKINVALID(0, byt, 1);
+        // Lead followed by non-continuation character < 0x80
+        CHECKINVALID(1, 65, 2);
+	// Lead followed by non-continuation character > 0xbf
+        CHECKINVALID(1, 0xc0, 2);
+    }
+
+    // Test overlong 2-byte
+    buf[0] = 0xc0;
+    for (byt = 0x81; byt <= 0xbf; byt++) {
+        CHECKINVALID(1, byt, 2);
+    }
+    buf[0] = 0xc1;
+    for (byt = 0x80; byt <= 0xbf; byt++) {
+        CHECKINVALID(1, byt, 2);
+    }
+
+    // Test overlong 3-byte
+    buf[0] = 0xe0;
+    buf[2] = 0x80;
+    for (byt = 0x80; byt <= 0x9f; byt++) {
+        CHECKINVALID(1, byt, 3);
+    }
+
+    // Test overlong 4-byte
+    buf[0] = 0xf0;
+    buf[2] = 0x80;
+    buf[3] = 0x80;
+    for (byt = 0x80; byt <= 0x8f; byt++) {
+        CHECKINVALID(1, byt, 4);
+    }
+
+    // Test 4-byte > 0x10ffff
+    buf[0] = 0xf4;
+    buf[2] = 0x80;
+    buf[3] = 0x80;
+    for (byt = 0x90; byt <= 0xbf; byt++) {
+        CHECKINVALID(1, byt, 4);
+    }
+    buf[1] = 0x80;
+    for (byt = 0xf5; byt <= 0xf7; byt++) {
+        CHECKINVALID(0, byt, 4);
+    }
+
+    // Test 5-byte
+    buf[4] = 0x80;
+    for (byt = 0xf8; byt <= 0xfb; byt++) {
+        CHECKINVALID(0, byt, 5);
+    }
+
+    // Test 6-byte
+    buf[5] = 0x80;
+    for (byt = 0xfc; byt <= 0xfd; byt++) {
+        CHECKINVALID(0, byt, 6);
+    }
+
+    // Test 7-byte
+    buf[6] = 0x80;
+    CHECKINVALID(0, 0xfe, 7);
+
+    // Three and above byte sequences
+    for (byt = 0xe0; byt < 0xf0; byt++) {
+        // Lead followed by only 1 continuation byte
+        CHECKINVALID(0, byt, 2);
+        // Lead ended by non-continuation character < 0x80
+        CHECKINVALID(2, 65, 3);
+        // Lead ended by non-continuation character > 0xbf
+        CHECKINVALID(2, 0xc0, 3);
+    }
+
+    // 3-byte encoded surrogate character(s)
+    buf[0] = 0xed; buf[2] = 0x80;
+    // Single surrogate
+    CHECKINVALID(1, 0xa0, 3);
+    // Trailing surrogate first
+    CHECKINVALID(1, 0xb0, 3);
+
+    // Four byte sequences
+    buf[1] = 0x80;
+    for (byt = 0xf0; byt < 0xf5; byt++) {
+        // Lead followed by only 1 continuation bytes
+        CHECKINVALID(0, byt, 2);
+        // Lead followed by only 2 continuation bytes
+        CHECKINVALID(0, byt, 3);
+        // Lead followed by non-continuation character < 0x80
+        CHECKINVALID(3, 65, 4);
+        // Lead followed by non-continuation character > 0xbf
+        CHECKINVALID(3, 0xc0, 4);
+
+    }
+
+     check(!error, "utf8proc_iterate FAILED %d tests out of %d", error, tests);
+     printf("utf8proc_iterate tests SUCCEEDED, (%d) tests passed.\n", tests);
+
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/iterate.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/normtest.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/normtest.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/normtest.c (added)
+++ subversion/upstream/utf8proc/test/normtest.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,64 @@
+#include "tests.h"
+
+#define CHECK_NORM(NRM, norm, src) {                                 \
+    char *src_norm = (char*) utf8proc_ ## NRM((utf8proc_uint8_t*) src);      \
+    check(!strcmp(norm, src_norm),                                  \
+          "normalization failed for %s -> %s", src, norm);          \
+    free(src_norm);                                                 \
+}
+
+int main(int argc, char **argv)
+{
+     char *buf = NULL;
+     size_t bufsize = 0;
+     FILE *f = argc > 1 ? fopen(argv[1], "r") : NULL;
+     char source[1024], NFC[1024], NFD[1024], NFKC[1024], NFKD[1024];
+
+     check(f != NULL, "error opening NormalizationTest.txt");
+     while (getline(&buf, &bufsize, f) > 0) {
+          size_t offset;
+          lineno += 1;
+
+          if (buf[0] == '@') {
+               printf("line %zd: %s", lineno, buf + 1);
+               continue;
+          }
+          else if (lineno % 1000 == 0)
+               printf("checking line %zd...\n", lineno);
+
+          if (buf[0] == '#') continue;
+
+          offset = encode(source, buf);
+          offset += encode(NFC, buf + offset);
+          offset += encode(NFD, buf + offset);
+          offset += encode(NFKC, buf + offset);
+          offset += encode(NFKD, buf + offset);
+
+          CHECK_NORM(NFC, NFC, source);
+          CHECK_NORM(NFC, NFC, NFC);
+          CHECK_NORM(NFC, NFC, NFD);
+          CHECK_NORM(NFC, NFKC, NFKC);
+          CHECK_NORM(NFC, NFKC, NFKD);
+
+          CHECK_NORM(NFD, NFD, source);
+          CHECK_NORM(NFD, NFD, NFC);
+          CHECK_NORM(NFD, NFD, NFD);
+          CHECK_NORM(NFD, NFKD, NFKC);
+          CHECK_NORM(NFD, NFKD, NFKD);
+
+          CHECK_NORM(NFKC, NFKC, source);
+          CHECK_NORM(NFKC, NFKC, NFC);
+          CHECK_NORM(NFKC, NFKC, NFD);
+          CHECK_NORM(NFKC, NFKC, NFKC);
+          CHECK_NORM(NFKC, NFKC, NFKD);
+
+          CHECK_NORM(NFKD, NFKD, source);
+          CHECK_NORM(NFKD, NFKD, NFC);
+          CHECK_NORM(NFKD, NFKD, NFD);
+          CHECK_NORM(NFKD, NFKD, NFKC);
+          CHECK_NORM(NFKD, NFKD, NFKD);
+     }
+     fclose(f);
+     printf("Passed tests after %zd lines!\n", lineno);
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/normtest.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/printproperty.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/printproperty.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/printproperty.c (added)
+++ subversion/upstream/utf8proc/test/printproperty.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,49 @@
+/* simple test program to print out the utf8proc properties for a codepoint */
+
+#include "tests.h"
+
+int main(int argc, char **argv)
+{
+     int i;
+
+     for (i = 1; i < argc; ++i) {
+          unsigned int c;
+          if (!strcmp(argv[i], "-V")) {
+               printf("utf8proc version %s\n", utf8proc_version());
+               continue;
+          }
+          check(sscanf(argv[i],"%x",&c) == 1, "invalid hex input %s", argv[i]);
+          const utf8proc_property_t *p = utf8proc_get_property(c);
+          printf("U+%s:\n"
+                 "  category = %s\n"
+                 "  combining_class = %d\n"
+                 "  bidi_class = %d\n"
+                 "  decomp_type = %d\n"
+                 "  uppercase_mapping = %x\n"
+                 "  lowercase_mapping = %x\n"
+                 "  titlecase_mapping = %x\n"
+                 "  comb_index = %d\n"
+                 "  bidi_mirrored = %d\n"
+                 "  comp_exclusion = %d\n"
+                 "  ignorable = %d\n"
+                 "  control_boundary = %d\n"
+                 "  boundclass = %d\n"
+                 "  charwidth = %d\n",
+                 argv[i],
+                 utf8proc_category_string(c),
+                 p->combining_class,
+                 p->bidi_class,
+                 p->decomp_type,
+                 utf8proc_toupper(c),
+                 utf8proc_tolower(c),
+                 utf8proc_totitle(c),
+                 p->comb_index,
+                 p->bidi_mirrored,
+                 p->comp_exclusion,
+                 p->ignorable,
+                 p->control_boundary,
+                 p->boundclass,
+                 utf8proc_charwidth(c));
+     }
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/printproperty.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/tests.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/tests.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/tests.c (added)
+++ subversion/upstream/utf8proc/test/tests.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,46 @@
+/* Common functions for our test programs. */
+
+#include "tests.h"
+
+size_t lineno = 0;
+
+void check(int cond, const char *format, ...)
+{
+     if (!cond) {
+          va_list args;
+          fprintf(stderr, "line %zd: ", lineno);
+          va_start(args, format);
+          vfprintf(stderr, format, args);
+          va_end(args);
+          fprintf(stderr, "\n");
+          exit(1);
+     }
+}
+
+size_t skipspaces(const char *buf, size_t i)
+{
+    while (isspace(buf[i])) ++i;
+    return i;
+}
+
+/* if buf points to a sequence of codepoints encoded as hexadecimal strings,
+   separated by whitespace, and terminated by any character not in
+   [0-9a-fA-F] or whitespace, then stores the corresponding utf8 string
+   in dest, returning the number of bytes read from buf */
+size_t encode(char *dest, const char *buf)
+{
+     size_t i = 0, j, d = 0;
+     for (;;) {
+          int c;
+          i = skipspaces(buf, i);
+          for (j=i; buf[j] && strchr("0123456789abcdef", tolower(buf[j])); ++j)
+               ; /* find end of hex input */
+          if (j == i) { /* no codepoint found */
+               dest[d] = 0; /* NUL-terminate destination string */
+               return i + 1;
+          }
+          check(sscanf(buf + i, "%x", (unsigned int *)&c) == 1, "invalid hex input %s", buf+i);
+          i = j; /* skip to char after hex input */
+          d += utf8proc_encode_char(c, (utf8proc_uint8_t *) (dest + d));
+     }
+}

Propchange: subversion/upstream/utf8proc/test/tests.c
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/tests.h
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/tests.h?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/tests.h (added)
+++ subversion/upstream/utf8proc/test/tests.h Wed Sep 20 20:55:39 2017
@@ -0,0 +1,23 @@
+/* Common functions and includes for our test programs. */
+
+/*
+ * Set feature macro to enable getline() and wcwidth().
+ *
+ * Please refer to section 2.2.1 of POSIX.1-2008:
+ * http://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_02_01_02
+ */
+#define _XOPEN_SOURCE 700
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <ctype.h>
+#include <string.h>
+#include <stdarg.h>
+
+#include "../utf8proc.h"
+
+extern size_t lineno;
+
+void check(int cond, const char *format, ...);
+size_t skipspaces(const char *buf, size_t i);
+size_t encode(char *dest, const char *buf);

Propchange: subversion/upstream/utf8proc/test/tests.h
------------------------------------------------------------------------------
    svn:eol-style = native

Added: subversion/upstream/utf8proc/test/valid.c
URL: http://svn.apache.org/viewvc/subversion/upstream/utf8proc/test/valid.c?rev=1809081&view=auto
==============================================================================
--- subversion/upstream/utf8proc/test/valid.c (added)
+++ subversion/upstream/utf8proc/test/valid.c Wed Sep 20 20:55:39 2017
@@ -0,0 +1,41 @@
+#include "tests.h"
+#include <ctype.h>
+#include <wchar.h>
+
+int main(int argc, char **argv)
+{
+     int c, error = 0;
+
+     (void) argc; /* unused */
+     (void) argv; /* unused */
+
+     /* some simple sanity tests of  */
+     for (c = 0; c < 0xd800; c++) {
+        if (!utf8proc_codepoint_valid(c)) {
+           fprintf(stderr, "Failed: codepoint_valid(%04x) -> false\n", c);
+           error++;
+        }
+     }
+     for (;c < 0xe000; c++) {
+        if (utf8proc_codepoint_valid(c)) {
+           fprintf(stderr, "Failed: codepoint_valid(%04x) -> true\n", c);
+           error++;
+        }
+     }
+     for (;c < 0x110000; c++) {
+        if (!utf8proc_codepoint_valid(c)) {
+           fprintf(stderr, "Failed: codepoint_valid(%06x) -> false\n", c);
+           error++;
+        }
+     }
+     for (;c < 0x110010; c++) {
+        if (utf8proc_codepoint_valid(c)) {
+           fprintf(stderr, "Failed: codepoint_valid(%06x) -> true\n", c);
+           error++;
+        }
+     }
+     check(!error, "utf8proc_codepoint_valid FAILED %d tests.", error);
+     printf("Validity tests SUCCEEDED.\n");
+
+     return 0;
+}

Propchange: subversion/upstream/utf8proc/test/valid.c
------------------------------------------------------------------------------
    svn:eol-style = native



Mime
View raw message