Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D1323200D15 for ; Thu, 5 Oct 2017 14:07:24 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CF8CE160BDA; Thu, 5 Oct 2017 12:07:24 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C7CBC1609DA for ; Thu, 5 Oct 2017 14:07:23 +0200 (CEST) Received: (qmail 60920 invoked by uid 500); 5 Oct 2017 11:56:58 -0000 Mailing-List: contact commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list commits@lucene.apache.org Received: (qmail 60809 invoked by uid 99); 5 Oct 2017 11:56:57 -0000 Received: from Unknown (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 05 Oct 2017 11:56:57 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 148D1F5C8F; Thu, 5 Oct 2017 11:56:23 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: ab@apache.org To: commits@lucene.apache.org Date: Thu, 05 Oct 2017 11:56:30 -0000 Message-Id: <01ee7a31835c4c159df1f43dfddb3119@git.apache.org> In-Reply-To: <37920e774e8342a9839dc9955c9e23a7@git.apache.org> References: <37920e774e8342a9839dc9955c9e23a7@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [09/50] lucene-solr:feature/autoscaling_72: addBackcompatIndexes.py: Create 'sorted' indexes; on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0 indexes. archived-at: Thu, 05 Oct 2017 12:07:25 -0000 addBackcompatIndexes.py: Create 'sorted' indexes; on major releases, produce moreterms, dvupdates, and empty indexes. Add 7.0.0 indexes. Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/5afd41de Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/5afd41de Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/5afd41de Branch: refs/heads/feature/autoscaling_72 Commit: 5afd41de114ff62a755634c4a49091f88bde22cd Parents: dc47c6e Author: Steve Rowe Authored: Sat Sep 30 20:44:01 2017 -0400 Committer: Steve Rowe Committed: Sat Sep 30 20:44:01 2017 -0400 ---------------------------------------------------------------------- dev-tools/scripts/addBackcompatIndexes.py | 67 ++++++++++++++----- .../index/TestBackwardsCompatibility.java | 38 ++++++++--- .../org/apache/lucene/index/dvupdates.7.0.0.zip | Bin 0 -> 13401 bytes .../org/apache/lucene/index/empty.7.0.0.zip | Bin 0 -> 375 bytes .../org/apache/lucene/index/index.7.0.0-cfs.zip | Bin 0 -> 15594 bytes .../apache/lucene/index/index.7.0.0-nocfs.zip | Bin 0 -> 15586 bytes .../org/apache/lucene/index/moreterms.7.0.0.zip | Bin 0 -> 175822 bytes .../org/apache/lucene/index/sorted.7.0.0.zip | Bin 0 -> 74558 bytes 8 files changed, 79 insertions(+), 26 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/dev-tools/scripts/addBackcompatIndexes.py ---------------------------------------------------------------------- diff --git a/dev-tools/scripts/addBackcompatIndexes.py b/dev-tools/scripts/addBackcompatIndexes.py index 543fad6..1cada01 100644 --- a/dev-tools/scripts/addBackcompatIndexes.py +++ b/dev-tools/scripts/addBackcompatIndexes.py @@ -31,12 +31,24 @@ import re import shutil def create_and_add_index(source, indextype, index_version, current_version, temp_dir): + if not current_version.is_back_compat_with(index_version): + prefix = 'unsupported' + else: + prefix = { + 'cfs': 'index', + 'nocfs': 'index', + 'sorted': 'sorted', + 'moreterms': 'moreterms', + 'dvupdates': 'dvupdates', + 'emptyIndex': 'empty' + }[indextype] if indextype in ('cfs', 'nocfs'): dirname = 'index.%s' % indextype + filename = '%s.%s-%s.zip' % (prefix, index_version, indextype) else: dirname = indextype - prefix = 'index' if current_version.is_back_compat_with(index_version) else 'unsupported' - filename = '%s.%s-%s.zip' % (prefix, index_version, indextype) + filename = '%s.%s.zip' % (prefix, index_version) + print(' creating %s...' % filename, end='', flush=True) module = 'backward-codecs' index_dir = os.path.join('lucene', module, 'src/test/org/apache/lucene/index') @@ -47,7 +59,11 @@ def create_and_add_index(source, indextype, index_version, current_version, temp test = { 'cfs': 'testCreateCFS', - 'nocfs': 'testCreateNoCFS' + 'nocfs': 'testCreateNoCFS', + 'sorted': 'testCreateSortedIndex', + 'moreterms': 'testCreateMoreTermsIndex', + 'dvupdates': 'testCreateIndexWithDocValuesUpdates', + 'emptyIndex': 'testCreateEmptyIndex' }[indextype] ant_args = ' '.join([ '-Dtests.bwcdir=%s' % temp_dir, @@ -83,16 +99,22 @@ def create_and_add_index(source, indextype, index_version, current_version, temp print('done') def update_backcompat_tests(types, index_version, current_version): - print(' adding new indexes to backcompat tests...', end='', flush=True) + print(' adding new indexes %s to backcompat tests...' % types, end='', flush=True) module = 'lucene/backward-codecs' filename = '%s/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java' % module - matcher = re.compile(r'final static String\[\] oldNames = {|};' if current_version.is_back_compat_with(index_version) - else r'final String\[\] unsupportedNames = {|};') + if not current_version.is_back_compat_with(index_version): + matcher = re.compile(r'final String\[\] unsupportedNames = {|};'), + elif 'sorted' in types: + matcher = re.compile(r'final static String\[\] oldSortedNames = {|};') + else: + matcher = re.compile(r'final static String\[\] oldNames = {|};') + + strip_dash_suffix_re = re.compile(r'-.*') def find_version(x): x = x.strip() - end = x.index("-") - return scriptutil.Version.parse(x[1:end]) + x = re.sub(strip_dash_suffix_re, '', x) # remove the -suffix if any + return scriptutil.Version.parse(x) class Edit(object): start = None @@ -100,10 +122,12 @@ def update_backcompat_tests(types, index_version, current_version): if self.start: # find where this version should exist i = len(buffer) - 1 - v = find_version(buffer[i]) - while i >= self.start and v.on_or_after(index_version): - i -= 1 + previous_version_exists = '};' not in line # Version list closure => there are no versions + if previous_version_exists: # Only look if there is a version here v = find_version(buffer[i]) + while i >= self.start and v.on_or_after(index_version): + i -= 1 + v = find_version(buffer[i]) i += 1 # readjust since we skipped past by 1 # unfortunately python doesn't have a range remove from list... @@ -111,14 +135,20 @@ def update_backcompat_tests(types, index_version, current_version): while i < len(buffer) and index_version.on_or_after(find_version(buffer[i])): buffer.pop(i) - if i == len(buffer) and not buffer[-1].strip().endswith(","): + if i == len(buffer) and previous_version_exists and not buffer[-1].strip().endswith(","): # add comma buffer[-1] = buffer[-1].rstrip() + ",\n" - last = buffer[-1] - spaces = ' ' * (len(last) - len(last.lstrip())) + if previous_version_exists: + last = buffer[-1] + spaces = ' ' * (len(last) - len(last.lstrip())) + else: + spaces = ' ' for (j, t) in enumerate(types): - newline = spaces + ('"%s-%s"' % (index_version, t)) + if t == 'sorted': + newline = spaces + ('"sorted.%s"') % index_version + else: + newline = spaces + ('"%s-%s"' % (index_version, t)) if j < len(types) - 1 or i < len(buffer): newline += ',' buffer.insert(i, newline + '\n') @@ -215,9 +245,16 @@ def main(): current_version = scriptutil.Version.parse(scriptutil.find_current_version()) create_and_add_index(source, 'cfs', c.version, current_version, c.temp_dir) create_and_add_index(source, 'nocfs', c.version, current_version, c.temp_dir) + create_and_add_index(source, 'sorted', c.version, current_version, c.temp_dir) + if c.version.minor == 0 and c.version.bugfix == 0: + create_and_add_index(source, 'moreterms', c.version, current_version, c.temp_dir) + create_and_add_index(source, 'dvupdates', c.version, current_version, c.temp_dir) + create_and_add_index(source, 'emptyIndex', c.version, current_version, c.temp_dir) + print ('\nMANUAL UPDATE REQUIRED: edit TestBackwardsCompatibility to enable moreterms, dvupdates, and empty index testing') print('\nAdding backwards compatibility tests') update_backcompat_tests(['cfs', 'nocfs'], c.version, current_version) + update_backcompat_tests(['sorted'], c.version, current_version) print('\nTesting changes') check_backcompat_tests() http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java index c202973..88900d4 100644 --- a/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java +++ b/lucene/backward-codecs/src/test/org/apache/lucene/index/TestBackwardsCompatibility.java @@ -156,7 +156,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase { IndexWriterConfig conf = new IndexWriterConfig(analyzer) .setMergePolicy(mp).setUseCompoundFile(false); IndexWriter writer = new IndexWriter(dir, conf); - LineFileDocs docs = new LineFileDocs(null); + LineFileDocs docs = new LineFileDocs(random()); for(int i=0;i<50;i++) { writer.addDocument(docs.nextDoc()); } @@ -275,12 +275,33 @@ public class TestBackwardsCompatibility extends LuceneTestCase { dir.close(); } + public void testCreateEmptyIndex() throws Exception { + Path indexDir = getIndexDir().resolve("emptyIndex"); + Files.deleteIfExists(indexDir); + IndexWriterConfig conf = new IndexWriterConfig(new MockAnalyzer(random())) + .setUseCompoundFile(false).setMergePolicy(NoMergePolicy.INSTANCE); + try (Directory dir = newFSDirectory(indexDir); + IndexWriter writer = new IndexWriter(dir, conf)) { + writer.flush(); + } + } + final static String[] oldNames = { + "7.0.0-cfs", + "7.0.0-nocfs" }; public static String[] getOldNames() { return oldNames; } + + final static String[] oldSortedNames = { + "sorted.7.0.0" + }; + + public static String[] getOldSortedNames() { + return oldSortedNames; + } final String[] unsupportedNames = { "1.9.0-cfs", @@ -765,7 +786,7 @@ public class TestBackwardsCompatibility extends LuceneTestCase { Directory targetDir2 = newDirectory(); IndexWriter w = new IndexWriter(targetDir2, newIndexWriterConfig(new MockAnalyzer(random()))); IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> TestUtil.addIndexesSlowly(w, reader)); - assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 6 into this index which has been created by major version 7"); + assertEquals(e.getMessage(), "Cannot merge a segment that has been created with major version 7 into this index which has been created by major version 8"); w.close(); targetDir2.close(); @@ -1449,14 +1470,13 @@ public class TestBackwardsCompatibility extends LuceneTestCase { public static final String emptyIndex = "empty.7.0.0.zip"; public void testUpgradeEmptyOldIndex() throws Exception { - assumeTrue("Reenable when 7.0 is released", false); Path oldIndexDir = createTempDir("emptyIndex"); TestUtil.unzip(getDataInputStream(emptyIndex), oldIndexDir); Directory dir = newFSDirectory(oldIndexDir); newIndexUpgrader(dir).upgrade(); - checkAllSegmentsUpgraded(dir, 6); + checkAllSegmentsUpgraded(dir, 7); dir.close(); } @@ -1464,7 +1484,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase { public static final String moreTermsIndex = "moreterms.7.0.0.zip"; public void testMoreTerms() throws Exception { - assumeTrue("Reenable when 7.0 is released", false); Path oldIndexDir = createTempDir("moreterms"); TestUtil.unzip(getDataInputStream(moreTermsIndex), oldIndexDir); Directory dir = newFSDirectory(oldIndexDir); @@ -1509,7 +1528,6 @@ public class TestBackwardsCompatibility extends LuceneTestCase { } public void testDocValuesUpdates() throws Exception { - assumeTrue("Reenable when 7.0 is released", false); Path oldIndexDir = createTempDir("dvupdates"); TestUtil.unzip(getDataInputStream(dvUpdatesIndex), oldIndexDir); Directory dir = newFSDirectory(oldIndexDir); @@ -1572,12 +1590,10 @@ public class TestBackwardsCompatibility extends LuceneTestCase { } public void testSortedIndex() throws Exception { - assumeTrue("Reenable when 7.0 is released", false); - String[] versions = new String[] {}; - for(String version : versions) { + for(String name : oldSortedNames) { Path path = createTempDir("sorted"); - InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream("sorted." + version + ".zip"); - assertNotNull("Sorted index index " + version + " not found", resource); + InputStream resource = TestBackwardsCompatibility.class.getResourceAsStream(name + ".zip"); + assertNotNull("Sorted index index " + name + " not found", resource); TestUtil.unzip(resource, path); // TODO: more tests http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip new file mode 100644 index 0000000..6c94eae Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/dvupdates.7.0.0.zip differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip new file mode 100644 index 0000000..d8dac24 Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/empty.7.0.0.zip differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip new file mode 100644 index 0000000..1419931 Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-cfs.zip differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip new file mode 100644 index 0000000..83c3242 Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/index.7.0.0-nocfs.zip differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip new file mode 100644 index 0000000..5366c09 Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/moreterms.7.0.0.zip differ http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/5afd41de/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip ---------------------------------------------------------------------- diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip new file mode 100644 index 0000000..13948dc Binary files /dev/null and b/lucene/backward-codecs/src/test/org/apache/lucene/index/sorted.7.0.0.zip differ