Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EF0E56B98 for ; Thu, 16 Jun 2011 19:35:44 +0000 (UTC) Received: (qmail 9090 invoked by uid 500); 16 Jun 2011 19:35:43 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 9039 invoked by uid 500); 16 Jun 2011 19:35:43 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 9032 invoked by uid 99); 16 Jun 2011 19:35:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:35:43 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [85.25.71.29] (HELO mail.troja.net) (85.25.71.29) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Jun 2011 19:35:35 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.troja.net (Postfix) with ESMTP id ED1FED36002 for ; Thu, 16 Jun 2011 21:35:14 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at mail.troja.net Received: from mail.troja.net ([127.0.0.1]) by localhost (megaira.troja.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id J6NEpVx3PUCg for ; Thu, 16 Jun 2011 21:35:09 +0200 (CEST) Received: from VEGA (port-92-196-130-188.dynamic.qsc.de [92.196.130.188]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.troja.net (Postfix) with ESMTPSA id 15A6FD36001 for ; Thu, 16 Jun 2011 21:35:08 +0200 (CEST) From: "Uwe Schindler" To: Subject: RE: svn commit: r1135956 - in /lucene/dev/branches/branch_3x: ./ lucene/ lucene/backwards/ lucene/backwards/src/test-framework/ lucene/backwards/src/test/ solr/ solr/contrib/dataimporthandler/ solr/contrib/dataimporthandler/src/main/java/org/apache/solr/h Date: Thu, 16 Jun 2011 21:35:17 +0200 Message-ID: <009601cc2c5c$864194c0$92c4be40$@thetaphi.de> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQL6CskGkRb22j/vzJOf3sf4nrOYo5Jkoz9Q Content-Language: de X-Virus-Checked: Checked by ClamAV on apache.org Shalin, i had to comment out your test because the finally block does not = compile with Java 5 (Solr 3.1), Jenkins is down at the moment, so did = not catch earlier. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de > -----Original Message----- > From: shalin@apache.org [mailto:shalin@apache.org] > Sent: Wednesday, June 15, 2011 10:36 AM > To: commits@lucene.apache.org > Subject: svn commit: r1135956 - in /lucene/dev/branches/branch_3x: ./ > lucene/ lucene/backwards/ lucene/backwards/src/test-framework/ > lucene/backwards/src/test/ solr/ solr/contrib/dataimporthandler/ > solr/contrib/dataimporthandler/src/main/java/org/apache/solr/ha... >=20 > Author: shalin > Date: Wed Jun 15 08:36:06 2011 > New Revision: 1135956 >=20 > URL: http://svn.apache.org/viewvc?rev=3D1135956&view=3Drev > Log: > SOLR-2551 -- Check dataimport.properties for write access (if = delta-import is > supported in DIH configuration) before starting an import >=20 > Modified: > lucene/dev/branches/branch_3x/ (props changed) > lucene/dev/branches/branch_3x/lucene/ (props changed) > lucene/dev/branches/branch_3x/lucene/backwards/ (props changed) > lucene/dev/branches/branch_3x/lucene/backwards/src/test/ (props > changed) > lucene/dev/branches/branch_3x/lucene/backwards/src/test-framework/ > (props changed) > lucene/dev/branches/branch_3x/solr/ (props changed) >=20 > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/CHANGES. > txt >=20 > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/DataImporter.java >=20 > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/SolrWriter.java >=20 > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/test/ja > va/org/apache/solr/handler/dataimport/TestSqlEntityProcessorDelta.java >=20 > Modified: > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/CHANGES. > txt > URL: > = http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/solr/contrib > = /dataimporthandler/CHANGES.txt?rev=3D1135956&r1=3D1135955&r2=3D1135956&vi= > ew=3Ddiff > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/CHANGES. > txt (original) > +++ > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/CHANGES > +++ .txt Wed Jun 15 08:36:06 2011 > @@ -11,7 +11,8 @@ $Id$ >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 3.3.0-dev = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 > -(No Changes) > +* SOLR-2551: Check dataimport.properties for write access (if > +delta-import is supported > + in DIH configuration) before starting an import (C S, shalin) >=20 > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 3.2.0 = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >=20 >=20 > Modified: > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/DataImporter.java > URL: > = http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/solr/contrib > = /dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/Dat > aImporter.java?rev=3D1135956&r1=3D1135955&r2=3D1135956&view=3Ddiff > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/DataImporter.java (original) > +++ > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/mai > +++ n/java/org/apache/solr/handler/dataimport/DataImporter.java Wed > Jun > +++ 15 08:36:06 2011 > @@ -39,6 +39,7 @@ import org.apache.commons.io.IOUtils; >=20 > import javax.xml.parsers.DocumentBuilder; > import javax.xml.parsers.DocumentBuilderFactory; > +import java.io.File; > import java.io.StringReader; > import java.text.SimpleDateFormat; > import java.util.*; > @@ -85,6 +86,8 @@ public class DataImporter { >=20 > private final Map coreScopeSession; >=20 > + private boolean isDeltaImportSupported =3D false; > + > /** > * Only for testing purposes > */ > @@ -113,7 +116,9 @@ public class DataImporter { > initEntity(e, fields, false); > verifyWithSchema(fields); > identifyPk(e); > - } > + if = (e.allAttributes.containsKey(SqlEntityProcessor.DELTA_QUERY)) > + isDeltaImportSupported =3D true; > + } > } >=20 > private void verifyWithSchema(Map fields) = { @@ > -350,6 +355,7 @@ public class DataImporter { >=20 > try { > docBuilder =3D new DocBuilder(this, writer, requestParams); > + checkWritablePersistFile(writer); > docBuilder.execute(); > if (!requestParams.debug) > cumulativeStatistics.add(docBuilder.importStatistics); > @@ -364,6 +370,15 @@ public class DataImporter { >=20 > } >=20 > + private void checkWritablePersistFile(SolrWriter writer) { > + File persistFile =3D writer.getPersistFile(); > + boolean isWritable =3D persistFile.exists() ? = persistFile.canWrite() : > persistFile.getParentFile().canWrite(); > + if (isDeltaImportSupported && !isWritable) { > + throw new DataImportHandlerException(SEVERE, > persistFile.getAbsolutePath() + > + " is not writable. Delta imports are supported by data = config but will > not work."); > + } > + } > + > public void doDeltaImport(SolrWriter writer, RequestParams > requestParams) { > LOG.info("Starting Delta Import"); > setStatus(Status.RUNNING_DELTA_DUMP); > @@ -371,6 +386,7 @@ public class DataImporter { > try { > setIndexStartTime(new Date()); > docBuilder =3D new DocBuilder(this, writer, requestParams); > + checkWritablePersistFile(writer); > docBuilder.execute(); > if (!requestParams.debug) > cumulativeStatistics.add(docBuilder.importStatistics); >=20 > Modified: > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/SolrWriter.java > URL: > = http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/solr/contrib > = /dataimporthandler/src/main/java/org/apache/solr/handler/dataimport/Sol > rWriter.java?rev=3D1135956&r1=3D1135955&r2=3D1135956&view=3Ddiff > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/main/j > ava/org/apache/solr/handler/dataimport/SolrWriter.java (original) > +++ > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/mai > +++ n/java/org/apache/solr/handler/dataimport/SolrWriter.java Wed Jun = 15 > +++ 08:36:06 2011 > @@ -100,13 +100,10 @@ public class SolrWriter { >=20 > try { > props.putAll(p); > - String filePath =3D configDir; > - if (configDir !=3D null && !configDir.endsWith(File.separator)) > - filePath +=3D File.separator; > - filePath +=3D persistFilename; > - propOutput =3D new FileOutputStream(filePath); > + File persistFile =3D getPersistFile(); > + propOutput =3D new FileOutputStream(persistFile); > props.store(propOutput, null); > - log.info("Wrote last indexed time to " + persistFilename); > + log.info("Wrote last indexed time to " + > + persistFile.getAbsolutePath()); > } catch (FileNotFoundException e) { > throw new > DataImportHandlerException(DataImportHandlerException.SEVERE, > "Unable to persist Index Start Time", e); @@ -123,6 = +120,14 @@ > public class SolrWriter { > } > } >=20 > + File getPersistFile() { > + String filePath =3D configDir; > + if (configDir !=3D null && !configDir.endsWith(File.separator)) > + filePath +=3D File.separator; > + filePath +=3D persistFilename; > + return new File(filePath); > + } > + > void finish() { > try { > processor.finish(); >=20 > Modified: > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/test/ja > va/org/apache/solr/handler/dataimport/TestSqlEntityProcessorDelta.java > URL: > = http://svn.apache.org/viewvc/lucene/dev/branches/branch_3x/solr/contrib > = /dataimporthandler/src/test/java/org/apache/solr/handler/dataimport/Test > = SqlEntityProcessorDelta.java?rev=3D1135956&r1=3D1135955&r2=3D1135956&view= =3D > diff > = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- > = lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/test/ja > va/org/apache/solr/handler/dataimport/TestSqlEntityProcessorDelta.java > (original) > +++ > lucene/dev/branches/branch_3x/solr/contrib/dataimporthandler/src/tes > +++ = t/java/org/apache/solr/handler/dataimport/TestSqlEntityProcessorDelt > +++ a.java Wed Jun 15 08:36:06 2011 > @@ -20,6 +20,8 @@ import org.junit.Before; import = org.junit.BeforeClass; > import org.junit.Test; >=20 > +import java.io.File; > +import java.io.FileOutputStream; > import java.util.ArrayList; > import java.util.Collections; > import java.util.List; > @@ -92,7 +94,37 @@ public class TestSqlEntityProcessorDelta > public void testCompositePk_FullImport() throws Exception { > add1document(); > } > - > + > + @Test > + @SuppressWarnings("unchecked") > + public void testNonWritablePersistFile() throws Exception { > + // See SOLR-2551 > + String configDir =3D = h.getCore().getResourceLoader().getConfigDir(); > + String filePath =3D configDir; > + if (configDir !=3D null && !configDir.endsWith(File.separator)) > + filePath +=3D File.separator; > + filePath +=3D "dataimport.properties"; > + File f =3D new File(filePath); > + // execute the test only if we are able to set file to read only = mode > + if ((f.exists() || f.createNewFile()) && f.setReadOnly()) { > + try { > + List parentRow =3D new ArrayList(); > + parentRow.add(createMap("id", "1")); > + MockDataSource.setIterator(FULLIMPORT_QUERY, > + parentRow.iterator()); > + > + List childRow =3D new ArrayList(); > + childRow.add(createMap("desc", "hello")); > + MockDataSource.setIterator("select * from y where y.A=3D'1'", = childRow > + .iterator()); > + > + runFullImport(dataConfig_delta); > + assertQ(req("id:1"), "//*[@numFound=3D'0']"); > + } finally { > + f.setWritable(true); > + } > + } > + } > + > // WORKS >=20 > @Test >=20 --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org