lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Don Bosco Durai <bo...@apache.org>
Subject Re: Encryption to Solr indexes – Using Custom Codec
Date Mon, 20 Jun 2016 17:09:51 GMT
Mohit, just curious. Have you considered just encrypting the volume where there index is stored?
It will be done at the OS level and performance wise it will be much better and easier to
manage. Would it address your requirement? Or is it purely academic exercise for you.

Bosco


From:  "Sidana, Mohit" <Mohit.Sidana@wolterskluwer.com>
Reply-To:  <solr-user@lucene.apache.org>
Date:  Monday, June 20, 2016 at 7:42 AM
To:  "solr-user@lucene.apache.org" <solr-user@lucene.apache.org>
Subject:   Encryption to Solr indexes – Using Custom Codec

Hello,

As Part of my studies I am exploring the solutions which can be used for Lucene/Solr Index
encryption.

I found the patch open on Apache JIRA - Codec for index-level encryption (LUCENE-6966). https://issues.apache.org/jira/browse/LUCENE-6966
and I am currently trying to test this Custom codec with Solr to perform secure search over
some sensitive records. 

I've decided to follow the path described in Solr wiki, setting up Simple Text Codec and further
tried to use Encrypted codec Source. 

Here are the additional details.

I've created a basic jar file out of this source code (Build it as a jar from Eclipse using
Maven Plugin).

The Solr installation I'm using to test this is the Solr 6.0.0 unzipped, and started via its
embedded Jetty server and using the single core.

I've placed my jar with the codec in [My_Core\ instance Dir.]\ Lib

In:

[$SolrDir]\Solr\ My_Core \conf\solrconfig.xml

I've added the following lines:

<lib dir="./lib/" regex=".*\.jar"/> 

<codecFactory class="solr.SchemaCodecFactory"/>

Then in the schema.xml file, I've declared some field and field Types that should use this
codec:

<field name="City" type="strings"/>

  <field name="Country" type="strings"/>

  <field name="Name" type="strings"/>

  <field name="Payment" type="strings"/>

  <field name="Price" type="tlongs"/>

  <field name="Product" type="strings"/>

  <field name="State" type="strings"/>

 

<!-- set the per field posting format as below -->

<fieldType name="strings" class="solr.StrField" sortMissingLast="true" docValues="true"
multiValued="true" postingsFormat="EncryptedLucene50"/>

<fieldType name="tlongs" class="solr.TrieLongField" positionIncrementGap="0" docValues="true"
multiValued="true" precisionStep="8" postingsFormat="EncryptedLucene50"/>

 

I'm pretty sure I've followed all the steps described in Solr Wiki; however, when I actually
try to use custom codec implementation (named "Encrypted Codec") to index some sample CSV
data using simple post tool 

java -Dtype=text/csv -Durl=http://localhost:8983/solr/My_Core /update -jar  post.jar  Sales.csv

and I have also tried doing the same with SolrJ but I have faced the same error.

SolrClient server = new HttpSolrClient("http://localhost:8983/solr/My_Core ");

              SolrInputDocument doc = new SolrInputDocument();

 

              doc.addField("id", "1234");

              doc.addField("name", "A lovely summer holiday");

 

              try {

                    server.add(doc);

                    server.commit();

                     System.out.println("Document added!");

              } catch (SolrServerException | IOException e) {

                    e.printStackTrace();

              }

 

       }

 

}

I get the attached errors in Solr log.

org.apache.solr.common.SolrException: Exception writing document id b3e01ada-d0f1-4ddf-ad6a-2828bfe619a3
to the index; possible analysis error.

                at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:181)

                at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:68)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.AddSchemaFieldsUpdateProcessorFactory$AddSchemaFieldsUpdateProcessor.processAdd(AddSchemaFieldsUpdateProcessorFactory.java:335)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldNameMutatingUpdateProcessorFactory$1.processAdd(FieldNameMutatingUpdateProcessorFactory.java:74)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.FieldMutatingUpdateProcessor.processAdd(FieldMutatingUpdateProcessor.java:117)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:936)

                at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1091)

                at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:714)

                at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)

                at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:48)

                at org.apache.solr.update.processor.AbstractDefaultValueUpdateProcessorFactory$DefaultValueUpdateProcessor.processAdd(AbstractDefaultValueUpdateProcessorFactory.java:93)

                at org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:391)

                at org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:43)

                at org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:360)

                at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:30)

                at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:94)

                at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)

                at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)

                at org.apache.solr.core.SolrCore.execute(SolrCore.java:2033)

                at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)

                at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)

                at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)

                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)

                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)

                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)

                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)

                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

                at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)

                at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)

                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

                at org.eclipse.jetty.server.Server.handle(Server.java:518)

                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)

                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)

                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)

                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)

                at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)

                at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)

                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)

                at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)

                at java.lang.Thread.run(Thread.java:745)

 

Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed

                at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)

                at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)

                at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1488)

                at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:282)

                at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:214)

                at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:169)

                ... 58 more

 

Caused by: java.lang.NoClassDefFoundError: org/apache/lucene/codecs/encrypted/DummyCipherFactory

                at org.apache.lucene.codecs.encrypted.DummyEncryptedLucene60Codec$DummyEncryptedLucene50PostingsFormat.getCipherFactory(DummyEncryptedLucene60Codec.java:67)

                at org.apache.lucene.codecs.encrypted.EncryptedLucene50PostingsFormat.fieldsConsumer(EncryptedLucene50PostingsFormat.java:479)

                at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.write(PerFieldPostingsFormat.java:196)

                at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:107)

                at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:134)

                at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:423)

                at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:502)

                at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:614)

                at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2815)

                at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2989)

                at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2956)

                at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:619)

                at org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1453)

                at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1256)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                at java.util.concurrent.FutureTask.run(FutureTask.java:266)

                at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                ... 1 more

 

Caused by: java.lang.ClassNotFoundException: org.apache.lucene.codecs.encrypted.DummyCipherFactory

                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

                at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814)

                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

                ... 22 more

 

>From the output I assume that Solr is not picking up the jar with this custom codec, and
I don't get why? 

What I find strange is that, I have also tried to run this codec with Lucene and eclipse by
writing some code for indexing and searching and same codec jar is working just fine.

                     System.out.println("Indexing to directory '" + indexPath + "'...");

 

                     Directory dir = FSDirectory.open(Paths.get(indexPath));

 

                     Analyzer analyzer = new StandardAnalyzer();

                     IndexWriterConfig iwc = new IndexWriterConfig(analyzer);

 

                    // set encrypted codec //This encrypted Codec is being used for indexing

                    iwc.setCodec(new DummyEncryptedLucene60Codec());

 

                    if (create) {

                          // Create a new index in the directory, removing any

                          // previously indexed documents:

                          iwc.setOpenMode(OpenMode.CREATE);

                     } else {

                          // Add new documents to an existing index:

                          iwc.setOpenMode(OpenMode.CREATE_OR_APPEND);

                     }

 

                     IndexWriter writer = new IndexWriter(dir, iwc);

                    indexDocs(writer, docDir);

 

                    writer.close();

 

I'm guessing that As Solr uses Java SPI options to load this codec which loads things differently,
and there's something missing...?

My question is, what might I be doing wrong here or  what's missing so that codec jar can
be picked up by Solr?

Thanks.

 

 

 

 

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message