lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Nazemian <alinazem...@gmail.com>
Subject Re: integrating Accumulo with solr
Date Sun, 27 Jul 2014 05:20:06 GMT
Dear Jack,
Hi,
One more thing to mention: I dont want to use solr or lucence for indexing
accumulo or full text search inside that. I am looking for have both in a
sync mode. I mean import some parts of data to solr for indexing. For this
purpose probably I need something like trigger in RDBMS, I have to define
something (probably with accumulo iterator) to import to solr on inserting
new data.
Regards.

On Fri, Jul 25, 2014 at 12:59 PM, Ali Nazemian <alinazemian@gmail.com>
wrote:

> Dear Jack,
> Actually I am going to do benefit-cost analysis for in-house developement
> or going for sqrrl support.
> Best regards.
>
>
> On Thu, Jul 24, 2014 at 11:48 PM, Jack Krupansky <jack@basetechnology.com>
> wrote:
>
>> Like I said, you're going to have to be a real, hard-core gunslinger to
>> do that well. Sqrrl uses Lucene directly, BTW:
>>
>> "Full-Text Search: Utilizing open-source Lucene and custom indexing
>> methods, Sqrrl Enterprise users can conduct real-time, full-text search
>> across data in Sqrrl Enterprise."
>>
>> See:
>> http://sqrrl.com/product/search/
>>
>> Out of curiosity, why are you not using that integrated Lucene support of
>> Sqrrl Enterprise?
>>
>>
>> -- Jack Krupansky
>>
>> -----Original Message----- From: Ali Nazemian
>> Sent: Thursday, July 24, 2014 3:07 PM
>>
>> To: solr-user@lucene.apache.org
>> Subject: Re: integrating Accumulo with solr
>>
>> Dear Jack,
>> Thank you. I am aware of datastax but I am looking for integrating
>> accumulo
>> with solr. This is something like what sqrrl guys offer.
>> Regards.
>>
>>
>> On Thu, Jul 24, 2014 at 7:27 PM, Jack Krupansky <jack@basetechnology.com>
>> wrote:
>>
>>  If you are not a "true hard-core gunslinger" who is willing to dive in
>>> and
>>> integrate the code yourself, instead you should give serious
>>> consideration
>>> to a product such as DataStax Enterprise that fully integrates and
>>> packages
>>> a NoSQL database (Cassandra) and Solr for search. The security aspects
>>> are
>>> still a work in progress, but certainly headed in the right direction.
>>> And
>>> it has Hadoop and Spark integration as well.
>>>
>>> See:
>>> http://www.datastax.com/what-we-offer/products-services/
>>> datastax-enterprise
>>>
>>> -- Jack Krupansky
>>>
>>> -----Original Message----- From: Ali Nazemian
>>> Sent: Thursday, July 24, 2014 10:30 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: integrating Accumulo with solr
>>>
>>>
>>> Thank you very much. Nice Idea but how can Solr and Accumulo can be
>>> synchronized in this way?
>>> I know that Solr can be integrated with HDFS and also Accumulo works on
>>> the
>>> top of HDFS. So can I use HDFS as integration point? I mean set Solr to
>>> use
>>> HDFS as a source of documents as well as the destination of documents.
>>> Regards.
>>>
>>>
>>> On Thu, Jul 24, 2014 at 4:33 PM, Joe Gresock <jgresock@gmail.com> wrote:
>>>
>>>  Ali,
>>>
>>>>
>>>> Sounds like a good choice.  It's pretty standard to store the primary
>>>> storage id as a field in Solr so that you can search the full text in
>>>> Solr
>>>> and then retrieve the full document elsewhere.
>>>>
>>>> I would recommend creating a document structure in Solr with whatever
>>>> fields you want indexed (most likely as text_en, etc.), and then store a
>>>> "string" field named "content_id", which would be the Accumulo row id
>>>> that
>>>> you look up with a scan.
>>>>
>>>> One caveat -- Accumulo will be protected at the cell level, but if you
>>>> need
>>>> your Solr search results to be protected by complex authorization
>>>> strings
>>>> similar to Accumulo, you will need to write your own QParserPlugin and
>>>> use
>>>> post filtering:
>>>> http://java.dzone.com/articles/custom-security-filtering-solr
>>>>
>>>> The code you see in that article is written for an earlier version of
>>>> Solr,
>>>> but it's not too difficult to adjust it for the latest (we've done so in
>>>> our project).  Once you've implemented this, you would store an
>>>> "authorizations" string field in each Solr document, and pass in the
>>>> authorizations that the user has access to in the fq parameter of every
>>>> query.  It's also not too bad to write something that parses the
>>>> Accumulo
>>>> authorizations string (like A&B&(C|D|E|F)) and interpret it accordingly
>>>> in
>>>> the QParserPlugin.
>>>>
>>>> This will give you true row level security in Solr and Accumulo, and it
>>>> performs quite well in Solr.
>>>>
>>>> Let me know if you have any other questions.
>>>>
>>>> Joe
>>>>
>>>>
>>>> On Thu, Jul 24, 2014 at 4:07 AM, Ali Nazemian <alinazemian@gmail.com>
>>>> wrote:
>>>>
>>>> > Dear Joe,
>>>> > Hi,
>>>> > I am going to store the crawl web pages in accumulo as the main
>>>> storage
>>>> > part of my project and I need to give these data to solr for indexing
>>>> >
>>>> and
>>>> > user searches. I need to do some social and web analysis on my data
as
>>>> well
>>>> > as having some security features. Therefore accumulo is my choice for
>>>> >
>>>> the
>>>> > database part and for index and search I am going to use Solr. Would
>>>> > you
>>>> > please guide me through that?
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Jul 24, 2014 at 1:28 AM, Joe Gresock <jgresock@gmail.com>
>>>> wrote:
>>>> >
>>>> > > We store data in both Solr and Accumulo -- do you have more details
>>>> about
>>>> > > what kind of data and indexing you want?  Is there a reason you're
>>>> > thinking
>>>> > > of using both databases in particular?
>>>> > >
>>>> > >
>>>> > > On Wed, Jul 23, 2014 at 5:17 AM, Ali Nazemian <
>>>> alinazemian@gmail.com>
>>>> > > wrote:
>>>> > >
>>>> > > > Dear All,
>>>> > > > Hi,
>>>> > > > I was wondering is there anybody out there that tried to integrate
>>>> Solr
>>>> > > > with Accumulo? I was thinking about using Accumulo on top
of HDFS
>>>> >
>>>> > > and
>>>> > > using
>>>> > > > Solr to index data inside Accumulo? Do you have any idea how
can I
>>>> > > > do
>>>> > > such
>>>> > > > integration?
>>>> > > >
>>>> > > > Best regards.
>>>> > > >
>>>> > > > --
>>>> > > > A.Nazemian
>>>> > > >
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > I know what it is to be in need, and I know what it is to have
> >
>>>> plenty.
>>>>  I
>>>> > > have learned the secret of being content in any and every situation,
>>>> > > whether well fed or hungry, whether living in plenty or in want.
 I
>>>> >
>>>> > can
>>>> > do
>>>> > > all this through him who gives me strength.    *-Philippians
>>>> 4:12-13*
>>>> > >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > A.Nazemian
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> I know what it is to be in need, and I know what it is to have plenty.
>>>>  I
>>>> have learned the secret of being content in any and every situation,
>>>> whether well fed or hungry, whether living in plenty or in want.  I can
>>>> do
>>>> all this through him who gives me strength.    *-Philippians 4:12-13*
>>>>
>>>>
>>>>
>>>
>>> --
>>> A.Nazemian
>>>
>>>
>>
>>
>> --
>> A.Nazemian
>>
>
>
>
> --
> A.Nazemian
>



-- 
A.Nazemian

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message