lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Robinson <mark123lea...@gmail.com>
Subject Re: Solr architecture
Date Wed, 10 Feb 2016 21:26:19 GMT
Thanks everyone for your suggestions.
Based on it I am planning to have a doc per event.



On Wed, Feb 10, 2016 at 3:38 AM, Emir Arnautovic <
emir.arnautovic@sematext.com> wrote:

> Hi Mark,
> Appending session actions just to be able to return more than one session
> without retrieving large number of results is not good tradeoff. Like
> Upayavira suggested, you should consider storing one action per doc and
> aggregate on read time or push to Solr once session ends and aggregate on
> some other layer.
> If you are thinking handling infrastructure might be too much, you may
> consider using some of logging services to hold data. One such service is
> Sematext's Logsene (http://sematext.com/logsene).
>
> Thanks,
> Emir
>
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> On 10.02.2016 03:22, Mark Robinson wrote:
>
>> Thanks for your replies and suggestions!
>>
>> Why I store all events related to a session under one doc?
>> Each session can have about 500 total entries (events) corresponding to
>> it.
>> So when I try to retrieve a session's info it can back with around 500
>> records. If it is this compounded one doc per session, I can retrieve more
>> sessions at a time with one doc per session.
>> eg under a sessionId an array of eventA activities, eventB activities
>>   (using json). When an eventA activity again occurs, we will read all
>> that
>> data for that session, append this extra info to evenA data and push the
>> whole session related data back (indexing) to Solr. Like this for many
>> sessions parallely.
>>
>>
>> Why NRT?
>> Parallely many sessions are being written (4Million sessions hence
>> 4Million
>> docs per day). A person can do this querying any time.
>>
>> It is just a look up?
>> Yes. We just need to retrieve all info for a session and pass it on to
>> another system. We may even do some extra querying on some data like
>> timestamps, pageurl etc in that info added to a session.
>>
>> Thinking of having the data separate from the actual Solr Instance and
>> mention the loc of the dataDir in solrconfig.
>>
>> If Solr is not a good option could you please suggest something which will
>> satisfy this use case with min response time while querying.
>>
>> Thanks!
>> Mark
>>
>> On Tue, Feb 9, 2016 at 6:02 PM, Daniel Collins <danwcollins@gmail.com>
>> wrote:
>>
>> So as I understand your use case, its effectively logging actions within a
>>> user session, why do you have to do the update in NRT?  Why not just log
>>> all the user session events (with some unique key, and ensuring the
>>> session
>>> Id is in the document somewhere), then when you want to do the query, you
>>> join on the session id, and that gives you all the data records for that
>>> session. I don't really follow why it has to be 1 document (which you
>>> continually update). If you really need that aggregation, couldn't that
>>> happen offline?
>>>
>>> I guess your 1 saving grace is that you query using the unique ID (in
>>> your
>>> scenario) so you could use the real-time get handler, since you aren't
>>> doing a complex query (strictly its not a search, its a raw key lookup).
>>>
>>> But I would still question your use case, if you go the Solr route for
>>> that
>>> kind of scale with querying and indexing that much, you're going to have
>>> to
>>> throw a lot of hardware at it, as Jack says probably in the order of
>>> hundreds of machines...
>>>
>>> On 9 February 2016 at 19:00, Upayavira <uv@odoko.co.uk> wrote:
>>>
>>> Bear in mind that Lucene is optimised towards high read lower write.
>>>> That is, it puts in a lot of effort at write time to make reading
>>>> efficient. It sounds like you are going to be doing far more writing
>>>> than reading, and I wonder whether you are necessarily choosing the
>>>> right tool for the job.
>>>>
>>>> How would you later use this data, and what advantage is there to
>>>> storing it in Solr?
>>>>
>>>> Upayavira
>>>>
>>>> On Tue, Feb 9, 2016, at 03:40 PM, Mark Robinson wrote:
>>>>
>>>>> Hi,
>>>>> Thanks for all your suggestions. I took some time to get the details
to
>>>>> be
>>>>> more accurate. Please find what I have gathered:-
>>>>>
>>>>> My data being indexed is something like this.
>>>>> I am basically capturing all data related to a user session.
>>>>> Inside a session I have categorized my actions like actionA, actionB
>>>>> etc..,
>>>>> per page.
>>>>> So each time an action pertaining to say actionA or actionB etc.. (in
>>>>> each
>>>>> page) happens, it is updated in Solr under that session (sessionId).
>>>>>
>>>>> So in short there is only one doc pertaining to a single session
>>>>> (identified by sessionid) in my Solr index and that is retrieved and
>>>>> updated
>>>>> whenever a new action under that session occurs.
>>>>> We expect upto 4Million session per day.
>>>>>
>>>>> On an average *one session's* *doc has a size* of *3MB to 20MB*.
>>>>> So if it is *4Million sessions per day*, each session writing around
>>>>>
>>>> *500
>>>
>>>> times to Solr*, it is* 2Billion writes or (indexing) per day to Solr*.
>>>>> As it is one doc per session, it is *4Million docs per day*.
>>>>> This is around *80K docs indexed per second* during *peak* hours and
>>>>> around *15K
>>>>> docs indexed per second* into Solr during* non-peak* hours.
>>>>> Number of queries per second is around *320 queries per second*.
>>>>>
>>>>>
>>>>> 1. Average size of a doc
>>>>>       3MB to 20MB
>>>>> 2. Query types:-
>>>>>       Until that session is in progress, whatever data is there for
>>>>> that
>>>>> session so far is queried and the new action's details captured and
>>>>> appended to existing data already captured        related to that
>>>>>
>>>> session
>>>
>>>> and indexed back into Solr. So, longer the session the more data
>>>>> retrieved
>>>>> for each subsequent query to get current data captured for that
>>>>>
>>>> session.
>>>
>>>>       Also querying can be done on timestamp etc... which is captured
>>>>>       along
>>>>> with each action.
>>>>> 3. Are docs grouped somehow?
>>>>>       All data related to a session are retrieved from Solr, updated
>>>>> and
>>>>> indexed back to Solr based on sessionId. No other grouping.
>>>>> 4. Are they time sensitive (NRT or offline process does this)
>>>>>       As mentioned above this is in NRT. Each time a new user action
in
>>>>>       that
>>>>> session happens, we need to query existing session info already
>>>>>
>>>> captured
>>>
>>>> related to that session and        append this new data  to this
>>>>>
>>>> existing
>>>
>>>> info retrieved and index it back to Solr.
>>>>> 5. Will they update or it is rebuild every time, etc.
>>>>>       Each time a new user action occurs, the full data pertaining to
>>>>>
>>>> that
>>>
>>>> session so far captured is retrieved from Solr, the extra latest data
>>>>> pertaining to this new action is appended      and indexed  back to
>>>>>
>>>> Solr.
>>>
>>>> 6. And the other thing you haven't told us is whether you plan on
>>>>> _adding_
>>>>> 2B docs a day or whether that number is the total corpus size and you
>>>>>
>>>> are
>>>
>>>> re-indexing the 2B docs/day. IOW, if you are  adding 2B docs/day, 30
>>>>>
>>>> days
>>>
>>>> later do you have 2B docs or 60B docs in your
>>>>>     corpus?
>>>>>     We are expecting around 4 million sessions per day (per session 500
>>>>> writes to Solr), which turns out to be 2B indexing done per day. So
>>>>>
>>>> after
>>>
>>>> 30 days it would be 4Milion*30          docs in the index.
>>>>> 7. Is there any aging of docs
>>>>>       No we always query against the whole corpus present.
>>>>> 8. Is any doc deleted?
>>>>>       No all data remains in the index
>>>>>
>>>>> Any suggestion is very welcome!
>>>>>
>>>>> Thanks!
>>>>> Mark.
>>>>>
>>>>>
>>>>> On Mon, Feb 8, 2016 at 3:30 PM, Jack Krupansky <
>>>>>
>>>> jack.krupansky@gmail.com
>>>
>>>> wrote:
>>>>>
>>>>> Oops... at 100 qps for a single node you would need 120 nodes to get
>>>>>>
>>>>> to 12K
>>>>
>>>>> qps and 800 nodes to get 80K qps, but that is just an extremely rough
>>>>>> ballpark estimate, not some precise and firm number. And that's if
>>>>>>
>>>>> all
>>>
>>>> the
>>>>
>>>>> queries can be evenly distributed throughout the cluster and don't
>>>>>>
>>>>> require
>>>>
>>>>> fanout to other shards, which effectively turns each incoming query
>>>>>>
>>>>> into n
>>>>
>>>>> queries where n is the number of shards.
>>>>>>
>>>>>> -- Jack Krupansky
>>>>>>
>>>>>> On Mon, Feb 8, 2016 at 12:07 PM, Jack Krupansky <
>>>>>>
>>>>> jack.krupansky@gmail.com>
>>>>
>>>>> wrote:
>>>>>>
>>>>>> So is there any aging or TTL (in database terminology) of older
>>>>>>>
>>>>>> docs?
>>>
>>>> And do all of your queries need to query all of the older documents
>>>>>>>
>>>>>> all
>>>>
>>>>> of
>>>>>>
>>>>>>> the time or is there a clear hierarchy of querying for aged
>>>>>>>
>>>>>> documents,
>>>>
>>>>> like
>>>>>>
>>>>>>> past 24-hours vs. past week vs. past year vs. older than a year?
>>>>>>>
>>>>>> Sure,
>>>>
>>>>> you
>>>>>>
>>>>>>> can always use a function query to boost by the inverse of document
>>>>>>>
>>>>>> age,
>>>>
>>>>> but Solr would be more efficient with filter queries or separate
>>>>>>>
>>>>>> indexes
>>>>
>>>>> for different time scales.
>>>>>>>
>>>>>>> Are documents ever updated or are they write-once?
>>>>>>>
>>>>>>> Are documents explicitly deleted?
>>>>>>>
>>>>>>> Technically you probably could meet those specs, but... how many
>>>>>>> organizations have the resources and the energy to do so?
>>>>>>>
>>>>>>> As a back of the envelope calculation, if Solr gave you 100 queries
>>>>>>>
>>>>>> per
>>>>
>>>>> second per node, that would mean you would need 1,200 nodes. It
>>>>>>>
>>>>>> would
>>>
>>>> also
>>>>>>
>>>>>>> depend on whether those queries are very narrow so that a single
>>>>>>>
>>>>>> node can
>>>>
>>>>> execute them or if they require fanout to other shards and then
>>>>>>>
>>>>>> aggregation
>>>>>>
>>>>>>> of results from those other shards.
>>>>>>>
>>>>>>> -- Jack Krupansky
>>>>>>>
>>>>>>> On Mon, Feb 8, 2016 at 11:24 AM, Erick Erickson <
>>>>>>>
>>>>>> erickerickson@gmail.com
>>>>
>>>>> wrote:
>>>>>>>
>>>>>>> Short form: You really have to prototype. Here's the long form:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>> https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/
>>>
>>>> I've seen between 20M and 200M docs fit on a single piece of
>>>>>>>>
>>>>>>> hardware,
>>>>
>>>>> so you'll absolutely have to shard.
>>>>>>>>
>>>>>>>> And the other thing you haven't told us is whether you plan
on
>>>>>>>> _adding_ 2B docs a day or whether that number is the total
corpus
>>>>>>>>
>>>>>>> size
>>>>
>>>>> and you are re-indexing the 2B docs/day. IOW, if you are adding 2B
>>>>>>>> docs/day, 30 days later do you have 2B docs or 60B docs in
your
>>>>>>>> corpus?
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Erick
>>>>>>>>
>>>>>>>> On Mon, Feb 8, 2016 at 8:09 AM, Susheel Kumar <
>>>>>>>>
>>>>>>> susheel2777@gmail.com>
>>>>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Also if you are expecting indexing of 2 billion docs
as NRT or
>>>>>>>>>
>>>>>>>> if
>>>
>>>> it
>>>>
>>>>> will
>>>>>>>>
>>>>>>>>> be offline (during off hours etc).  For more accurate
sizing you
>>>>>>>>>
>>>>>>>> may
>>>>
>>>>> also
>>>>>>>>
>>>>>>>>> want to index say 10 million documents which may give
you idea
>>>>>>>>>
>>>>>>>> how
>>>
>>>> much
>>>>>>
>>>>>>> is
>>>>>>>>
>>>>>>>>> your index size and then use that for extrapolation to
come up
>>>>>>>>>
>>>>>>>> with
>>>>
>>>>> memory
>>>>>>>>
>>>>>>>>> requirements.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Susheel
>>>>>>>>>
>>>>>>>>> On Mon, Feb 8, 2016 at 11:00 AM, Emir Arnautovic <
>>>>>>>>> emir.arnautovic@sematext.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi Mark,
>>>>>>>>>> Can you give us bit more details: size of docs, query
types,
>>>>>>>>>>
>>>>>>>>> are
>>>
>>>> docs
>>>>
>>>>> grouped somehow, are they time sensitive, will they update or
>>>>>>>>>>
>>>>>>>>> it
>>>
>>>> is
>>>>
>>>>> rebuild
>>>>>>>>
>>>>>>>>> every time, etc.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Emir
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 08.02.2016 16:56, Mark Robinson wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>> We have a requirement where we would need to
index around 2
>>>>>>>>>>>
>>>>>>>>>> Billion
>>>>
>>>>> docs
>>>>>>>>
>>>>>>>>> in
>>>>>>>>>>> a day.
>>>>>>>>>>> The queries against this indexed data set can
be around 80K
>>>>>>>>>>>
>>>>>>>>>> queries
>>>>
>>>>> per
>>>>>>>>
>>>>>>>>> second during peak time and during non peak hours around
12K
>>>>>>>>>>>
>>>>>>>>>> queries
>>>>
>>>>> per
>>>>>>>>
>>>>>>>>> second.
>>>>>>>>>>>
>>>>>>>>>>> Can Solr realize this huge volumes.
>>>>>>>>>>>
>>>>>>>>>>> If so, assuming we have no constraints for budget
what would
>>>>>>>>>>>
>>>>>>>>>> be
>>>
>>>> a
>>>>
>>>>> recommended Solr set up (number of shards, number of Solr
>>>>>>>>>>>
>>>>>>>>>> instances
>>>>
>>>>> etc...)
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> Mark
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized
Log
>>>>>>>>>>
>>>>>>>>> Management
>>>>>>
>>>>>>> Solr & Elasticsearch Support * http://sematext.com/
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message