lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dorian Hoxha <dorian.ho...@gmail.com>
Subject Re: Max vertical scaling in your experience ? (1 instance/server)
Date Fri, 16 Dec 2016 18:40:33 GMT
On Fri, Dec 16, 2016 at 5:55 PM, Erick Erickson <erickerickson@gmail.com>
wrote:

> You said that the data expires, but you haven't said
> how many docs you need to host at a time.

The data will expire in ~30 minutes average. Many of them are updates on
the same document (this makes it worse because updates are delete+insert)

> At 10M/second
> inserts you'll.... need a boatload of shards. All of the
> conversation about one beefy machine .vs. lots of not-so-beefy
> machines should wait until you answer that question.

Note, there will be multiple beefy machines(just less compared to small
machines). I was asking what would be a big enough one, so 1 instance will
be able to use the full server.

> For
> instance, indexing some _very_ simple documents on my
> laptop can hit 10,000 docs/second/shard. So you're talking
> 1,000 shards here. Indexing more complex docs I might
> get 1,000 docs/second/shard, so then you'd need 10,000
> shards. Don't take these as hard numbers, I'm
> just trying to emphasize that you'll need to do scaling
> exercises to see if what you want to do is reasonable given
> your constraints.
>
Of course. I think I've done ~80K/s/server on a previous project (it wasn't
the bottleneck so didn't bother too much) but there are too many knobs that
will change that number.

>
> If those 10M docs/second are bursty and you can stand some
> latency, then that's one set of considerations. If it's steady-state
> it's another. In either case you need some _serious_ design work
> before you go forward.
>
I expect 10M to be the burst. But it needs to handle the burst. And I don't
think I will do 10M requests, but small batches.

>
> And then you want to facet (fq clauses aren't nearly as expensive)
> and want 2 second commit intervals.
>
It is what it is.

>
> You _really_ need to stand up some test systems and see what
> performance you can get before launching off on trying to do the
> whole thing. Fortunately, you can stand up, say, a 4 shard system
> and tune it and drive it as fast as you possibly can and extrapolate
> from there.
>
My ~thinking~ would be to have 1 instance/server + 1 shard/core + 1 thread
for each shard. Assuming I remove all "blocking" disk operations (don't
know if network is async?), it should be ~best~ scenario. I'll have to see
how it functions more though.

>
> But to reiterate. This is a very high indexing rate that very few
> organizations have attempted. You _really_ need to do a
> proof-of-concept _then_ plan.
>
It's why I posted, to ask what have people used as big machine, and I would
test on that.

>
> Here's the long form of this argument:
> https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-th
> e-abstract-why-we-dont-have-a-definitive-answer/


>
> Best,
> Erick
>
Thanks!

>
> On Fri, Dec 16, 2016 at 5:19 AM, GW <thegeoforce@gmail.com> wrote:
> > Layer 2 bridge SAN is just for my Apache/apps on Conga so they can be
> spun
> > on up any host with a static IP. This has nothing to do with Solr which
> is
> > running on plain old hardware.
> >
> > Solrcloud is on a real cluster not on a SAN.
> >
> > The bit about dead with no error. I got this from a post I made asking
> > about the best way to deploy apps. Was shown some code on making your app
> > zookeeper aware. I am just getting to this so I'm talking from my ass. A
> ZK
> > aware program will have a list of nodes ready for business verses a plain
> > old Round Robin. If data on a machine is corrupted you can get 0 docs
> found
> > while a ZK aware app will know that node is shite.
> >
> >
> >
> >
> >
> >
> >
> > On 16 December 2016 at 07:20, Dorian Hoxha <dorian.hoxha@gmail.com>
> wrote:
> >
> >> On Fri, Dec 16, 2016 at 12:39 PM, GW <thegeoforce@gmail.com> wrote:
> >>
> >> > Dorian,
> >> >
> >> > From my reading, my belief is that you just need some beefy machines
> for
> >> > your zookeeper ensemble so they can think fast.
> >>
> >> Zookeeper need to think fast enough for cluster state/changes. So I
> think
> >> it scales with the number of machines/collections/shards and not
> documents.
> >>
> >> > After that your issues are
> >> > complicated by drive I/O which I believe is solved by using shards. If
> >> you
> >> > have a collection running on top of a single drive array it should not
> >> > compare to writing to a dozen drive arrays. So a whole bunch of light
> >> duty
> >> > machines that have a decent amount of memory and barely able process
> >> faster
> >> > than their drive I/O will serve you better.
> >> >
> >> My dataset will be lower than total memory, so I expect no query to hit
> >> disk.
> >>
> >> >
> >> > I think the Apache big data mandate was to be horizontally scalable to
> >> > infinity with cheap consumer hardware. In my minds eye you are not
> going
> >> to
> >> > get crazy input rates without a big horizontal drive system.
> >> >
> >> There is overhead with small machines, and with very big machines
> (pricy).
> >> So something in the middle.
> >> So small cluster of big machines or big cluster of small machines.
> >>
> >> >
> >> > I'm in the same boat. All the scaling and roll out documentation
> seems to
> >> > reference the Witch Doctor's secret handbook.
> >> >
> >> > I just started into making my applications ZK aware and really just
> >> > starting to understand the architecture. After a whole year I still
> feel
> >> > weak while at the same time I have traveled far. I still feel like an
> >> > amateur.
> >> >
> >> > My plans are to use bridge tools in Linux so all my machines are
> sitting
> >> on
> >> > the switch with layer 2. Then use Conga to monitor which apps need to
> be
> >> > running. If a server dies, it's apps are spun up on one of the other
> >> > servers using the original IP and mac address through a bridge
> firewall
> >> > gateway so there is no hold up with with mac phreaking like layer 3.
> >> Layer
> >> > 3 does not like to see a route change with a mac address. My apps
> will be
> >> > on a SAN ~ Data on as many shards/machines as financially possible.
> >> >
> >> By conga you mean https://sourceware.org/cluster/conga/spec/ ?
> >> Also SAN may/will suck like someone answered in your thread.
> >>
> >> >
> >> > I was going to put a bunch of Apache web servers in round robin to
> talk
> >> to
> >> > Solr but discovered that a Solr node can be dead and not report
> errors.
> >> >
> >> Please explain more "dead but no error".
> >>
> >> > It's all rough at the moment but it makes total sense to send Solr
> >> requests
> >> > based on what ZK says is available verses a round robin.
> >> >
> >> Yes, like I&other commenter wrote on your thread.
> >>
> >> >
> >> > Will keep you posted on my roll out if you like.
> >> >
> >> > Best,
> >> >
> >> > GW
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On 16 December 2016 at 03:31, Dorian Hoxha <dorian.hoxha@gmail.com>
> >> wrote:
> >> >
> >> > > Hello searchers,
> >> > >
> >> > > I'm researching solr for a project that would require a
> >> > max-inserts(10M/s)
> >> > > and some heavy facet+fq on top of that, though on low qps.
> >> > >
> >> > > And I'm trying to find blogs/slides where people have used some big
> >> > > machines instead of hundreds of small ones.
> >> > >
> >> > > 1. Largest I've found is this
> >> > > <https://sbdevel.wordpress.com/2016/11/30/70tb-16b-docs-
> >> > > 4-machines-1-solrcloud/>
> >> > > with 16cores + 384GB ram but they were using 25! solr4 instances /
> >> server
> >> > > which seems wasteful to me ?
> >> > >
> >> > > I know that 1 solr can have max ~29-30GB heap because GC is
> >> > wasteful/sucks
> >> > > after that, and you should leave the other amount to the os for
> >> > file-cache.
> >> > > 2. But do you think 1 instance will be able to fully-use a
> 256GB/20core
> >> > > machine ?
> >> > >
> >> > > 3. Like to share your findings/links with big-machine clusters ?
> >> > >
> >> > > Thank You
> >> > >
> >> >
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message