incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: General guidance on blur-shard server
Date Sat, 07 Mar 2015 05:30:46 GMT
On Friday, March 6, 2015, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> Don't know if this helps anyone. When I was trying different deployment
> options on the drawing board, one particular config stood out when
> co-locating DNs & Shard-servers...
>
> 1. Run Data-Nodes & Shard-servers co-located…


I would recommend this for sure.


> 2. Make sure all such machines are placed in one rack [Say RackA]…


Is this for network purposes ?


> 3. Start Data-Nodes alone {without Shard-servers} in a separate RackB…


I have not tried this before, I assume this is just to lower write overhead
on the shard severs?


>
> With such a config in place, hadoop places the 1st write-copy locally in
> RackA. 2nd & 3rd Copies will be placed in RackB.


I thought the normal hdfs replica rules were once local. One remote rack
once same rack.


>
> RackA machines can use better disks like SSDs or SAS15k drives with lower
> storage capacity. RackB machines will be commodity/JBOD machines with high
> storage capacity [like a 2*4TB 7.2k SATA etc…]
>
> Reads will go via shard-servers and most of the time access
> local-but-excellent disk sub-system
>
> Writes will experience bit of latency because of 2 racks but write-load on
> shard-servers will be much-less {No 2nd & 3rd copy to be written}
>
> Apologies if this is basic stuff for the community.


Not basic stuff at all. Thanks for information on your setup. It is
interesting that you are using a mixed server setup across racks. How did
land on your current configuration ?


>
> --
> Ravi
>
> On Mon, Mar 2, 2015 at 11:31 AM, Ravikumar Govindarajan <
> ravikumar.govindarajan@gmail.com <javascript:;>> wrote:
>
> > Many thanks Aaron…
> >
> > Ok that sounds about right.  This will greatly depend on your data, how
> >> many fields, how many terms, etc.
> >
> >
> > Fields will be around 20-25.. But majority of searches happen on only 5-6
> > fields. Term count ~= 2.5 million.
> >
> > I would recommend using the latest Java7
> >> (perhaps Java8 but I haven't tested with it), and use the G1 garbage
> >> collector if you plan on running larger heaps
> >
> >
> > We are using latest version of 1.7. Actually we imported around 2TB of
> > data as dry-run with just 16GB heap without major GC issues using good
> old
> > CMS. There is no sorting/faceting during searches also. My reluctance
> stems
> > from the fact that I am not quite familiar with G1 :)
> >
> > I am favouring more for write-thru cache [at-least around 50Gb] rather
> > than read-cache because we have a lot of free RAM available & I feel read
> > cache is going to use very less RAM. But I am not sure about this. Any
> > pointers will greatly help.
> >
> > I would also recommend that you increase the
> >> block cache buffer and file buffer sizes from 8K to 64K
> >
> >
> > This is one issue we faced during dry-run. Marking-up from 8K to 16K
> > solved the issue. I thought 32K must be a good fit for us. Will surely
> > explore this…
> >
> > Thanks again for helping out
> >
> > On Sat, Feb 28, 2015 at 3:04 AM, Aaron McCurry <amccurry@gmail.com
> <javascript:;>> wrote:
> >
> >> On Fri, Feb 27, 2015 at 1:29 AM, Ravikumar Govindarajan <
> >> ravikumar.govindarajan@gmail.com <javascript:;>> wrote:
> >>
> >> > Hi,
> >> >
> >> > I need a general guidance on number of machines/shards required for
> our
> >> > first blur set-up
> >> >
> >> > Some data as follows
> >> >
> >> > 1. Shard-Server Config : 128GB RAM, 16-core dual socket with
> >> > hyper-threading. 32 procs
> >> > 2. Total dataset size: 10TB. With rep-factor=3, total
> cluster-size=30TB.
> >> > Pre-populated via
> >> >     MR or Thrift...
> >> > 3. We receive very less queries per minute [600-900 queries]. But the
> >> > response times for
> >> >     every query must be <=150 ms
> >> >
> >> > Initially we thought we can create 500 shards each of 20GB size with
> >> around
> >> > 20 machines.
> >> >
> >>
> >> Ok that sounds about right.  This will greatly depend on your data, how
> >> many fields, how many terms, etc.
> >>
> >>
> >> >
> >> > Can each shard-server machine with above specs handle 25 shards? Is
> >> such a
> >> > configuration over-utilized/under-utilized?
> >> >
> >>
> >> Again, it likely depends.  I would recommend using the latest Java7
> >> (perhaps Java8 but I haven't tested with it), and use the G1 garbage
> >> collector if you plan on running larger heaps.  Whatever is leftover can
> >> be
> >> allocated to the block cache.  I would also recommend that you increase
> >> the
> >> block cache buffer and file buffer sizes from 8K to 64K.  This will also
> >> decrease heap pressure for the number of entries in the block cache lru
> >> map.
> >>
> >>
> >> >
> >> > How do folks run in production. Any numbers/pointers will be really
> >> helpful
> >> >
> >>
> >> Generally large shard servers (like the ones you are suggesting) with a
> >> few
> >> controllers (6-12) are typical setups.
> >>
> >> If I can provide more details I will follow up.
> >>
> >> Aaron
> >>
> >>
> >> >
> >> > --
> >> > Ravi
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message