lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jesus Olivan <jesus.oli...@letgo.com>
Subject Re: Full import alternatives
Date Fri, 13 Apr 2018 17:34:20 GMT
hi Shawn,

first of all, thanks for your answer.

How you import simultaneously these 6 shards?

2018-04-13 19:30 GMT+02:00 Shawn Heisey <apache@elyograg.org>:

> On 4/13/2018 11:03 AM, Jesus Olivan wrote:
> > thanks for your answer. It happens that when we launch full import
> process
> > didn't finished (we wait for more than 60 hours last time, and we
> cancelled
> > it, because this is not an acceptable time for us) There weren't any
> errors
> > in solr logfile simply because it was working fine. The problem is that
> it
> > lasted eternally and didn't finish. We tried it on Aurora cluster under
> > AWS, and after 20 hours of work, it failed due to lack of space in Aurora
> > tmp folder.
>
> 375 million documents importing from MySQL with one DIH import is going
> to take quite a while.
>
> The last full rebuild I did of my main index took 21.61 hours.  This is
> an index where six large shards build simultaneously, using DIH, each
> one having more than 30 million documents.  If I were to build it as a
> single 180 million document import, it would probably take 5 days, maybe
> longer.
>
> We had another index (since retired) that had more than 400 million
> total documents, built similarly with multiple shards at the same time.
> The last rebuild I can remember on that index took about two days.
>
> Thanks,
> Shawn
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message