lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Floyd Wu <floyd...@gmail.com>
Subject Re: What is the best approach to send lots of XML Messages to Solr to build index?
Date Tue, 17 Jun 2014 03:06:31 GMT
Hi Mikhail
Thanks for you suggestions.
Floyd


2014-06-16 17:28 GMT+08:00 Mikhail Khludnev <mkhludnev@griddynamics.com>:

> On Mon, Jun 16, 2014 at 6:57 AM, Floyd Wu <floyd.wu@gmail.com> wrote:
>
> > Hi Mikhail,
> > What is the pros. to disable tlog?
> >
> I consumes the heap much providing the benefits (real-time get, recover
> uncommitted docs on failure) which are not necessary in old-school bulk
> index scenario.
>
>
> > Each of my xml file contained to doc, one is main content and the other
> is
> > acl.
> >
> How I can guess how many of them you have? Once again submitting a few
> (let's say ten) huge files in parallel allows to utilize indexing jvm
> fully, and yields the best performance.
>
>
> > Currently I'm not using SolrCloud due to my poor understanding of this
> > architecture and pros/cons.
> > The main system is developed using .Net C# so using SolrJ won't be a
> > solution.
> >
> anyway, if you submit small requests by C# code via REST, make sure that
> http keep-alive is enabled, and you don't waste time for establishing TCP
> connection. I might be wrong but I've thought Lucid guys provide some C#
> client or just its' scratch for Solr. Don't they?
>
>
> >
> > Floyd
> >
> >
> >
> > 2014-06-15 18:14 GMT+08:00 Mikhail Khludnev <mkhludnev@griddynamics.com
> >:
> >
> > > Hello Floyd,
> > >
> > > Did you consider to disable tlog?
> > > Does a file consist of many docs?
> > > Do you have SolrCloud? Do you use just sh/curl or have a java program?
> > > DIH is not really performant so far. Submitting roughly ten huge files
> in
> > > parallel is a way to perform good. Once again, nuke tlog.
> > >
> > >
> > > On Sun, Jun 15, 2014 at 12:44 PM, Floyd Wu <floyd.wu@gmail.com> wrote:
> > >
> > > > Hi,
> > > > I have many XML Message file formatted like this
> > > > https://wiki.apache.org/solr/UpdateXmlMessages
> > > >
> > > > These files are generated by my index builder daily.
> > > > Currently I am sending these file through http post to Solr but
> > > sometimes I
> > > > hit OOM exception or pending too many tlog.
> > > >
> > > > Do you have better way to "import" these files to Solr to build
> index?
> > > >
> > > > Thanks for the suggestion
> > > >
> > > > Floyd
> > > >
> > >
> > >
> > >
> > > --
> > > Sincerely yours
> > > Mikhail Khludnev
> > > Principal Engineer,
> > > Grid Dynamics
> > >
> > > <http://www.griddynamics.com>
> > >  <mkhludnev@griddynamics.com>
> > >
> >
>
>
>
> --
> Sincerely yours
> Mikhail Khludnev
> Principal Engineer,
> Grid Dynamics
>
> <http://www.griddynamics.com>
>  <mkhludnev@griddynamics.com>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message