drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacques Nadeau <jacq...@dremio.com>
Subject Re: Drill Sql Max row size
Date Tue, 08 Sep 2015 15:15:39 GMT
Yes, a reader works a batch at a time. Typically, we use 4095 as the
default number of records to produce. I believe the Elastic approach is
based on using cursormark.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Tue, Sep 8, 2015 at 7:53 AM, Sudip Mukherjee <smukherjee@commvault.com>
wrote:

> Yes, there is no problem in that . Was just wondering if there is a
> mechanism of getting data by batches from the record reader.
> Actually , I wrote a basic plugin to solr and was thinking if I paginate
> or use cursormark params in the recordreader.
>
> Thanks,
> Sudip
>
> -----Original Message-----
> From: Tomer Shiran [mailto:tshiran@dremio.com]
> Sent: 08 September 2015 PM 08:16
> To: dev@drill.apache.org
> Subject: Re: Drill Sql Max row size
>
> That won't be a problem. There's actually no limit on how many
> records/rows you can have.
>
> > On Sep 8, 2015, at 2:00 AM, Sudip Mukherjee <smukherjee@commvault.com>
> wrote:
> >
> > Hi,
> > I have somewhere around a million records. But columns are less in
> > numbers (up to 10)
> >
> > Thanks,
> > Sudip
> >
> > -----Original Message-----
> > From: Jacques Nadeau [mailto:jacques@dremio.com]
> > Sent: 08 September 2015 AM 06:58
> > To: dev@drill.apache.org
> > Subject: Re: Drill Sql Max row size
> >
> > Generally, no. That being said, Drill will probably struggle if you
> start reading records where one or more cells is greater than a few hundred
> kilobytes (or mbs) or more than several hundred columns/fields. What size
> records are you working with?
> >
> > --
> > Jacques Nadeau
> > CTO and Co-Founder, Dremio
> >
> > On Mon, Sep 7, 2015 at 4:49 AM, Sudip Mukherjee
> > <smukherjee@commvault.com>
> > wrote:
> >
> >> Hi Devs,
> >>
> >> Is there a max row limit which I can configure while pulling data
> >> from underlying datasource? If there is a large data-set would drill
> >> fetch like page by page?
> >>
> >> Thanks,
> >> Sudip
> >>
> >>
> >>
> >> ***************************Legal
> >> Disclaimer***************************
> >> "This communication may contain confidential and privileged material
> >> for the sole use of the intended recipient. Any unauthorized review,
> >> use or distribution by others is strictly prohibited. If you have
> >> received the message by mistake, please advise the sender by reply
> >> email and delete the message. Thank you."
> >> *********************************************************************
> >> *
> >
> >
> >
> > ***************************Legal Disclaimer***************************
> > "This communication may contain confidential and privileged material
> > for the sole use of the intended recipient. Any unauthorized review,
> > use or distribution by others is strictly prohibited. If you have
> > received the message by mistake, please advise the sender by reply email
> and delete the message. Thank you."
> > **********************************************************************
>
>
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material for
> the
> sole use of the intended recipient. Any unauthorized review, use or
> distribution
> by others is strictly prohibited. If you have received the message by
> mistake,
> please advise the sender by reply email and delete the message. Thank you."
> **********************************************************************
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message