Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
MIME-Version: 1.0
In-Reply-To: <CALqceLGXne-QF1v7L+LRtE9C4+n3m40z_RPYeOf4AJNKZpv2PA@mail.gmail.com>
References: <f66f4b2f7b5a45d5b2f66b6eecb61e0d@SN2PR4204MB0349.048d.mgd.msft.net>
 <CALqceLGXne-QF1v7L+LRtE9C4+n3m40z_RPYeOf4AJNKZpv2PA@mail.gmail.com>
From: Hanifi GUNES <hanifigunes@gmail.com>
Date: Fri, 20 May 2016 16:56:57 -0700
Message-ID: <CAHcC4n_e04Ore8Q2uCwEim0hiP2oS_nfan7XJdMy9=Q=4f_Oxg@mail.gmail.com>
Subject: Re: Connection and Memory management with Apache Drill
To: dev <dev@drill.apache.org>
Cc: nitin.a.sareen@accenture.com, maneesh.kothari@accenture.com,
	h.p.kumar@accenture.com
Content-Type: multipart/alternative; boundary=001a1141f340bf725905334ed928
archived-at: Fri, 20 May 2016 23:57:24 -0000

--001a1141f340bf725905334ed928
Content-Type: text/plain; charset=UTF-8

There is no magic formula for (1). The answer entirely depends on what sort
of data you are working with, what sort of queries you are running and what
sort of SLAs you are willing to accept. Regardless, you can configure
memory requirements of Drill as to your wish modifying bootstrap
configuration. However, it is important to understand that Drill treats all
compute resources the same - at least for the time being, which means a
homogenous cluster setup is preferable.

(2) is relatively easy. A user connection is kept alive until query
completes. There are few exceptions to this but generally speaking a
connection is closed upon query reaches to a terminal state.

(3) this sounds more of a 3rd party question than Drill one. Drill offers a
JDBC interface that you might be interested in exploring. Perhaps you can
let community know whether we are spring-jdbc compatible or not.


-Hanifi

2016-05-20 4:20 GMT-07:00 Khurram Faraaz <kfaraaz@maprtech.com>:

> Hello Jasbir,
>
> Can you please elaborate on (1)
> *>> 1.       Memory and CPU requirements of Apache Drill.*
>
> I assume you already have Drill installed, since you mention that you are
> using Drill.
> Did you want to know the different memory options that Drill provides ?
> Here is a list from sys.options table.
>
> 0: jdbc:drill:schema=dfs.tmp> select * from sys.options where name like
> '%mem%';
>
> +-----------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> |                     name                      |   kind   |  type   |
>  status  |   num_val   | string_val  | bool_val  | float_val  |
>
> +-----------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | planner.memory.average_field_width            | LONG     | SYSTEM  |
> DEFAULT  | 8           | null        | null      | null       |
> | planner.memory.enable_memory_estimation       | BOOLEAN  | SYSTEM  |
> DEFAULT  | null        | null        | false     | null       |
> | planner.memory.hash_agg_table_factor          | DOUBLE   | SYSTEM  |
> DEFAULT  | null        | null        | null      | 1.1        |
> | planner.memory.hash_join_table_factor         | DOUBLE   | SYSTEM  |
> DEFAULT  | null        | null        | null      | 1.1        |
> | planner.memory.max_query_memory_per_node      | LONG     | SYSTEM  |
> DEFAULT  | 2147483648  | null        | null      | null       |
> | planner.memory.non_blocking_operators_memory  | LONG     | SYSTEM  |
> DEFAULT  | 64          | null        | null      | null       |
> | planner.memory_limit                          | LONG     | SYSTEM  |
> DEFAULT  | 268435456   | null        | null      | null       |
>
> +-----------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> 7 rows selected (0.367 seconds)
>
> You can set/unset change these config parameters using ALTER SYSTEM SET or
> the ALTER SESSION SET commands if you want to make those changes at the
> SYSTEM/SESSION levels appropriately.
>
> Thanks,
> Khurram
>
> On Fri, May 20, 2016 at 2:58 PM, <jasbir.sing@accenture.com> wrote:
>
> > Can you please let me know the details. It's impacting our task.
> >
> > Regards,
> > Jasbir Singh
> >
> > From: Sing, Jasbir
> > Sent: Monday, May 16, 2016 6:19 PM
> > To: dev@drill.apache.org
> > Cc: Sareen, Nitin A. <nitin.a.sareen@accenture.com>; Kothari, Maneesh <
> > maneesh.kothari@accenture.com>; Kumar, H. P. <h.p.kumar@accenture.com>
> > Subject: Connection and Memory management with Apache Drill
> >
> > Hi,
> >
> > We are using Apache Drill to read parquet files. We are facing few issues
> > and need your guidance -
> >
> > Problems -
> >
> > 1.       Memory and CPU requirements of Apache Drill.
> >
> > 2.       How Drill manages connections with queries fired and when it
> gets
> > closed?
> >
> > 3.       Does Apache Drill provides compatibility with spring-jdbc ? If
> > Yes, how to use it and if Not, how to make connection pool of 50
> > connections.
> >
> > Help in this regards with be appreciated.
> >
> > Regards,
> > Jasbir Singh
> >
> >
> >
> > ________________________________
> >
> > This message is for the designated recipient only and may contain
> > privileged, proprietary, or otherwise confidential information. If you
> have
> > received it in error, please notify the sender immediately and delete the
> > original. Any other use of the e-mail by you is prohibited. Where allowed
> > by local law, electronic communications with Accenture and its
> affiliates,
> > including e-mail and instant messaging (including content), may be
> scanned
> > by our systems for the purposes of information security and assessment of
> > internal compliance with Accenture policy.
> >
> >
> ______________________________________________________________________________________
> >
> > www.accenture.com
> >
>

--001a1141f340bf725905334ed928--