spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: Does it has a way to config limit in query on STS by default?
Date Tue, 30 Aug 2016 14:55:39 GMT
I tried both of the following with STS but neither works for me.

Starting STS with --hiveconf hive.limit.optimize.fetch.max=50

and

Setting common.max_count in Zeppelin

Without setting such limits, a query that outputs lots of rows could cause
the driver to OOM and makes TS unusable. Any workarounds or thoughts?


On Tue, Aug 2, 2016 at 7:29 AM Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> I don't think it really works and it is vague. Is it rows, blocks, network?
>
>
>
> [image: Inline images 1]
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
> On 2 August 2016 at 12:09, Chanh Le <giaosudau@gmail.com> wrote:
>
>> Hi Ayan,
>> You mean
>> common.max_count = 1000
>> Max number of SQL result to *display to prevent the browser overload*.
>> This is common properties for all connections
>>
>>
>>
>>
>> It already set default in Zeppelin but I think it doesn’t work with Hive.
>>
>>
>> DOC: http://zeppelin.apache.org/docs/0.7.0-SNAPSHOT/interpreter/jdbc.html
>>
>>
>> On Aug 2, 2016, at 6:03 PM, ayan guha <guha.ayan@gmail.com> wrote:
>>
>> Zeppelin already has a param for jdbc
>> On 2 Aug 2016 19:50, "Mich Talebzadeh" <mich.talebzadeh@gmail.com> wrote:
>>
>>> Ok I have already set up mine
>>>
>>>     <property>
>>>     <name>hive.limit.optimize.fetch.max</name>
>>>     <value>50000</value>
>>>     <description>
>>>       Maximum number of rows allowed for a smaller subset of data for
>>> simple LIMIT, if it is a fetch query.
>>>       Insert queries are not restricted by this limit.
>>>     </description>
>>>   </property>
>>>
>>> I am surprised that yours was missing. What did you set it up to?
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> Dr Mich Talebzadeh
>>>
>>>
>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>
>>>
>>> http://talebzadehmich.wordpress.com
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>> On 2 August 2016 at 10:18, Chanh Le <giaosudau@gmail.com> wrote:
>>>
>>>> I tried and it works perfectly.
>>>>
>>>> Regards,
>>>> Chanh
>>>>
>>>>
>>>> On Aug 2, 2016, at 3:33 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>>> wrote:
>>>>
>>>> OK
>>>>
>>>> Try that
>>>>
>>>> Another tedious way is to create views in Hive based on tables and use
>>>> limit on those views.
>>>>
>>>> But try that parameter first if it does anything.
>>>>
>>>> HTH
>>>>
>>>>
>>>> Dr Mich Talebzadeh
>>>>
>>>>
>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>
>>>>
>>>> http://talebzadehmich.wordpress.com
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>> On 2 August 2016 at 09:13, Chanh Le <giaosudau@gmail.com> wrote:
>>>>
>>>>> Hi Mich,
>>>>> I use Spark Thrift Server basically it acts like Hive.
>>>>>
>>>>> I see that there is property in Hive.
>>>>>
>>>>> hive.limit.optimize.fetch.max
>>>>>
>>>>>    - Default Value: 50000
>>>>>    - Added In: Hive 0.8.0
>>>>>
>>>>> Maximum number of rows allowed for a smaller subset of data for simple
>>>>> LIMIT, if it is a fetch query. Insert queries are not restricted by this
>>>>> limit.
>>>>>
>>>>>
>>>>> Is that related to the problem?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Aug 2, 2016, at 2:55 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
>>>>> wrote:
>>>>>
>>>>> This is a classic problem on any RDBMS
>>>>>
>>>>> Set the limit on the number of rows returned like maximum of 50K rows
>>>>> through JDBC
>>>>>
>>>>> What is your JDBC connection going to? Meaning which RDBMS if any?
>>>>>
>>>>> HTH
>>>>>
>>>>> Dr Mich Talebzadeh
>>>>>
>>>>>
>>>>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>>>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>>>>
>>>>>
>>>>> http://talebzadehmich.wordpress.com
>>>>>
>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>>> any loss, damage or destruction of data or any other property which may
>>>>> arise from relying on this email's technical content is explicitly
>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>> arising from such loss, damage or destruction.
>>>>>
>>>>>
>>>>>
>>>>> On 2 August 2016 at 08:41, Chanh Le <giaosudau@gmail.com> wrote:
>>>>>
>>>>>> Hi everyone,
>>>>>> I setup STS and use Zeppelin to query data through JDBC connection.
>>>>>> A problem we are facing is users usually forget to put limit in the
>>>>>> query so it causes hang the cluster.
>>>>>>
>>>>>> SELECT * FROM tableA;
>>>>>>
>>>>>> Is there anyway to config the limit by default ?
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Chanh
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message