hawq-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shailesh Doshi <sdo...@pivotal.io>
Subject Re: user Digest 17 May 2016 02:59:27 -0000 Issue 67
Date Thu, 19 May 2016 15:31:16 GMT
test

On Mon, May 16, 2016 at 10:59 PM, <
user-digest-help@hawq.incubator.apache.org> wrote:

>
> user Digest 17 May 2016 02:59:27 -0000 Issue 67
>
> Topics (messages 212 through 215)
>
> Re: no partition for partitioning key make gpload failed.
>         212 by: Lei Chang
>         213 by: Gagan Brahmi
>         214 by: yin.zhb.163.com
>
> Re: HAWQ YARN RPC Errors
>         215 by: Wen Lin
>
> Administrivia:
>
> ---------------------------------------------------------------------
> To post to the list, e-mail: user@hawq.incubator.apache.org
> To unsubscribe, e-mail: user-digest-unsubscribe@hawq.incubator.apache.org
> For additional commands, e-mail:
> user-digest-help@hawq.incubator.apache.org
>
> ----------------------------------------------------------------------
>
>
>
> ---------- Forwarded message ----------
> From: Lei Chang <lei_chang@apache.org>
> To: user <user@hawq.incubator.apache.org>
> Cc: dev <dev@hawq.incubator.apache.org>
> Date: Sun, 15 May 2016 09:16:11 +0800
> Subject: Re: no partition for partitioning key make gpload failed.
>
> Hi, i think it is because your data inserted has a partition key not in
> the range. for example, i can reproduce this on my laptop by the follow
> 'insert'. You can change you partition table definition accordingly.
>
> postgres=# insert into stocks
> values(1,1,'1','1',1,1,'1','20170101'::date);
>
>
> ERROR:  no partition for partitioning key  (seg1 localhost:40000 pid=35447)
> Thanks
> Lei
>
>
> On Sat, May 14, 2016 at 11:34 PM, yin.zhb@163.com <yin.zhb@163.com> wrote:
>
>> Hi,all:
>>     I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>>  ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>>     INPUT:
>>     - SOURCE:
>>         LOCAL_HOSTNAME:
>>         - stars71
>>         PORT: 8081
>>         FILE:
>>         - /data/data/*
>>     - COLUMNS:
>>         - storeno: bigint
>>         - itemno: bigint
>>         - itemname: text
>>         - spec: text
>>         - itemseq: bigint
>>         - stocks: bigint
>>         - unit: text
>>         - ymd: date
>>     - FORMAT: text
>>     - DELIMITER: E'\t'
>>     - ERROR_LIMIT: 25
>>     - ERROR_TABLE: stocks_err
>>     OUTPUT:
>>     - TABLE: stocks
>>     - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter
must have a length of one. Special characters must be quoted. gpload will assume this is a
sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR:  *no partition for partitioning key*
>>   (seg5 localhost:40000 pid=14591)
>>
>>  encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd")
SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted                = 0
>> 2016-05-14 23:24:20|INFO|rows Updated               = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
>
> ---------- Forwarded message ----------
> From: Gagan Brahmi <gaganbrahmi@gmail.com>
> To: user@hawq.incubator.apache.org
> Cc: dev <dev@hawq.incubator.apache.org>
> Date: Sat, 14 May 2016 20:19:14 -0700
> Subject: Re: no partition for partitioning key make gpload failed.
>
> Looks like you've a date in ymd column which falls outside the defined
> range for the two partitions.
>
> You can define a partition which can hold all the data which falls before 20150101
> with something like below:
>
> ...partition pre2015 end ('20150101'::date)...
>
>
> Regards,
>
> Gagan Brahmi
> On May 14, 2016 08:34, "yin.zhb@163.com" <yin.zhb@163.com> wrote:
>
>> Hi,all:
>>     I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>>  ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>>     INPUT:
>>     - SOURCE:
>>         LOCAL_HOSTNAME:
>>         - stars71
>>         PORT: 8081
>>         FILE:
>>         - /data/data/*
>>     - COLUMNS:
>>         - storeno: bigint
>>         - itemno: bigint
>>         - itemname: text
>>         - spec: text
>>         - itemseq: bigint
>>         - stocks: bigint
>>         - unit: text
>>         - ymd: date
>>     - FORMAT: text
>>     - DELIMITER: E'\t'
>>     - ERROR_LIMIT: 25
>>     - ERROR_TABLE: stocks_err
>>     OUTPUT:
>>     - TABLE: stocks
>>     - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter
must have a length of one. Special characters must be quoted. gpload will assume this is a
sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR:  *no partition for partitioning key*
>>   (seg5 localhost:40000 pid=14591)
>>
>>  encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd")
SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted                = 0
>> 2016-05-14 23:24:20|INFO|rows Updated               = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
> ---------- Forwarded message ----------
> From: "yin.zhb@163.com" <yin.zhb@163.com>
> To: user <user@hawq.incubator.apache.org>
> Cc: dev <dev@hawq.incubator.apache.org>
> Date: Sun, 15 May 2016 12:10:32 +0800
> Subject: Re: Re: no partition for partitioning key make gpload failed.
> The point is I set error limits, It should be skip the bad lines,Let the
> others go.
>
> ------------------------------
> yin.zhb@163.com
>
>
> *From:* Gagan Brahmi <gaganbrahmi@gmail.com>
> *Date:* 2016-05-15 11:19
> *To:* user <user@hawq.incubator.apache.org>
> *CC:* dev <dev@hawq.incubator.apache.org>
> *Subject:* Re: no partition for partitioning key make gpload failed.
>
> Looks like you've a date in ymd column which falls outside the defined
> range for the two partitions.
>
> You can define a partition which can hold all the data which falls before 20150101
> with something like below:
>
> ...partition pre2015 end ('20150101'::date)...
>
>
> Regards,
>
> Gagan Brahmi
> On May 14, 2016 08:34, "yin.zhb@163.com" <yin.zhb@163.com> wrote:
>
>> Hi,all:
>>     I am using hawq 1.1.0,When I using gpload there is a problem:
>>
>> [gpadmin@stars71 ~]$ psql -d postgres
>> postgres=# create table stocks
>> postgres-# (
>> postgres(# storeno bigint,
>> postgres(# itemno bigint,
>> postgres(# itemname text,
>> postgres(# spec text,
>> postgres(# itemseq bigint,
>> postgres(# stocks bigint,
>> postgres(# unit text,
>> postgres(# ymd date
>> postgres(# )
>> postgres-# distributed by (itemno)
>> postgres-# partition by range(ymd)
>> postgres-# (
>>
>> postgres(# partition p2015 start ('20150101'::date) end ('20160101'::date),
>> postgres(# partition p2016 start ('20160101'::date) end ('20170101'::date)
>> postgres(# );
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2015" for table "stocks"
>>
>> NOTICE:  CREATE TABLE will create partition "stocks_1_prt_p2016" for table "stocks"
>> CREATE TABLE
>> postgres=# \q
>>
>> [gpadmin@stars71
>>  ctl]$ gpfdist -d /data/data/ -p 8081 -l /home/gpadmin/log &
>> [gpadmin@stars71 ctl]$ cat stocks.ctl
>> ---
>> VERSION: 1.0.0.1
>> DATABASE: postgres
>> USER: gpadmin
>> HOST: stars71
>> PORT: 5432
>> GPLOAD:
>>     INPUT:
>>     - SOURCE:
>>         LOCAL_HOSTNAME:
>>         - stars71
>>         PORT: 8081
>>         FILE:
>>         - /data/data/*
>>     - COLUMNS:
>>         - storeno: bigint
>>         - itemno: bigint
>>         - itemname: text
>>         - spec: text
>>         - itemseq: bigint
>>         - stocks: bigint
>>         - unit: text
>>         - ymd: date
>>     - FORMAT: text
>>     - DELIMITER: E'\t'
>>     - ERROR_LIMIT: 25
>>     - ERROR_TABLE: stocks_err
>>     OUTPUT:
>>     - TABLE: stocks
>>     - MODE: INSERT
>> [gpadmin@stars71 ctl]$ gpload -f stocks.ctl
>> 2016-05-14 23:24:19|INFO|gpload session started 2016-05-14 23:24:19
>> 2016-05-14 23:24:19|INFO|setting schema 'public' for table 'stocks'
>>
>> 2016-05-14 23:24:20|INFO|started gpfdist -p 8081 -P 8082 -f "/data/data/*" -t 30
>>
>> 2016-05-14 23:24:20|WARN|A gpload control file processing warning occurred. A delimiter
must have a length of one. Special characters must be quoted. gpload will assume this is a
sql escape character sequence.
>> 2016-05-14 23:24:20|ERROR|ERROR:  *no partition for partitioning key*
>>   (seg5 localhost:40000 pid=14591)
>>
>>  encountered while running INSERT INTO public."stocks" ("storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd")
SELECT "storeno","itemno","itemname","spec","itemseq","stocks","unit","ymd" FROM ext_gpload20160514_232419_14569
>> 2016-05-14 23:24:20|INFO|rows Inserted                = 0
>> 2016-05-14 23:24:20|INFO|rows Updated               = 0
>> 2016-05-14 23:24:20|INFO|data formatting errors = 0
>> 2016-05-14 23:24:20|INFO|gpload failed
>>
>> It was failed,even if there is only one line.
>>
>> ------------------------------
>> yin.zhb@163.com
>>
>
>
> ---------- Forwarded message ----------
> From: Wen Lin <wlin@pivotal.io>
> To: dev@hawq.incubator.apache.org
> Cc: user@hawq.incubator.apache.org
> Date: Tue, 17 May 2016 10:59:16 +0800
> Subject: Re: HAWQ YARN RPC Errors
> Hi, Gagan,
>
> Where is the log? There is no attachment in your email.
>
> Thanks!
>
> On Sun, May 15, 2016 at 1:24 AM, Gagan Brahmi <gaganbrahmi@gmail.com>
> wrote:
>
>> Hi Wen,
>>
>> Please find attached logs which has a few instances of the occurrence
>> of the error.
>>
>>
>> Regards,
>> Gagan Brahmi
>>
>> On Thu, May 12, 2016 at 7:33 PM, Wen Lin <wlin@pivotal.io> wrote:
>> > Hi, Gagan,
>> >
>> > It seems a sync failure between QD and Resource Manager. Not related to
>> > libyarn 's RPC.
>> > Would you like to attach the master's log file? Thanks!
>> >
>> > On Fri, May 13, 2016 at 12:58 AM, Gagan Brahmi <gaganbrahmi@gmail.com>
>> > wrote:
>> >
>> >> Hi Team,
>> >>
>> >> Do we have some recommended tuning for the RPC warning/errors
>> >> encountered intermittently?
>> >>
>> >> The error which is seen is the following:
>> >>
>> >> WARNING:  Sync RPC framework (inet) finds exception raised.
>> >> ERROR:  failed to return resource to resource manager, failed to
>> >> receive content (pquery.c:991)
>> >>
>> >> This error however, disappears when we retry the query. There are
>> >> cases when the query is to be retried more than once.
>> >>
>> >> The error looks to be invoked when COMM2RM_CLIENT_FAIL_RECV is
>> encountered.
>> >>
>> >> The setup is using YARN resource manager. And the following is the
>> >> yarn-client configuration used:
>> >>
>> >> <configuration>
>> >>
>> >>     <property>
>> >>       <name>hadoop.security.authentication</name>
>> >>       <value>kerberos</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.connect.retry</name>
>> >>       <value>10</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.connect.tcpnodelay</name>
>> >>       <value>true</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.connect.timeout</name>
>> >>       <value>600000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.max.idle</name>
>> >>       <value>10000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.ping.interval</name>
>> >>       <value>10000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.read.timeout</name>
>> >>       <value>3600000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.socket.linger.timeout</name>
>> >>       <value>-1</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.timeout</name>
>> >>       <value>3600000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>rpc.client.write.timeout</name>
>> >>       <value>3600000</value>
>> >>     </property>
>> >>
>> >>     <property>
>> >>       <name>yarn.client.failover.max.attempts</name>
>> >>       <value>15</value>
>> >>     </property>
>> >>
>> >>   </configuration>
>> >>
>> >> I would appreciate some recommendations.
>> >>
>> >>
>> >> Regards,
>> >> Gagan Brahmi
>> >>
>>
>
>
>


-- 
*Shailesh Doshi | Data Engineering | Pivotal*
*Mobile: 513-604-9746 **| Email <sdoshi@pivotal.io>**: **sdoshi@pivotal.io*
<sdoshi@pivotal.io>
Big Data - Scale out & Open Source
<http://pivotal.io/big-data/pivotal-big-data-suite>

Mime
View raw message