hawq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lili Ma <...@pivotal.io>
Subject Re: [jira] [Created] (HAWQ-1426) hawq extract meets error after the table was reorganized.
Date Fri, 07 Apr 2017 06:29:41 GMT
Hi Cyrille,
Thanks for your reply!

Passing relfilenode instead of oid can fix the problem of reorganize, but
not truncate. Since truncate modify relfilenode, but does not create
pg_aosge_$ table name.

Below is the behavior after reorganized:
1. HDFS directory for the table is recreated
2. relfilenode in pg_class for this table changed to new HDFS directory
name.
3. segrelid in pg_appendonly for this table changed to a new value.
4. new pg_aoseg.pg_aoseg_$ table generated.

Below is the behavior after truncate:
1. HDFS directory for the table is recreated
2. relfilenode in pg_class for this table changed to new HDFS directory
name.
3. pg_aoseg.pg_aoseg_$ table name remain the same, but with content cleared.

So I think the more accurate way to identify is finding the table table
through segrelid.


About truncate implementation, may someone know the reason for it?

Thanks

Lili

On Fri, Apr 7, 2017 at 1:56 PM, Cyrille Lintz <clintz@pivotal.io> wrote:

> Hello,
>
> I think the issue comes from here (in /usr/local/hawq/bin/hawqextract):
>
> def get_paqseg_files(self, *oid*):
>         '''
>         Return rows in pg_paqseg_*`oid`* table, excluding
>         rows whose content id is -1.
>
>         Example:
>         >>> accessor.get_paqseg_files(35709)
>         >>> [{'fileno':'1', 'filesize':'320', 'tupcount':'10',
> 'eofuncompressed':'320'},
>         ...  {'fileno':'2', 'filesize':'880', 'tupcount':'27',
> 'eofuncompressed':'880'},
>         ...  {'fileno':'3', 'filesize':'160', 'tupcount':'5',
> 'eofuncompressed':'160'}]
>         '''
>         qry = """
>         SELECT segno as fileno, eof as filesize, tupcount, eofuncompressed
>         FROM* pg_aoseg.pg_paqseg_%d*
>         ORDER by fileno;
>         """ % oid
>         return self.exec_query(qry)
>
> With reorganize, the pg_paqseg_* table is recreated as
>  pg_paqseg_<relfilenode> instead of pg_paqseg_<oid>.
>
> You should pass to get_paqseg_files the relfilenode instead of oid.
> It will work also with truncate.
>
> Regards,
>
>
> *Cyrille LINTZ*Advisory Solution Architect  |  Pivotal Europe South
> Mobile: + 33 (0)6 11 48 71 10 | clintz@pivotal.io
>
> 2017-04-07 7:10 GMT+02:00 Lili Ma (JIRA) <jira@apache.org>:
>
> > Lili Ma created HAWQ-1426:
> > -----------------------------
> >
> >              Summary: hawq extract meets error after the table was
> > reorganized.
> >                  Key: HAWQ-1426
> >                  URL: https://issues.apache.org/jira/browse/HAWQ-1426
> >              Project: Apache HAWQ
> >           Issue Type: Bug
> >           Components: Command Line Tools
> >             Reporter: Lili Ma
> >             Assignee: Ed Espino
> >              Fix For: 2.3.0.0-incubating
> >
> >
> > After one table is reorganized, hawq extract the table will meet error.
> >
> > Reproduce Steps:
> > 1. create an AO table
> > 2. insert into several records into it
> > 3. Get the table reorganized.  "alter table a set with
> (reorganize=true);"
> > 4. run hawq extract, error thrown out.
> >
> >
> > For the bug fix, we should also guarantee that hawq extract should work
> if
> > the table is truncated and re-inserted.
> >
> >
> >
> > --
> > This message was sent by Atlassian JIRA
> > (v6.3.15#6346)
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message