impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Valencia Serrao" <vser...@us.ibm.com>
Subject Re: Issues while creating hbase data tables
Date Mon, 30 Jan 2017 06:53:14 GMT

Thanks for your inputs, Jim!. The steps you have outlined in previous
email, match with my findings too. I continued working further on the
issue. Finally, I was able to resolve it by rebuilding hadoop with
snappy-java support inclusion (Dsnappy.prefix)  in the mvn command.




From:	Jim Apple <jbapple@cloudera.com>
To:	Valencia Serrao/Austin/Contr/IBM@IBMUS
Cc:	"dev@impala" <dev@impala.incubator.apache.org>, Manish
            Patil/Austin/Contr/IBM@IBMUS, Nishidha
            Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
            Jagadale/Austin/Contr/IBM@IBMUS
Date:	01/27/2017 03:28 AM
Subject:	Re: Issues while creating hbase data tables



It looks to me like functional_schema_template.sql is called from
testdata/bin/generate-schema-statements.py, which is called from
bin/load-data.py, which is called from testdata/bin/create-load-data.sh,
which is called from buildall.sh

$ git grep schema_tem
...
testdata/bin/generate-schema-statements.py:def parse_schema_template_file
(file_name):
testdata/bin/generate-schema-statements.py:  schema_template_file =
os.path.join(DATASET_DIR, target_dataset,
testdata/bin/generate-schema-statements.py:
'%s_schema_template.sql' % target_dataset)
...
$ git grep generate-schema-statements
bin/load-data.py:GENERATE_SCHEMA_CMD = "generate-schema-statements.py
--exploration_strategy=%s "\
$ git grep load-data
...
buildall.sh:  "${IMPALA_HOME}/testdata/bin/create-load-data.sh" $
{CREATE_LOAD_DATA_ARGS} <<< Y
...
testdata/bin/create-load-data.sh:  if ! impala-python -u $
{IMPALA_HOME}/bin/load-data.py ${ARGS[@]} &> ${LOG_FILE}; then
testdata/bin/create-load-data.sh:    if ! impala-python -u $
{IMPALA_HOME}/bin/load-data.py --workloads all\




On Wed, Jan 4, 2017 at 10:32 PM, Valencia Serrao <vserrao@us.ibm.com>
wrote:
  Hi Jim,

  Thanks for the inputs. But our requirement is to get Impala 2.6 setup on
  Ubunutu16.04.

  I continued with debugging this issue, and i can confirm that the error
  is due to the table 'functional_hbase.hbasealltypeserror' not getting
  created.
  To ascertain this , I manually ran the hbase table creation query "create
  'functional_hbase.hbasealltypeserror', 'd' " in hbase shell and then
  tried the 'put' statements and it worked fine in hbase shell. Later, I
  tried to inject the same create query in '$
  {IMPALA_HOME}/testdata/HBaseAllTypesError/functional_hbase.hbasealltypeserror.hbase',
 used in dataloading sequence and that worked fine too.

  I now require to find where the hbase table creation is triggered. For
  this, I've tried to trace the script execution flow. However, I'm not
  able to clearly reach at the table creation point. Below is the script
  execution flow i was tracing.
  buildall.sh - > create-load-data.sh -> create-hbase.sh
  In create-hbase.sh, I see only the 'put' statements for the hbase table.
  I also found a table creation query for '
  functional_hbase.hbasealltypeserror' in '$
  {IMPALA_HOME}/testdata/datasets/functional/functional_schema_template.sql,
 but I could not clearly find where it is triggered.
  Am I missing out on something ?

  It would be great if you could guide me to find the point where the table
  'functional_hbase.hbasealltypeserror' is created.

  Regards,
  Valencia

  Inactive hide details for Jim Apple ---01/03/2017 09:06:00 PM---I think
  16.04 has not always worked well. IIRC, Impala 2.6 was Jim Apple
  ---01/03/2017 09:06:00 PM---I think 16.04 has not always worked well.
  IIRC, Impala 2.6 was released only a month or two after Ub

  From: Jim Apple <jbapple@cloudera.com>
  To: "dev@impala" <dev@impala.incubator.apache.org>
  Cc: Valencia Serrao/Austin/Contr/IBM@IBMUS, Nishidha
  Panpaliya/Austin/Contr/IBM@IBMUS, Sudarshan
  Jagadale/Austin/Contr/IBM@IBMUS, Manish Patil/Austin/Contr/IBM@IBMUS
  Date: 01/03/2017 09:06 PM
  Subject: Re: Issues while creating hbase data tables




  I think 16.04 has not always worked well. IIRC, Impala 2.6 was
  released only a month or two after Ubuntu 16.04. Our pre-merge testing
  and some of our developers use 14.04 with success.

  On Thu, Dec 29, 2016 at 8:17 PM, Valencia Serrao <vserrao@us.ibm.com>
  wrote:
  > Hi All,
  >
  > While working to get impala 2.6 setup on a Ubuntu16.04 on ppc64le we
  are
  > facing following issue while testdata loading:
  >
  > ..................................
  > 16/12/28 01:56:25 ERROR client.AsyncProcess: Failed to get region
  location
  > org.apache.hadoop.hbase.TableNotFoundException:
  > functional_hbase.hbasealltypeserror
  > .....................................
  >
  > Here's the complete log. (See attached file: HBaseLog.zip)
  >
  > Environment info:
  > HBase: hbase-1.2.0-cdh5.7.0
  > Hadoop: hadoop-2.6.0-cdh5.7.0
  > Command used to load data: ${IMPALA_HOME}/buildall.sh -noclean -notests
  > -testdata
  >
  > I had also tried using default hbase-1.2.0-cdh5.8.0, however, it throws
  a
  > InvalidTargetException and also the HRegionServer service would not
  start.
  > Therefore, I switched back to hbase5.7.0.
  >
  > Checking into the create_hbase.sh file, I executed the following query
  on
  > hbase shell manually and got the same error as above:
  > Query: put 'functional_hbase.hbasealltypeserror', '20', 'd:bigint_col',
  '0'
  >
  > IMO the required functional_hbase.hbasealltypeserror table is not
  getting
  > created(?). I am further tracing the code to get to the point where the
  > hbase tables are created i.e. in generate_schema script.
  >
  > When tried earlier on Ubuntu15.10, data loading was working fine with
  the
  > same environment.
  >
  > Please guide me on this issue.
  >
  > Regards,
  > Valencia













Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message