Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8F68E118E5 for ; Fri, 25 Jul 2014 00:14:37 +0000 (UTC) Received: (qmail 38471 invoked by uid 500); 25 Jul 2014 00:14:35 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 38407 invoked by uid 500); 25 Jul 2014 00:14:34 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 38395 invoked by uid 99); 25 Jul 2014 00:14:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jul 2014 00:14:34 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jpampliega@gmail.com designates 209.85.192.54 as permitted sender) Received: from [209.85.192.54] (HELO mail-qg0-f54.google.com) (209.85.192.54) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Jul 2014 00:14:33 +0000 Received: by mail-qg0-f54.google.com with SMTP id z60so4227130qgd.13 for ; Thu, 24 Jul 2014 17:14:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7uh56uMXD53HVi91/FhxLrc0Ws78ZSsIo9YR23PwcjI=; b=gSyictbHvPSN1eRGeSMNPDBIZ6yE1fj59IqQKSTV+4wbMefxx9LseKhePWXFMFvgIx Kqp7DFen8HUmLA+X0TE/ZF2A+gyjrPtTPsxvgZ9Lv8+HBl0MHvsmucsBUVcjjmAZ7Y7Y w6RVkP3xt2HhzTtm0kBj+Bisz6nxSqE8lmKxHa2QH6is21A6z/0AHlubZ1/sFk6+31bv O84VIRIiNsmsGBxMJxtk9T5rUleJ5wwdJ/R1XIaGQickFYJffWMCO9MlqgyUU1I0NTM5 nE3wZadOrZz1PEli+JygO6fdW3INo2Rt0Ubq9H2yTiDbQkVkiJoCSFZUM7gM8aEeHoIM NSBA== MIME-Version: 1.0 X-Received: by 10.140.44.67 with SMTP id f61mr20116976qga.44.1406247248268; Thu, 24 Jul 2014 17:14:08 -0700 (PDT) Received: by 10.140.104.178 with HTTP; Thu, 24 Jul 2014 17:14:08 -0700 (PDT) Received: by 10.140.104.178 with HTTP; Thu, 24 Jul 2014 17:14:08 -0700 (PDT) In-Reply-To: <33C64A5B-246C-4CC2-B546-BE5F5CA2F93B@hortonworks.com> References: <9B037E996504E94DADB79EF4A3799C7E9F717930@BLR-EC-MBX8.wipro.com> <33C64A5B-246C-4CC2-B546-BE5F5CA2F93B@hortonworks.com> Date: Thu, 24 Jul 2014 21:14:08 -0300 Message-ID: Subject: Re: CREATE TABLE throwing error for large number of columns (very long script) From: Juan Martin Pampliega To: user@hive.apache.org Cc: hive-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001a113943d8b3194304fef9739f X-Virus-Checked: Checked by ClamAV on apache.org --001a113943d8b3194304fef9739f Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Are you using MySQL or Postgres for the Metastore database? On Jul 24, 2014 9:08 PM, "Prasanth Jayachandran" < pjayachandran@hortonworks.com> wrote: > What version of hive are you using? What file format are you using? > > Thanks > Prasanth Jayachandran > > On Jul 24, 2014, at 5:03 PM, < > azaz.rasool@wipro.com> wrote: > > I am trying to Create a table in Hive. It=E2=80=99s a very long script co= ntained > large number of columns and also contains complex fields like STRUCT, ARR= AY > etc. > =C2=B7 Cannot create full table in one shot using CREATE TABLE > statement so need to first run CREATE and then ALTER > =C2=B7 If fields are just primitive fields (even if large in numb= ers) > then we can Create the table in parts (First running CREATE statement and > then running few ALTER statements) > =C2=B7 *However In few cases*, we have very large STRUCT fields i= nside > the table. Each STRUCT contains large number of columns. STRUCT cannot be > created separately by running ALTER statements because by default ALTER > works at field level and not inside a complex field like STRUCT > > When we tried to replace the column of Struct type (with few number of > columns) to Struct type (with more number of columns) =E2=80=93 Error mes= sage > appeared : FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. > > Is it a limitation from metastore (MySQL/Postgres) or something which can > be tweaked in Hive configuration? > > > Thanks & Regards, > > Azaz Rasool > > Mobile (US) : +1 917 691 6266 > > Mobile (India) : +91 776 097 4826 > > http://lnkd.in/_BDmdK > > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) an= d > may contain proprietary, confidential or privileged information. If you a= re > not the intended recipient, you should not disseminate, distribute or cop= y > this e-mail. Please notify the sender immediately and destroy all copies = of > this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > > www.wipro.com > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential= , > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified th= at > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediate= ly > and delete it from your system. Thank You. --001a113943d8b3194304fef9739f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Are you using MySQL or Postgres for the Metastore database? =

On Jul 24, 2014 9:08 PM, "Prasanth Jayachan= dran" <pjayachandr= an@hortonworks.com> wrote:
What version of hive are you using? Wha= t file format are you using?

Thanks
Prasanth Jayachandran

On Jul 24, 2014, at 5:03 PM, <azaz.rasool@wipro.com> <azaz.rasool@wipro.com&= gt; wrote:

I am trying to Create a= table in Hive. It=E2=80=99s a very long script contained large number of c= olumns and also contains complex fields like STRUCT, ARRAY etc.
<= div style=3D"margin:0in 0in 0.0001pt 0.5in;font-size:11pt;font-family:Calib= ri,sans-serif"> =C2=B7=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Cannot create full table in one shot = using CREATE TABLE statement so need to first run CREATE and then ALTER<= /u>
=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0If fields are just pri= mitive fields (even if large in numbers) then we can Create the table in pa= rts (First running CREATE statement and then running few ALTER statements)<= u>
=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0However In few case= s, we have very large STRUC= T fields inside the table. Each STRUCT contains large number of columns. ST= RUCT cannot be created separately by running ALTER statements because by de= fault ALTER works at field level and not inside a complex field like STRUCT=
=C2=A0
When we tried to replace the column o= f Struct type (with few number of columns) to Struct type (with more number= of columns) =E2=80=93=C2=A0E= rror message appeared : FAILED: Execution Error, return code 1 from org.apa= che.hadoop.hive.ql.exec.DDLTask. Unable to alter table.
=C2=A0
Is it a limitation from metastore (My= SQL/Postgres) or something which can be tweaked in Hive configuration?
=C2=A0

Thanks & Regards,

Azaz Rasool

Mobile (US) : +1 917 691 6266

<= p class=3D"MsoNormal" style=3D"margin:0in 0in 5pt;font-size:11pt;font-famil= y:Calibri,sans-serif"> Mobile (India) : +91 776 097 4826<= /p>

http://lnkd.in/_BDmdK

=C2=A0

The information contained in t= his electronic message and any attachments to this message are intended for= the exclusive use of the addressee(s) and may contain proprietary, confide= ntial or privileged information. If you are not the intended recipient, you= should not disseminate, distribute or copy this e-mail. Please notify the = sender immediately and destroy all copies of this message and any attachmen= ts.

WARNING: Computer viruses can be transmitted via email. The recipient sh= ould check this email and any attachments for the presence of viruses. The = company accepts no liability for any damage caused by any virus transmitted= by this email.

www.wipro.com



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --001a113943d8b3194304fef9739f--