avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron.Dossett <Aaron.Doss...@target.com>
Subject Re: Want to Add New Column in Avro Schema
Date Wed, 23 Mar 2016 13:38:18 GMT
And what happens if you simply update the .avsc file on HDFS?  Does 'describe test' show the
new columns?

From: "Lunagariya, Dhaval" <dhaval.lunagariya@citi.com<mailto:dhaval.lunagariya@citi.com>>
Reply-To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Date: Wednesday, March 23, 2016 at 8:29 AM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Cc: "'er.dcpatel@gmail.com<mailto:'er.dcpatel@gmail.com>'" <er.dcpatel@gmail.com<mailto:er.dcpatel@gmail.com>>
Subject: RE: Want to Add New Column in Avro Schema

Here is the DDL.

DROP TABLE IF EXISTS TEST;

CREATE EXTERNAL TABLE TEST
PARTITIONED BY (
COL1 STRING,
COL2 STRING
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS
INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION 'hdfs:///data/hive/TEST'
TBLPROPERTIES ('avro.schema.url'='hdfs:///user/Test.avsc');

Thanks,
Dhaval

From: Aaron.Dossett [mailto:Aaron.Dossett@target.com]
Sent: Wednesday, March 23, 2016 6:50 PM
To: user@avro.apache.org<mailto:user@avro.apache.org>
Cc: 'er.dcpatel@gmail.com<mailto:'er.dcpatel@gmail.com>'
Subject: Re: Want to Add New Column in Avro Schema

You shouldn't have to drop the table, just update the .avsc.  Can you share the DDL you use
to create the table?

From: "Lunagariya, Dhaval" <dhaval.lunagariya@citi.com<mailto:dhaval.lunagariya@citi.com>>
Reply-To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Date: Wednesday, March 23, 2016 at 8:17 AM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Cc: "'er.dcpatel@gmail.com<mailto:'er.dcpatel@gmail.com>'" <er.dcpatel@gmail.com<mailto:er.dcpatel@gmail.com>>
Subject: RE: Want to Add New Column in Avro Schema

Yes. I made require changes in .avsc file and I drop the table and re-created using updated
.avsc. But I am not getting existing data in that case.

Where am I wrong? Can you through some light

Thanks,
Dhaval

From: Aaron.Dossett [mailto:Aaron.Dossett@target.com]
Sent: Wednesday, March 23, 2016 6:36 PM
To: user@avro.apache.org<mailto:user@avro.apache.org>
Cc: 'er.dcpatel@gmail.com<mailto:'er.dcpatel@gmail.com>'
Subject: Re: Want to Add New Column in Avro Schema

If you create the external table by reference to the .avsc file (TBLPROPERTIES ( 'avro.schema.url'='hdfs://foo.avsc'))
the all you have to do is update that avsc file in a compatible way and Hive should reflect
the new schema.  I've implemented this pattern in my production system for several months
now.

-Aaron

From: "Lunagariya, Dhaval" <dhaval.lunagariya@citi.com<mailto:dhaval.lunagariya@citi.com>>
Reply-To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Date: Wednesday, March 23, 2016 at 6:32 AM
To: "user@avro.apache.org<mailto:user@avro.apache.org>" <user@avro.apache.org<mailto:user@avro.apache.org>>
Cc: "'er.dcpatel@gmail.com<mailto:'er.dcpatel@gmail.com>'" <er.dcpatel@gmail.com<mailto:er.dcpatel@gmail.com>>
Subject: Want to Add New Column in Avro Schema

Hey folks,

I want to add new column in existing Hive Table. We created external hive table with the help
of .avsc. Now I want to add new column in that table.

How can I do that without disturbing any data present in table?

Please Help.

Regards,
Dhaval


Mime
View raw message