nifi-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Wicks (pwicks)" <pwi...@micron.com>
Subject PutHiveQL Multiple Ordered Statements
Date Fri, 23 Sep 2016 12:14:44 GMT
I have a PutHDFS processor drop a file, I then have a long chain of ReplaceText -> PutHiveQL
processors that runs a series of steps.
The below ~4 steps allow me to take the file generated by NiFi in one format and move it into
the final table, which is ORC with several Timestamp columns (thus why I'm not using AvroToORC,
since I'd lose my Timestamps.

The exact HQL, all in one block, is roughly:

DROP TABLE `db.tbl_${filename}`;

CREATE TABLE ` db.tbl _${filename}`(
   Some list of columns goes here that exactly matches the schema of `prod_db.tbl`
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001'
STORED AS TEXTFILE;
 LOAD DATA INPATH '${absolute.hdfs.path}/${filename}' INTO TABLE ` db.tbl _${filename}`;
 INSERT INTO `prod_db.tbl`
SELECT * FROM ` db.tbl _${filename}`;
                DROP TABLE ` db.tbl _${filename}`;

Right now I'm having to split this into 5 separate ReplaceText steps, each one followed by
a PutHiveQL.  Is there a way I can push a multi-statement, order dependent, script like this
to Hive in a simpler way?

Thanks,
  Peter

Mime
View raw message