hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinay gupta <vingup2...@yahoo.com>
Subject Re: mapreduce on hive with HCatInputFormat and skip.header.line.count=1
Date Wed, 07 Jun 2017 23:30:32 GMT

Hello hive-users,  I am reading a hive table with skip.header.line.count  set to 1 in TBLPROPERTIES
In the driver code I do this.

    val hiveMetaStoreClient = new HiveMetaStoreClient(new HiveConf(job.getConfiguration,
HiveIngestDriver.getClass))
    val hiveTable:Table = hiveMetaStoreClient.getTable("default", "hiveTableName")
    val hiveTableProperties = new Properties()
    hiveTableProperties.putAll(hiveTable.getParameters)
    logger.info("size: {} getParameters: {}", hiveTable.getParametersSize, hiveTableProperties.toMap)
        val hCatInputFormat = HCatInputFormat.setInput(job.getConfiguration, "default",
"hiveTableName", "day=2017-06-01")    hCatInputFormat.setProperties(hiveTableProperties)

    job.setInputFormatClass(classOf[HCatInputFormat])



Log from above shows that skip.header.line.count is set correctly. Even then HCatInputFormat
is unable to apply this as I see the header row in the output.

""
size: 4 getParameters: {last_modified_by=myuser, last_modified_time=1468952183, transient_lastDdlTime=1468952183,
skip.header.line.count=1}""

Any suggestions???
Thanks,-Vinay

Mime
View raw message