ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mikhail Cherkasov <mcherka...@gridgain.com>
Subject Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Date Thu, 03 Aug 2017 13:57:57 GMT
Hi again,

so it's correct hadoop core-site.xml should have "hdfs://localhost:9000".

Hadoop works with a disk. So you write to IGFS, IGFS writes to the
secondary file system which is hadoop,
that means hadoop itself shouldn't know anything about IGFS,
otherwise, it will write to IGFS, which will write to secondary FS which is
hadoop, which will write to IGFS and so on.

However,  your code or tools like hive need to work with IGFS so they
should work with core-site.xml witn igfs://localhost:9000 .

As you said looks like hive still works with hadoop directly:
https://apacheignite-fs.readme.io/docs/running-apache-hive-over-ignited-hadoop#section-starting-hive

you need to feed core-site.xml to Hive with igfs://localhost:9000 and
    <property>
        <name>fs.igfs.impl</name>
        <value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
    </property>
    <property>
        <name>fs.AbstractFileSystem.igfs.impl</name>
        <value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
    </property>

So, for now, I think your very first question should be asked to Hive and
Hadoop guys because it doesn't
work when you work directly with Hadoop.

Thanks,
Mikhail.


On Thu, Aug 3, 2017 at 12:46 PM, csumi <schawla3@sapient.com> wrote:

> Hi Mikhail,
>
> Thanks for your response.
>
> I changed hdfs://localhost:9000 to igfs://localhost:9000 in the
> core-site.xml of hive home directory but core-site.xml of Hadoop home was
> still pointing to hdfs://localhost:9000. Issue persists.
>
> If I create partition using the insert command on hive I am getting rows on
> select query. Even after stopping all ignite nodes I am still getting the
> data on select query. Does this mean that hive is still pointing to hdfs?
>
> What else can we check here?
>
> Thanks!
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-
> data-gets-saved-to-Hive-partitioned-table-tp15725p15937.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>



-- 
Thanks,
Mikhail.

Mime
View raw message