ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From csumi <schaw...@sapient.com>
Subject Re: how to append data to IGFS so that data gets saved to Hive partitioned table?
Date Wed, 02 Aug 2017 09:35:11 GMT
Ignite-hadoop accelerator is configured as
https://apacheignite.readme.io/v1.0/docs/hadoop-accelerator#section-secondary-file-system.

Idea is to run hive queries on IGFS but looks like its not working that way.
Not sure how to confirm if hive is connecting to HDFS or IGFS. Also, I think
creating partition by passing comma separated string to fs.append is not
creating partition correctly which is resulting in no result on hive query.

Below are some of the configurations I have. Please let me know if I need to
share any more details.

core-site.xml is having these properties:

<property>
		<name>fs.defaultFS</name>
		<value>hdfs://localhost:9000</value>
	</property>
	
	
	<property>
		<name>fs.igfs.impl</name>
		<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
  </property>
  <property>
		<name>fs.AbstractFileSystem.igfs.impl</name>
		<value>org.apache.ignite.hadoop.fs.v2.IgniteHadoopFileSystem</value>
  </property>
      <property>
       
<name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
        <value>NEVER</value>
    </property>

ignite's default-config has below configuration

<property name="secondaryFileSystem">
                        <bean
class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
                            <property name="fileSystemFactory">
                                <bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
                                    <property name="uri"
value="hdfs://localhost:9000/"/>
									<property name="configPaths">
										<list>
											<value>D:/hadoop/etc/hadoop/core-site.xml</value>
										</list>
									</property>
                                </bean>
                            </property>
                        </bean>
                    </property>







--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/how-to-append-data-to-IGFS-so-that-data-gets-saved-to-Hive-partitioned-table-tp15725p15887.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message