ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Load data from Hadoop
Date Thu, 28 Jul 2016 08:24:27 GMT
Hi Labard,

Could you please explain your use case? Depending on your requirements we
could either suggest you to use some Ignite functionallity, or may be
resort to our Hadoop Accelerator module:
https://apacheignite-fs.readme.io/docs/hadoop-accelerator

Vladimir.

On Tue, Jul 26, 2016 at 6:26 PM, Labard <daivanov@at-consulting.ru> wrote:

> Hello.
> I'm trying load data from Hadoop(20G) to Ignite. Can anybode tell how
> should
> I do this?
>
> I have 5 servers in cluster. There are 8 ignite nodes and one hadoop node
> on
> each server.
>
> Now I have one client node which take data by jdbc(using Hive) from Hadoop
> and put it by IgniteStream.addData() into Ignite.
> But it is too slow. I assume because all data go through Hadoop nameNode
> and
> then load by one client node.
>
> If I will use loadCache instead of IgniteDataStream, I apprehend that Hive
> database can fall because of 40 queries for 20G files.
>
> In ideal I want to get data straight from Hadoop dataNodes and process a
> small piece on each Ignite node.
> Maybe I can use IGFS integrated with Hadoop, IgniteHadoopSecondryFileSystem
> to achieve this result?
>
> Sincerely, Dmitry
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Load-data-from-Hadoop-tp6544.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

Mime
View raw message