ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vkulichenko <valentin.kuliche...@gmail.com>
Subject Re: How use Hadoop accelerator
Date Thu, 10 Sep 2015 01:09:55 GMT
Hi,

Before making any other posts on this forum, please subscribe to the Apache
Ignite user mailing list, otherwise your messages are not forwarded there.
Refer to the instruction here:
http://apache-ignite-users.70518.x6.nabble.com/mailing_list/MailingListOptions.jtp?forum=1


jarredfox wrote
> I use hadoop 2.4.1 and Ignite 1.3.0 for test . 
>  
> I am trying to read a simple test for IGFS compared with HDFS 
> 
> I've configured the Hadoop FileSystem Cache (secondaryFileSystem) 
> 
> but, I got the same result. (No performance improvement.)
> 
> ...
> 
> Reading Testcode (HDFS & IGFS)
> HDFSTest.java
> <http://apache-ignite-users.70518.x6.nabble.com/file/n1314/HDFSTest.java>  
> 
> IGFSTest.java
> <http://apache-ignite-users.70518.x6.nabble.com/file/n1314/IGFSTest.java>  
> 
> Is there somewhere where I wrong or do any more?

First of all you should make sure that your test is reading the file that is
already in memory. When you read some piece of data for the first time, it
will still have to go to Hadoop, but with IGFS configured it will end up in
in-memory cache. Also note that you're using remote client which has to
transfer data over network, so I would not expect much performance
improvement in this test, because essentially you're comparing TCP vs file
I/O.

Can you describe your use case? Are you running any Hadoop mapreduce jobs?
Ignite Hadoop Accelerator is shipped with its own jobtracker implementation
which allows to increase jobs performance without any code changes. I would
recommend to take a look at documentation [1] and the blog by Konstantin
Boudnik where he demonstrates how Hadoop job can be accelerated with the
help of Ignite [2].

Let us know if you have more questions.

[1] https://apacheignite.readme.io/docs/hadoop-accelerator
[2] http://drcos.boudnik.org/2015/05/30-time-faster-hadoop-mapreduce.html

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-use-Hadoop-accelerator-tp1314p1327.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message