ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vijayendra bhati <veejayend...@yahoo.com.INVALID>
Subject Re: Spark/YARN
Date Wed, 04 May 2016 16:24:48 GMT
Hi Naden,I am just wondering why you want to start ignite node in Yarn.You can stary the independent
ignite node.What benefit you are getting by making part of yarn cluster.I am asking this because
i too had the same queries but latet on i switched to starting independent ignite node on
the same nodes where my node managers are running.I guess this too will give data locality
etc.Also you can create RDD and save data directly to cache using RDD,you dont need to explicitly
invoke ignite() or start().I can share the code if you want.
Regards,Vij

Sent from Yahoo Mail on Android 
 
  On Wed, 4 May, 2016 at 5:19 am, Franciscus, Naden<Naden.Franciscus@team.telstra.com>
wrote:   
Hey Alexey,

Sorry I meant HDFS :)

It would be good for IgniteContext to have the option to read
configuration from HDFS.
Again many users of Hadoop aren't going to have the ability to modify data
nodes to install
Ignite. But I will raise that as a separate issue.

The big issue is that there is a lot of code like this in documentation
and elsewhere:
ignite.cache("myCache").put(personId, person);

Which we can't use in any RDD/DF map, filter etc because neither
Ignite/IgniteCache are serializable.


Cheers,
Naden





On 4/05/2016, 5:42 AM, "Alexey Goncharuk" <alexey.goncharuk@gmail.com>
wrote:

>Hi Naden,
>
> But the IgniteContext within Spark doesn't allow you to read
>configuration
>> files from YARN.
>>
>
>I am a little bit confused. Ignite can be configured via basic Spring XML,
>and you can definitely read those XML files off HDFS or any other source.
>Is there any reason why XML does not work for you?
>
>Now it is impossible to make IgniteConfiguration serializable because it
>contains such components as SPIs, which are non-serializable by it's
>nature.
  

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message