ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shenson Joseph (Jira)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-12054) [Umbrella][Spark] Upgrade Spark module to 2.4
Date Mon, 25 Nov 2019 13:18:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-12054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981547#comment-16981547

Shenson Joseph commented on IGNITE-12054:

[~zaleslaw] [~nizhikov]

Please see my ignite-config file below.Can you please confirm if I am missing anything here?


<?xml version="1.0" encoding="UTF-8"?>

 Licensed to the Apache Software Foundation (ASF) under one or more
 contributor license agreements. See the NOTICE file distributed with
 this work for additional information regarding copyright ownership.
 The ASF licenses this file to You under the Apache License, Version 2.0
 (the "License"); you may not use this file except in compliance with
 the License. You may obtain a copy of the License at


Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 See the License for the specific language governing permissions and
 limitations under the License.

 Configuration example with Kubernetes IP finder enabled.
<beans xmlns="http://www.springframework.org/schema/beans"

<bean class="org.apache.ignite.configuration.IgniteConfiguration">
 <!-- Enabling Apache Ignite Persistent Store. -->
 <property name="dataStorageConfiguration">
 <bean class="org.apache.ignite.configuration.DataStorageConfiguration">
 <property name="defaultDataRegionConfiguration">
 <bean class="org.apache.ignite.configuration.DataRegionConfiguration">
 <property name="persistenceEnabled" value="true"/>
 <property name="igniteInstanceName" value="shared-grid"/>
 <property name="peerClassLoadingEnabled" value="true"/>
 <property name="clientMode" value="false"/>
 <!-- <property name="consistentId" value="rtanalytics"/>-->

<!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
 <property name="discoverySpi">
 <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
 <property name="ipFinder">
 Enables Kubernetes IP finder with default settings.
 <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder">
 <property name="namespace" value="nsp"/>
 <property name="serviceName" value="nsp-ignite"/>
 <property name="executorConfiguration">
 <bean class="org.apache.ignite.configuration.ExecutorConfiguration">
 <property name="name" value="baseline-window-optimizer"/>
 <property name="size" value="5"/> <!-- size == # of baseline periods -->

<property name="serviceConfiguration">
 <bean class="org.apache.ignite.services.ServiceConfiguration">
 <property name="name" value="BaselineWindowOptimizer"/>
 <!-- clusterSingleton = (maxPerNodeCount == 1 && totalCount == 1); -->
 <property name="maxPerNodeCount" value="1"/>
 <property name="totalCount" value="1"/>
 <property name="service">
 <ref bean="windowOptimizationService"/>

 <bean id="windowOptimizationService" class="com.nokia.nsp.rtanalytics.baseline.common.compute.WindowOptimizerService">
 <!-- TIME ZONE ; if empty defaults to System TimeZone-->
 <constructor-arg type="java.lang.String" value = "America/Toronto"/>

> [Umbrella][Spark] Upgrade Spark module to 2.4
> ---------------------------------------------
>                 Key: IGNITE-12054
>                 URL: https://issues.apache.org/jira/browse/IGNITE-12054
>             Project: Ignite
>          Issue Type: New Feature
>          Components: spark
>    Affects Versions: 2.8
>            Reporter: Denis A. Magda
>            Assignee: Alexey Zinoviev
>            Priority: Blocker
>              Labels: await
>             Fix For: 2.8
>         Attachments: ignite-spark-patch-new.diff
>          Time Spent: 20m
>  Remaining Estimate: 0h
> Users can't use APIs that are already available in Spark 2.4:
> https://stackoverflow.com/questions/57392143/persisting-spark-dataframe-to-ignite
> Let's upgrade Spark from 2.3 to 2.4 until we extract the Spark Integration as a separate
module that can support multiple Spark versions.

This message was sent by Atlassian Jira

View raw message