ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Integration of Spark and Ignite. Prototype.
Date Mon, 04 Dec 2017 23:03:43 GMT
Denis,

Nikolay was doing final changes and TC stabilization. I'm planning to do
final review this week, so hopefully we will merge the code soon.

-Val

On Mon, Dec 4, 2017 at 1:31 PM, Denis Magda <dmagda@apache.org> wrote:

> Nikolay, Val,
>
> Since we agreed to release the feature without the strategy support, can
> the current integration meet the world in 2.4 release? Please chime in this
> conversation:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Time-and-scope-for-Apache-Ignite-2-4-td24987.html
>
> —
> Denis
>
> > On Nov 28, 2017, at 5:42 PM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
> >
> > Denis,
> >
> > Agree. I will do the final review in next few days and merge the code.
> >
> > -Val
> >
> > On Tue, Nov 28, 2017 at 5:28 PM, Denis Magda <dmagda@apache.org> wrote:
> >
> >> Guys,
> >>
> >> Looking into the parallel discussion about the strategy support I would
> >> change my initial stance and support the idea of releasing the
> integration
> >> in its current state. Is the code ready to be merged into the master?
> Let’s
> >> concentrate on this first and handle the strategy support as a separate
> >> JIRA task. Agree?
> >>
> >> —
> >> Denis
> >>
> >>> On Nov 27, 2017, at 3:47 PM, Valentin Kulichenko <
> >> valentin.kulichenko@gmail.com> wrote:
> >>>
> >>> Nikolay,
> >>>
> >>> Let's estimate the strategy implementation work, and then decide
> weather
> >> to
> >>> merge the code in current state or not. If anything is unclear, please
> >>> start a separate discussion.
> >>>
> >>> -Val
> >>>
> >>> On Fri, Nov 24, 2017 at 5:42 AM, Николай Ижиков <
> nizhikov.dev@gmail.com>
> >>> wrote:
> >>>
> >>>> Hello, Val, Denis.
> >>>>
> >>>>> Personally, I think that we should release the integration only after
> >>>> the strategy is fully supported.
> >>>>
> >>>> I see two major reason to propose merge of DataFrame API
> implementation
> >>>> without custom strategy:
> >>>>
> >>>> 1. My PR is relatively huge, already. From my experience of
> interaction
> >>>> with Ignite community - the bigger PR becomes, the more time of
> >> commiters
> >>>> required to review PR.
> >>>> So, I propose to move smaller, but complete steps here.
> >>>>
> >>>> 2. It is not clear for me what exactly includes "custom strategy and
> >>>> optimization".
> >>>> Seems, that additional discussion required.
> >>>> I think, I can put my thoughts on the paper and start discussion right
> >>>> after basic implementation is done.
> >>>>
> >>>>> Custom strategy implementation is actually very important for this
> >>>> integration.
> >>>>
> >>>> Understand and fully agreed.
> >>>> I'm ready to continue work in that area.
> >>>>
> >>>> 23.11.2017 02:15, Denis Magda пишет:
> >>>>
> >>>> Val, Nikolay,
> >>>>>
> >>>>> Personally, I think that we should release the integration only after
> >> the
> >>>>> strategy is fully supported. Without the strategy we don’t really
> >> leverage
> >>>>> from Ignite’s SQL engine and introduce redundant data movement
> between
> >>>>> Ignite and Spark nodes.
> >>>>>
> >>>>> How big is the effort to support the strategy in terms of the amount
> of
> >>>>> work left? 40%, 60%, 80%?
> >>>>>
> >>>>> —
> >>>>> Denis
> >>>>>
> >>>>> On Nov 22, 2017, at 2:57 PM, Valentin Kulichenko <
> >>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>
> >>>>>> Nikolay,
> >>>>>>
> >>>>>> Custom strategy implementation is actually very important for this
> >>>>>> integration. Basically, it will allow to create a SQL query for
> Ignite
> >>>>>> and
> >>>>>> execute it directly on the cluster. Your current implementation only
> >>>>>> adds a
> >>>>>> new DataSource which means that Spark will fetch data in its own
> >> memory
> >>>>>> first, and then do most of the work (like joins for example). Does
> it
> >>>>>> make
> >>>>>> sense to you? Can you please take a look at this and provide your
> >>>>>> thoughts
> >>>>>> on how much development is implied there?
> >>>>>>
> >>>>>> Current code looks good to me though and I'm OK if the strategy is
> >>>>>> implemented as a next step in a scope of separate ticket. I will do
> >> final
> >>>>>> review early next week and will merge it if everything is OK.
> >>>>>>
> >>>>>> -Val
> >>>>>>
> >>>>>> On Thu, Oct 19, 2017 at 7:29 AM, Николай Ижиков <
> >> nizhikov.dev@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Hello.
> >>>>>>>
> >>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two
> >> Catalog
> >>>>>>>>
> >>>>>>> implementations and what is the difference?
> >>>>>>>
> >>>>>>> IgniteCatalog removed.
> >>>>>>>
> >>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have to
> be
> >>>>>>>>
> >>>>>>> set manually on SQLContext each time it's created....Is there any
> >> way to
> >>>>>>> automate this and improve usability?
> >>>>>>>
> >>>>>>> IgniteStrategy and IgniteOptimization are removed as it empty now.
> >>>>>>>
> >>>>>>> Actually, I think it makes sense to create a builder similar to
> >>>>>>>>
> >>>>>>> SparkSession.builder()...
> >>>>>>>
> >>>>>>> IgniteBuilder added.
> >>>>>>> Syntax looks like:
> >>>>>>>
> >>>>>>> ```
> >>>>>>> val igniteSession = IgniteSparkSession.builder()
> >>>>>>>   .appName("Spark Ignite catalog example")
> >>>>>>>   .master("local")
> >>>>>>>   .config("spark.executor.instances", "2")
> >>>>>>>   .igniteConfig(CONFIG)
> >>>>>>>   .getOrCreate()
> >>>>>>>
> >>>>>>> igniteSession.catalog.listTables().show()
> >>>>>>> ```
> >>>>>>>
> >>>>>>> Please, see updated PR - https://github.com/apache/
> ignite/pull/2742
> >>>>>>>
> >>>>>>> 2017-10-18 20:02 GMT+03:00 Николай Ижиков <nizhikov.dev@gmail.com
> >:
> >>>>>>>
> >>>>>>> Hello, Valentin.
> >>>>>>>>
> >>>>>>>> My answers is below.
> >>>>>>>> Dmitry, do we need to move discussion to Jira?
> >>>>>>>>
> >>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our
> >> codebase?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> As I mentioned earlier, to implement and override Spark Catalog
> one
> >>>>>>>> have
> >>>>>>>> to use internal(private) Spark API.
> >>>>>>>> So I have to use package `org.spark.sql.***` to have access to
> >> private
> >>>>>>>> class and variables.
> >>>>>>>>
> >>>>>>>> For example, SharedState class that stores link to ExternalCatalog
> >>>>>>>> declared as `private[sql] class SharedState` - i.e. package
> private.
> >>>>>>>>
> >>>>>>>> Can these classes reside under org.apache.ignite.spark instead?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> No, as long as we want to have our own implementation of
> >>>>>>>> ExternalCatalog.
> >>>>>>>>
> >>>>>>>> 2. IgniteRelationProvider contains multiple constants which I
> guess
> >> are
> >>>>>>>>>
> >>>>>>>> some king of config options. Can you describe the purpose of each
> of
> >>>>>>>> them?
> >>>>>>>>
> >>>>>>>> I extend comments for this options.
> >>>>>>>> Please, see my commit [1] or PR HEAD:
> >>>>>>>>
> >>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two
> >> Catalog
> >>>>>>>>>
> >>>>>>>> implementations and what is the difference?
> >>>>>>>>
> >>>>>>>> Good catch, thank you!
> >>>>>>>> After additional research I founded that only
> IgniteExternalCatalog
> >>>>>>>> required.
> >>>>>>>> I will update PR with IgniteCatalog remove in a few days.
> >>>>>>>>
> >>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op. What
> >> are
> >>>>>>>>>
> >>>>>>>> our plans on implementing them? Also, what exactly is planned in
> >>>>>>>> IgniteOptimization and what is its purpose?
> >>>>>>>>
> >>>>>>>> Actually, this is very good question :)
> >>>>>>>> And I need advice from experienced community members here:
> >>>>>>>>
> >>>>>>>> `IgniteOptimization` purpose is to modify query plan created by
> >> Spark.
> >>>>>>>> Currently, we have one optimization described in IGNITE-3084 [2]
> by
> >>>>>>>> you,
> >>>>>>>> Valentin :) :
> >>>>>>>>
> >>>>>>>> “If there are non-Ignite relations in the plan, we should fall
> back
> >> to
> >>>>>>>> native Spark strategies“
> >>>>>>>>
> >>>>>>>> I think we can go little further and reduce join of two Ignite
> >> backed
> >>>>>>>> Data Frames into single Ignite SQL query. Currently, this feature
> is
> >>>>>>>> unimplemented.
> >>>>>>>>
> >>>>>>>> *Do we need it now? Or we can postpone it and concentrates on
> basic
> >>>>>>>> Data
> >>>>>>>> Frame and Catalog implementation?*
> >>>>>>>>
> >>>>>>>> `Strategy` purpose, as you correctly mentioned in [2], is
> transform
> >>>>>>>> LogicalPlan into physical operators.
> >>>>>>>> I don’t have ideas how to use this opportunity. So I think we
> don’t
> >>>>>>>> need
> >>>>>>>> IgniteStrategy.
> >>>>>>>>
> >>>>>>>> Can you or anyone else suggest some optimization strategy to speed
> >> up
> >>>>>>>> SQL
> >>>>>>>> query execution?
> >>>>>>>>
> >>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have to
> >> be
> >>>>>>>>>
> >>>>>>>> set manually on SQLContext each time it's created....Is there any
> >> way
> >>>>>>>> to
> >>>>>>>> automate this and improve usability?
> >>>>>>>>
> >>>>>>>> These classes added to `extraOptimizations` when one using
> >>>>>>>> IgniteSparkSession.
> >>>>>>>> As far as I know, there is no way to automatically add these
> >> classes to
> >>>>>>>> regular SparkSession.
> >>>>>>>>
> >>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used in
> >>>>>>>>>
> >>>>>>>> IgniteCatalogExample but not in IgniteDataFrameExample, which is
> >>>>>>>> Confusing.
> >>>>>>>>
> >>>>>>>> DataFrame API is *public* Spark API. So anyone can provide
> >>>>>>>> implementation
> >>>>>>>> and plug it into Spark. That’s why IgniteDataFrameExample doesn’t
> >> need
> >>>>>>>> any
> >>>>>>>> Ignite specific session.
> >>>>>>>>
> >>>>>>>> Catalog API is *internal* Spark API. There is no way to plug
> custom
> >>>>>>>> catalog implementation into Spark [3]. So we have to use
> >>>>>>>> `IgniteSparkSession` that extends regular SparkSession and
> overrides
> >>>>>>>> links
> >>>>>>>> to `ExternalCatalog`.
> >>>>>>>>
> >>>>>>>> 7. To create IgniteSparkSession we first create IgniteContext. Is
> it
> >>>>>>>>>
> >>>>>>>> really needed? It looks like we can directly provide the
> >> configuration
> >>>>>>>> file; if IgniteSparkSession really requires IgniteContext, it can
> >>>>>>>> create it
> >>>>>>>> by itself under the hood.
> >>>>>>>>
> >>>>>>>> Actually, IgniteContext is base class for Ignite <-> Spark
> >> integration
> >>>>>>>> for now. So I tried to reuse it here. I like the idea to remove
> >>>>>>>> explicit
> >>>>>>>> usage of IgniteContext.
> >>>>>>>> Will implement it in a few days.
> >>>>>>>>
> >>>>>>>> Actually, I think it makes sense to create a builder similar to
> >>>>>>>>>
> >>>>>>>> SparkSession.builder()...
> >>>>>>>>
> >>>>>>>> Great idea! I will implement such builder in a few days.
> >>>>>>>>
> >>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for the
> >> case
> >>>>>>>>>
> >>>>>>>> when we don't have SQL configured on Ignite side?
> >>>>>>>>
> >>>>>>>> Yes, IgniteCacheRelation is Data Frame implementation for a
> >> key-value
> >>>>>>>> cache.
> >>>>>>>>
> >>>>>>>> I thought we decided not to support this, no? Or this is something
> >>>>>>>>> else?
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>> My understanding is following:
> >>>>>>>>
> >>>>>>>> 1. We can’t support automatic resolving key-value caches in
> >>>>>>>> *ExternalCatalog*. Because there is no way to reliably detect key
> >> and
> >>>>>>>> value
> >>>>>>>> classes.
> >>>>>>>>
> >>>>>>>> 2. We can support key-value caches in regular Data Frame
> >>>>>>>> implementation.
> >>>>>>>> Because we can require user to provide key and value classes
> >>>>>>>> explicitly.
> >>>>>>>>
> >>>>>>>> 8. Can you clarify the query syntax in
> >> IgniteDataFrameExample#nativeS
> >>>>>>>>>
> >>>>>>>> parkSqlFromCacheExample2?
> >>>>>>>>
> >>>>>>>> Key-value cache:
> >>>>>>>>
> >>>>>>>> key - java.lang.Long,
> >>>>>>>> value - case class Person(name: String, birthDate: java.util.Date)
> >>>>>>>>
> >>>>>>>> Schema of data frame for cache is:
> >>>>>>>>
> >>>>>>>> key - long
> >>>>>>>> value.name - string
> >>>>>>>> value.birthDate - date
> >>>>>>>>
> >>>>>>>> So we can select data from data from cache:
> >>>>>>>>
> >>>>>>>> SELECT
> >>>>>>>> key, `value.name`,  `value.birthDate`
> >>>>>>>> FROM
> >>>>>>>> testCache
> >>>>>>>> WHERE key >= 2 AND `value.name` like '%0'
> >>>>>>>>
> >>>>>>>> [1] https://github.com/apache/ignite/pull/2742/commits/faf3ed6fe
> >>>>>>>> bf417bc59b0519156fd4d09114c8da7
> >>>>>>>> [2] https://issues.apache.org/jira/browse/IGNITE-3084?focusedCom
> >>>>>>>> mentId=15794210&page=com.atlassian.jira.plugin.system.issuet
> >>>>>>>> abpanels:comment-tabpanel#comment-15794210
> >>>>>>>> [3] https://issues.apache.org/jira/browse/SPARK-17767?focusedCom
> >>>>>>>> mentId=15543733&page=com.atlassian.jira.plugin.system.issuet
> >>>>>>>> abpanels:comment-tabpanel#comment-15543733
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> 18.10.2017 04:39, Dmitriy Setrakyan пишет:
> >>>>>>>>
> >>>>>>>> Val, thanks for the review. Can I ask you to add the same comments
> >> to
> >>>>>>>> the
> >>>>>>>>
> >>>>>>>>> ticket?
> >>>>>>>>>
> >>>>>>>>> On Tue, Oct 17, 2017 at 3:20 PM, Valentin Kulichenko <
> >>>>>>>>> valentin.kulichenko@gmail.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Nikolay, Anton,
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I did a high level review of the code. First of all, impressive
> >>>>>>>>>> results!
> >>>>>>>>>> However, I have some questions/comments.
> >>>>>>>>>>
> >>>>>>>>>> 1. Why do we have org.apache.spark.sql.ignite package in our
> >>>>>>>>>> codebase?
> >>>>>>>>>> Can
> >>>>>>>>>> these classes reside under org.apache.ignite.spark instead?
> >>>>>>>>>> 2. IgniteRelationProvider contains multiple constants which I
> >> guess
> >>>>>>>>>> are
> >>>>>>>>>> some king of config options. Can you describe the purpose of
> each
> >> of
> >>>>>>>>>> them?
> >>>>>>>>>> 3. IgniteCatalog vs. IgniteExternalCatalog. Why do we have two
> >>>>>>>>>> Catalog
> >>>>>>>>>> implementations and what is the difference?
> >>>>>>>>>> 4. IgniteStrategy and IgniteOptimization are currently no-op.
> What
> >>>>>>>>>> are
> >>>>>>>>>> our
> >>>>>>>>>> plans on implementing them? Also, what exactly is planned in
> >>>>>>>>>> IgniteOptimization and what is its purpose?
> >>>>>>>>>> 5. I don't like that IgniteStrategy and IgniteOptimization have
> >> to be
> >>>>>>>>>> set
> >>>>>>>>>> manually on SQLContext each time it's created. This seems to be
> >> very
> >>>>>>>>>> error
> >>>>>>>>>> prone. Is there any way to automate this and improve usability?
> >>>>>>>>>> 6. What is the purpose of IgniteSparkSession? I see it's used
> >>>>>>>>>> in IgniteCatalogExample but not in IgniteDataFrameExample, which
> >> is
> >>>>>>>>>> confusing.
> >>>>>>>>>> 7. To create IgniteSparkSession we first create IgniteContext.
> Is
> >> it
> >>>>>>>>>> really
> >>>>>>>>>> needed? It looks like we can directly provide the configuration
> >>>>>>>>>> file; if
> >>>>>>>>>> IgniteSparkSession really requires IgniteContext, it can create
> >> it by
> >>>>>>>>>> itself under the hood. Actually, I think it makes sense to
> create
> >> a
> >>>>>>>>>> builder
> >>>>>>>>>> similar to SparkSession.builder(), it would be good if our APIs
> >> here
> >>>>>>>>>> are
> >>>>>>>>>> consistent with Spark APIs.
> >>>>>>>>>> 8. Can you clarify the query syntax
> >>>>>>>>>> inIgniteDataFrameExample#nativeSparkSqlFromCacheExample2?
> >>>>>>>>>> 9. Do I understand correctly that IgniteCacheRelation is for the
> >> case
> >>>>>>>>>> when
> >>>>>>>>>> we don't have SQL configured on Ignite side? I thought we
> decided
> >>>>>>>>>> not to
> >>>>>>>>>> support this, no? Or this is something else?
> >>>>>>>>>>
> >>>>>>>>>> Thanks!
> >>>>>>>>>>
> >>>>>>>>>> -Val
> >>>>>>>>>>
> >>>>>>>>>> On Tue, Oct 17, 2017 at 4:40 AM, Anton Vinogradov <
> >>>>>>>>>> avinogradov@gridgain.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Sounds awesome.
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'll try to review API & tests this week.
> >>>>>>>>>>>
> >>>>>>>>>>> Val,
> >>>>>>>>>>> Your review still required :)
> >>>>>>>>>>>
> >>>>>>>>>>> On Tue, Oct 17, 2017 at 2:36 PM, Николай Ижиков <
> >>>>>>>>>>> nizhikov.dev@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Yes
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> 17 окт. 2017 г. 2:34 PM пользователь "Anton Vinogradov" <
> >>>>>>>>>>>> avinogradov@gridgain.com> написал:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> So, it will be able to start regular spark and ignite
> clusters
> >>>>>>>>>>>>> and,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> using
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> peer classloading via spark-context, perform any DataFrame
> >> request,
> >>>>>>>>>>>>
> >>>>>>>>>>>>> correct?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, Oct 17, 2017 at 2:25 PM, Николай Ижиков <
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> nizhikov.dev@gmail.com>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello, Anton.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> An example you provide is a path to a master *local* file.
> >>>>>>>>>>>>>> These libraries are added to the classpath for each remote
> >> node
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> running
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> submitted job.
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Please, see documentation:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/
> >>>>>>>>>>>>>> spark/SparkContext.html#addJar(java.lang.String)
> >>>>>>>>>>>>>> http://spark.apache.org/docs/latest/api/java/org/apache/
> >>>>>>>>>>>>>> spark/SparkContext.html#addFile(java.lang.String)
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2017-10-17 13:10 GMT+03:00 Anton Vinogradov <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> avinogradov@gridgain.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> :
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Nikolay,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> With Data Frame API implementation there are no
> requirements
> >> to
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ignite files on spark worker nodes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> What do you mean? I see code like:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> spark.sparkContext.addJar(MAVEN_HOME +
> >>>>>>>>>>>>>>> "/org/apache/ignite/ignite-core/2.3.0-SNAPSHOT/ignite-
> >>>>>>>>>>>>>>> core-2.3.0-SNAPSHOT.jar")
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Mon, Oct 16, 2017 at 5:22 PM, Николай Ижиков <
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> nizhikov.dev@gmail.com>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello, guys.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I have created example application to run Ignite Data
> Frame
> >> on
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> standalone
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Spark cluster.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> With Data Frame API implementation there are no
> >> requirements to
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> have
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>> any
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ignite files on spark worker nodes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I ran this application on the free dataset: ATP tennis
> match
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> statistics.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> data - https://github.com/nizhikov/atp_matches
> >>>>>>>>>>>>>>>> app - https://github.com/nizhikov/ignite-spark-df-example
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Valentin, do you have a chance to look at my changes?
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> 2017-10-12 6:03 GMT+03:00 Valentin Kulichenko <
> >>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> :
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Nikolay,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Sorry for delay on this, got a little swamped lately. I
> >> will
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> do
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>> my
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> best
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> review the code this week.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> -Val
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Mon, Oct 9, 2017 at 11:48 AM, Николай Ижиков <
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> nizhikov.dev@gmail.com>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello, Valentin.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Did you have a chance to look at my changes?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Now I think I have done almost all required features.
> >>>>>>>>>>>>>>>>>> I want to make some performance test to ensure my
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>> work
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> properly with a significant amount of data.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> And I definitely need some feedback for my changes.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> 2017-10-09 18:45 GMT+03:00 Николай Ижиков <
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> nizhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>> :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello, guys.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Which version of Spark do we want to use?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 1. Currently, Ignite depends on Spark 2.1.0.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>    * Can be run on JDK 7.
> >>>>>>>>>>>>>>>>>>>    * Still supported: 2.1.2 will be released soon.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2. Latest Spark version is 2.2.0.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>    * Can be run only on JDK 8+
> >>>>>>>>>>>>>>>>>>>    * Released Jul 11, 2017.
> >>>>>>>>>>>>>>>>>>>    * Already supported by huge vendors(Amazon for
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> example).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> Note that in IGNITE-3084 I implement some internal Spark
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> API.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>> So It will take some effort to switch between Spark 2.1 and
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2.2
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> 2017-09-27 2:20 GMT+03:00 Valentin Kulichenko <
> >>>>>>>>>>>>>>>>>>> valentin.kulichenko@gmail.com>:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> I will review in the next few days.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> -Val
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> On Tue, Sep 26, 2017 at 2:23 PM, Denis Magda <
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> dmagda@apache.org
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hello Nikolay,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> This is good news. Finally this capability is coming
> to
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Ignite.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Val, Vladimir, could you do a preliminary review?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Answering on your questions.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 1. Yardstick should be enough for performance
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> measurements.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> As a
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Spark
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> user, I will be curious to know what’s the point of this
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> integration.
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Probably we need to compare Spark + Ignite and Spark +
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hive
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> or
> >>>>>>>>>>>>
> >>>>>>>>>>>> Spark +
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> RDBMS cases.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> 2. If Spark community is reluctant let’s include the
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> module
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> in
> >>>>>>>>>>>>
> >>>>>>>>>>>> ignite-spark integration.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> —
> >>>>>>>>>>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Sep 25, 2017, at 11:14 AM, Николай Ижиков <
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> nizhikov.dev@gmail.com>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Hello, guys.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Currently, I’m working on integration between Spark
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>> Ignite
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> [1].
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> For now, I implement following:
> >>>>>>>>>>>>>>>>>>>>>>   * Ignite DataSource implementation(
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> IgniteRelationProvider)
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>   * DataFrame support for Ignite SQL table.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>   * IgniteCatalog implementation for a transparent
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> resolving
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>> of
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ignites
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> SQL tables.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Implementation of it can be found in PR [2]
> >>>>>>>>>>>>>>>>>>>>>> It would be great if someone provides feedback for a
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> prototype.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I made some examples in PR so you can see how API
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> suppose
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> to
> >>>>>>>>>>>>
> >>>>>>>>>>>> be
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> used [3].
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> [4].
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> I need some advice. Can you help me?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 1. How should this PR be tested?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Of course, I need to provide some unit tests. But
> what
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>> scalability
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> tests, etc.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Maybe we need some Yardstick benchmark or similar?
> >>>>>>>>>>>>>>>>>>>>>> What are your thoughts?
> >>>>>>>>>>>>>>>>>>>>>> Which scenarios should I consider in the first
> place?
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> 2. Should we provide Spark Catalog implementation
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> inside
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>> Ignite
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> codebase?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> A current implementation of Spark Catalog based on
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> *internal
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>> Spark
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> API*.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Spark community seems not interested in making Catalog
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> API
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> public
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>> or
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> including Ignite Catalog in Spark code base [5], [6].
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> *Should we include Spark internal API implementation
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> inside
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>> Ignite
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> code
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> base?*
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Or should we consider to include Catalog
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> implementation
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>> in
> >>>>>>>>>>>
> >>>>>>>>>>> some
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> external
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> module?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> That will be created and released outside Ignite?(we
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> still
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> can
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> support
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> develop it inside Ignite community).
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> [1] https://issues.apache.org/
> jira/browse/IGNITE-3084
> >>>>>>>>>>>>>>>>>>>>>> [2] https://github.com/apache/ignite/pull/2742
> >>>>>>>>>>>>>>>>>>>>>> [3] https://github.com/apache/
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff-
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>> f4ff509cef3018e221394474775e0905
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> [4] https://github.com/apache/
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> ignite/pull/2742/files#diff-
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>> f2b670497d81e780dfd5098c5dd8a89c
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> [5] http://apache-spark-developers-list.1001551.n3.
> >>>>>>>>>>>>>>>>>>>>>> nabble.com/Spark-Core-Custom-
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Catalog-Integration-between-
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>> Apache-Ignite-and-Apache-Spark-td22452.html
> >>>>>>>>>>>>
> >>>>>>>>>>>>> [6] https://issues.apache.org/jira/browse/SPARK-17767
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>>>>> Nikolay Izhikov
> >>>>>>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>>> Nikolay Izhikov
> >>>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Nikolay Izhikov
> >>>>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>> Nikolay Izhikov
> >>>>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Nikolay Izhikov
> >>>>>>>>>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Nikolay Izhikov
> >>>>>>> NIzhikov.dev@gmail.com
> >>>>>>>
> >>>>>>>
> >>>>>
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message