spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jose-torres <...@git.apache.org>
Subject [GitHub] spark pull request #22009: [SPARK-24882][SQL] improve data source v2 API
Date Wed, 08 Aug 2018 16:05:05 GMT
Github user jose-torres commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22009#discussion_r208641014
  
    --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/streaming/StreamingReadSupport.java
---
    @@ -0,0 +1,49 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.sql.sources.v2.reader.streaming;
    +
    +import org.apache.spark.sql.sources.v2.reader.ReadSupport;
    +
    +/**
    + * A base interface for streaming read support. This is package private and is invisible
to data
    + * sources. Data sources should implement concrete streaming read support interfaces:
    + * {@link MicroBatchReadSupport} or {@link ContinuousReadSupport}.
    + */
    +interface StreamingReadSupport extends ReadSupport {
    +
    +  /**
    +   * Returns the initial offset for a streaming query to start reading from. Note that
the
    +   * streaming data source should not assume that it will start reading from its
    +   * {@link #initialOffset()} value: if Spark is restarting an existing query, it will
restart from
    +   * the check-pointed offset rather than the initial one.
    +   */
    +  Offset initialOffset();
    +
    +  /**
    +   * Deserialize a JSON string into an Offset of the implementation-defined offset type.
    +   *
    +   * @throws IllegalArgumentException if the JSON does not encode a valid offset for
this reader
    +   */
    +  Offset deserializeOffset(String json);
    --- End diff --
    
    As I said, I'm fine with defining arbitrary JSON strings as the single non-customizable
offset type, if you think that would be better. (I think they would have to be strings, because
making a JSON object the type would mean packaging some JSON library into the API.) I don't
think it would ever be correct to have an Offset class which doesn't trivially reduce to a
key-value map.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message