Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 18 May 2017 07:20:04 +0000 (UTC)
From: "Sahana HA (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13073033.1495091947000.238365.1495092004120@Atlassian.JIRA>
In-Reply-To: <JIRA.13073033.1495091947000@Atlassian.JIRA>
References: <JIRA.13073033.1495091947000@Atlassian.JIRA> <JIRA.13073033.1495091947502@jira-lw-us.apache.org>
Subject: [jira] [Created] (SPARK-20794) Spark show() command on dataset does
 not retrieve consistent rows from DASHDB data source
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 18 May 2017 07:20:09 -0000

Sahana HA created SPARK-20794:
---------------------------------

             Summary: Spark show() command on dataset does not retrieve consistent rows from DASHDB data source
                 Key: SPARK-20794
                 URL: https://issues.apache.org/jira/browse/SPARK-20794
             Project: Spark
          Issue Type: Question
          Components: Spark Core
    Affects Versions: 2.0.0
            Reporter: Sahana HA
            Priority: Minor


When the user creates the dataframe from DASHDB data source (which is a relational database) and executes df.show(5) it returns different result sets or rows during each execution. We are aware that show(5) will pick the first 5 rows from existing partition and hence it is not guaranteed to be consistent across each execution. 

However when we try the same show(5) command against S3 storage or bluemixobject store (non-relational data source) we always get the same result sets or rows in order, across each execution.

We just wanted to confirm why the difference between DASHDB and other data source like S3/Bluemixobjectstore ? Is the issue with spark or DASHDB alone ? or is the inconsistent rows behavior is there for all relational data source ?

Repro snippet:

-- Load the data from dashdb
val dashdb = sqlContext.read.format("packageName").options(dashdbreadOptions).load


-- execution #1

dashdb.show(5)

+--------------------+------------+-----------------+-------+-----+-------------+------+---+--------------+------------+
|        PRODUCT_LINE|PRODUCT_TYPE|CUST_ORDER_NUMBER|   CITY|STATE|      COUNTRY|GENDER|AGE|MARITAL_STATUS|  PROFESSION|
+--------------------+------------+-----------------+-------+-----+-------------+------+---+--------------+------------+
|Personal Accessories|     Eyewear|           107861|Rutland|   VT|United States|     F| 39|       Married|       Sales|
|   Camping Equipment|    Lanterns|           189003| Sydney|  NSW|    Australia|     F| 20|        Single| Hospitality|
|   Camping Equipment|Cooking Gear|           107863| Sydney|  NSW|    Australia|     F| 20|        Single| Hospitality|
|Personal Accessories|     Eyewear|           189005|Villach|   NA|      Austria|     F| 37|       Married|Professional|
|Personal Accessories|     Eyewear|           107865|Villach|   NA|      Austria|     F| 37|       Married|Professional|
+--------------------+------------+-----------------+-------+-----+-------------+------+---+--------------+------------+
only showing top 5 rows


-- execution #2


dashdb.show(5)

+--------------------+------------+-----------------+------------+-----+--------------+------+---+--------------+-----------+
|        PRODUCT_LINE|PRODUCT_TYPE|CUST_ORDER_NUMBER|        CITY|STATE|       COUNTRY|GENDER|AGE|MARITAL_STATUS| PROFESSION|
+--------------------+------------+-----------------+------------+-----+--------------+------+---+--------------+-----------+
|Mountaineering Eq...|       Tools|           112835|  Portsmouth|   NA|United Kingdom|     M| 24|        Single|      Other|
|   Camping Equipment|Cooking Gear|           193902|Jacksonville|   FL| United States|     F| 22|        Single|Hospitality|
|   Camping Equipment|       Packs|           112837|Jacksonville|   FL| United States|     F| 22|        Single|Hospitality|
|Mountaineering Eq...|        Rope|           193904|Jacksonville|   FL| United States|     F| 31|       Married|      Other|
|      Golf Equipment|     Putters|           112839|Jacksonville|   FL| United States|     F| 31|       Married|      Other|
+--------------------+------------+-----------------+------------+-----+--------------+------+---+--------------+-----------+
only showing top 5 rows


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org