hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <>
Subject [GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1333: [HUDI-589][DOCS] Fix querying_data page
Date Fri, 14 Feb 2020 02:44:38 GMT
vinothchandar commented on a change in pull request #1333: [HUDI-589][DOCS] Fix querying_data

 File path: docs/_docs/
 @@ -84,55 +102,53 @@ using the hive session property for incremental queries: `set
 would ensure Map Reduce execution is chosen for a Hive query, which combines partitions (comma
 separated) and calls InputFormat.listStatus() only once with all those partitions.
-## Spark
+## Spark datasource
-Spark provides much easier deployment & management of Hudi jars and bundles into jobs/notebooks.
At a high level, there are two ways to access Hudi tables in Spark.
+Hudi COPY_ON_WRITE tables can be queried via Spark datasource similar to how standard datasources
work (e.g: ``). 
+Both snapshot querying and incremental querying are supported here. Typically spark jobs
require adding `--jars <path to jar>/hudi-spark-bundle_2.11:0.5.1-incubating`
+to classpath of drivers and executors. Refer [building Hudi](
for build instructions. 
+When using spark shell instead of --jars, --packages can also be used to fetch the hudi-spark-bundle
like this: `--packages org.apache.hudi:hudi-spark-bundle_2.11:0.5.1-incubating`
+For sample setup, refer to [Setup spark-shell in quickstart](/docs/quick-start-guide.html#setup-spark-shell).
- - **Hudi DataSource** : Supports Read Optimized, Incremental Pulls similar to how standard
datasources (e.g: ``) work.
- - **Read as Hive tables** : Supports all three query types, including the snapshot queries,
relying on the custom Hudi input formats again like Hive.
- In general, your spark job needs a dependency to `hudi-spark` or `hudi-spark-bundle_2.*-x.y.z.jar`
needs to be on the class path of driver & executors (hint: use `--jars` argument)
+## Spark SQL
+Supports all query types across both Hudi table types, relying on the custom Hudi input formats
again like Hive. 
+Typically notebook users and spark-shell users leverage spark sql for querying Hudi tables.
Please add hudi-spark-bundle 
+as described above via --jars or --packages.
-### Read optimized query
-Pushing a path filter into sparkContext as follows allows for read optimized querying of
a Hudi hive table using SparkSQL. 
-This method retains Spark built-in optimizations for reading Parquet files like vectorized
reading on Hudi tables.
-spark.sparkContext.hadoopConfiguration.setClass("mapreduce.input.pathFilter.class", classOf[org.apache.hudi.hadoop.HoodieROTablePathFilter],
-If you prefer to glob paths on DFS via the datasource, you can simply do something like below
to get a Spark dataframe to work with. 
+### Snapshot query {#spark-snapshot-query}
+By default, Spark SQL will try to use its own parquet support instead of Hive SerDe when
reading from Hive metastore parquet tables. 
 Review comment:
   own parquet reader

This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

With regards,
Apache Git Services

View raw message