hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [incubator-hudi] vinothchandar commented on a change in pull request #1348: HUDI-597 Enable incremental pulling from defined partitions
Date Sun, 23 Feb 2020 05:55:07 GMT
vinothchandar commented on a change in pull request #1348: HUDI-597 Enable incremental pulling
from defined partitions
URL: https://github.com/apache/incubator-hudi/pull/1348#discussion_r382966852
 
 

 ##########
 File path: hudi-spark/src/main/scala/org/apache/hudi/IncrementalRelation.scala
 ##########
 @@ -100,17 +101,22 @@ class IncrementalRelation(val sqlContext: SQLContext,
         .get, classOf[HoodieCommitMetadata])
       fileIdToFullPath ++= metadata.getFileIdAndFullPaths(basePath).toMap
     }
+    val pathGlobPattern = optParams.getOrElse(DataSourceReadOptions.INCR_PATH_GLOB_OPT_KEY,
"")
+    val filteredFullPath = if(!pathGlobPattern.equals("")) {
+      val globMatcher = new GlobPattern("*" + pathGlobPattern)
 
 Review comment:
   should we leave the `*` to the user? i.e let the user pass in `*` if needed? or is that
needed for the matching...
   
   I am not familiar with this class per se.. 
   
   Also http://hadoop.apache.org/docs/r2.8.0/api/allclasses-noframe.html does not seem to
have `GlobPattern` is this class still around.. Was a bit confused by that.. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message