hudi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From vinoy...@apache.org
Subject [incubator-hudi] branch hudi_test_suite_refactor updated (5f22849 -> 7a0794a)
Date Thu, 13 Feb 2020 10:38:44 GMT
This is an automated email from the ASF dual-hosted git repository.

vinoyang pushed a change to branch hudi_test_suite_refactor
in repository https://gitbox.apache.org/repos/asf/incubator-hudi.git.


    omit 5f22849  [HUDI-592] Remove duplicated dependencies in the pom file of test suite
module
    omit 044759a  [HUDI-503] Add hudi test suite documentation into the README file of the
test suite module (#1191)
    omit b66ba8d  [MINOR] Fix TestHoodieTestSuiteJob#testComplexDag failure
    omit eaa4fb0  [HUDI-441] Rename WorkflowDagGenerator and some class names in test package
    omit 33246c4  Fixed resource leak in HiveTestService about hive meta store
    omit 7d67d5e  [HUDI-591] Support Spark version upgrade
    omit 0456214  [HUDI-442] Fix TestComplexKeyGenerator#testSingleValueKeyGenerator and testMultipleValueKeyGenerator
NPE
    omit f8d25b1  [MINOR] Fix compile error about the deletion of HoodieActiveTimeline#createNewCommitTime
    omit 53e73e5  [HUDI-391] Rename module name from hudi-bench to hudi-test-suite and fix
some checkstyle issues (#1102)
    omit d5549f5  [HUDI-394] Provide a basic implementation of test suite
     add 2bb0c21  Fix conversion of Spark struct type to Avro schema
     add 9b2944a  [MINOR] Refactor unnecessary boxing inside TypedProperties code (#1227)
     add 87fdb76  Adding util methods to assist in adding deletion support to Quick Start
     add 2b2f23a  Fixing delete util method
     add 2248fd9  Fixing checkstyle issues
     add 7aa3ce3  [MINOR] Fix redundant judgment statement (#1231)
     add dd09abb  [HUDI-335] Improvements to DiskBasedMap used by ExternalSpillableMap, for
write and random/sequential read paths, by introducing bufferedRandmomAccessFile
     add 1daba24  Add GlobalDeleteKeyGenerator
     add b39458b  [MINOR] Make constant fields final in HoodieTestDataGenerator (#1234)
     add 8a3a503  [MINOR] Fix missing @Override annotation on BufferedRandomAccessFile method
(#1236)
     add c2c0f6b  [HUDI-509] Renaming code in sync with cWiki restructuring (#1212)
     add baa6b5e  [HUDI-537] Introduce `repair overwrite-hoodie-props` CLI command (#1241)
     add 0a07752  [HUDI-527] scalastyle-maven-plugin moved to pluginManagement as it is only
used in hoodie-spark and hoodie-cli modules.
     add 923e2b4  [HUDI-535] Ensure Compaction Plan is always written in .aux folder to avoid
0.5.0/0.5.1 reader-writer compatibility issues (#1229)
     add 292c1e2  [HUDI-238] Make Hudi support Scala 2.12 (#1226)
     add 5471d8f  [MINOR] Add toString method to TimelineLayoutVersion to make it more readable
(#1244)
     add 3f4966d  [MINOR] Fix PMC in DOAP] (#1247)
     add d0ee95e  [HUDI-552] Fix the schema mismatch in Row-to-Avro conversion (#1246)
     add 9489d0f  [HUDI-551] Abstract a test case class for DFS Source to make it extensible
(#1239)
     add 7087e7d  [HUDI-556] Add lisence for PR#1233
     add ba54a7e  [HUDI-559] : Make the timeline layout version default to be null version
     add 6e59c1c  Moving to 0.5.2-SNAPSHOT on master branch.
     add 924bf51  [MINOR] Download KEYS file when validating release candidate (#1259)
     add b6e2993  [MINOR] Update the javadoc of HoodieTableMetaClient#scanFiles (#1263)
     add a54535e  [MINOR] Fix invalid maven repo address (#1265)
     add a46fea9  [MINOR] Change deploy_staging_jars script to take in scala version (#1269)
     add 8e3d81c  [MINOR] Change deploy_staging_jars script to take in scala version (#1270)
     add ed54eb2  [MINOR] Add missing licenses (#1271)
     add fc8d4a7  [MINOR] fix license issue (#1273)
     add 1e79cbc  [HUDI-549] update Github README with instructions to build with Scala 2.12
(#1275)
     add cdb028f  [MINOR] Fix missing groupId / version property of dependency
     add 56a4e0d  [MINOR] Fix invalid issue url & quickstart url (#1282)
     add 362a9b9  [MINOR] Remove junit-dep dependency
     add c06ec8b  [MINOR] Fix assigning to configuration more times (#1291)
     add 6f34be1  HUDI-117 Close file handle before throwing an exception due to append failure.
Add test cases to handle/verify stage failure scenarios.
     add 652224e  [HUDI-578] Trim recordKeyFields and partitionPathFields in ComplexKeyGenerator
(#1281)
     add f27c7a1  [HUDI-564] Added new test cases for HoodieLogFormat and HoodieLogFormatVersion.
     add 5b7bb14  [HUDI-583] Code Cleanup, remove redundant code, and other changes (#1237)
     add 0026234  [MINOR] Updated DOAP with 0.5.1 release (#1300)
     add fcf9e4a  [MINOR] Updated DOAP with 0.5.1 release (#1301)
     add d07ac58  Increase test coverage for HoodieReadClient
     add 347e297  [HUDI-596] Close KafkaConsumer every time (#1303)
     add 594da28  [HUDI-595] code cleanup, refactoring code out of PR# 1159 (#1302)
     add 4de0fcf  [HUDI-566] Added new test cases for class HoodieTimeline, HoodieDefaultTimeline
and HoodieActiveTimeline.
     add 425e3e6  [HUDI-585] Optimize the steps of building with scala-2.12 (#1293)
     add 46842f4  [MINOR] Remove the declaration of thrown RuntimeException (#1305)
     add c1516df  [HUDI-499] Allow update partition path with GLOBAL_BLOOM (#1187)
     add 462fd02  [HUDI-571] Add 'commits show archived' command to CLI
     add 1fb0b00  [HUDI-570] - Improve test coverage for FSUtils.java
     add d26dc0b  [HUDI-587] Fixed generation of jacoco coverage reports.
     add 5fdf5a1  [HUDI-560] Remove legacy IdentityTransformer (#1264)
     add 578efa2  [HUDI-582] Update NOTICE year
     add e5a69ed  [HUDI-478] Fix too many files with unapproved license when execute build_local_docker_images
(#1323)
     add d2c872e  [HUDI-605] Avoid calculating the size of schema redundantly (#1317)
     add 63b4216  CLI - add option to print additional commit metadata
     add 285c691  [HUDI-394] Provide a basic implementation of test suite
     add 43fbc0c  [HUDI-391] Rename module name from hudi-bench to hudi-test-suite and fix
some checkstyle issues (#1102)
     add 3471112  [MINOR] Fix compile error about the deletion of HoodieActiveTimeline#createNewCommitTime
     add 99b2ae0  [HUDI-442] Fix TestComplexKeyGenerator#testSingleValueKeyGenerator and testMultipleValueKeyGenerator
NPE
     add 1136002  [HUDI-591] Support Spark version upgrade
     add aeac933  Fixed resource leak in HiveTestService about hive meta store
     add e175677  [HUDI-441] Rename WorkflowDagGenerator and some class names in test package
     add 4eafca9  [MINOR] Fix TestHoodieTestSuiteJob#testComplexDag failure
     add 1f64ca0  [HUDI-503] Add hudi test suite documentation into the README file of the
test suite module (#1191)
     add e064262  [HUDI-592] Remove duplicated dependencies in the pom file of test suite
module
     add 7a0794a  Fix compile error after rebasing the branch

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (5f22849)
            \
             N -- N -- N   refs/heads/hudi_test_suite_refactor (7a0794a)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 LICENSE                                            |  10 +
 NOTICE                                             |   2 +-
 README.md                                          |  18 +-
 doap_HUDI.rdf                                      |  10 +-
 docker/demo/config/kafka-source.properties         |   4 +-
 docker/demo/hive-batch1.commands                   |   4 +-
 docker/demo/hive-batch2-after-compaction.commands  |   4 +-
 docker/demo/hive-table-check.commands              |   2 +-
 docker/demo/presto-batch1.commands                 |  22 +-
 .../demo/presto-batch2-after-compaction.commands   |  22 +-
 docker/demo/presto-table-check.commands            |  18 +
 docker/demo/sparksql-batch1.commands               |   4 +-
 docker/demo/sparksql-batch2.commands               |   4 +-
 docker/demo/sparksql-incremental.commands          |  10 +-
 docker/hoodie/hadoop/base/pom.xml                  |   2 +-
 docker/hoodie/hadoop/datanode/pom.xml              |   2 +-
 docker/hoodie/hadoop/historyserver/pom.xml         |   2 +-
 docker/hoodie/hadoop/hive_base/pom.xml             |   7 +-
 docker/hoodie/hadoop/namenode/pom.xml              |   2 +-
 docker/hoodie/hadoop/pom.xml                       |   4 +-
 docker/hoodie/hadoop/prestobase/pom.xml            |   3 +-
 docker/hoodie/hadoop/spark_base/Dockerfile         |   2 +-
 docker/hoodie/hadoop/spark_base/pom.xml            |   2 +-
 docker/hoodie/hadoop/sparkadhoc/pom.xml            |   2 +-
 docker/hoodie/hadoop/sparkmaster/pom.xml           |   2 +-
 docker/hoodie/hadoop/sparkworker/pom.xml           |   2 +-
 hudi-cli/pom.xml                                   |  21 +-
 .../apache/hudi/cli/commands/CommitsCommand.java   | 163 +++++++-
 .../hudi/cli/commands/CompactionCommand.java       |   6 +-
 .../hudi/cli/commands/FileSystemViewCommand.java   |  42 +-
 .../apache/hudi/cli/commands/RepairsCommand.java   |  38 ++
 .../apache/hudi/cli/commands/SparkEnvCommand.java  |   3 +-
 .../scala/org/apache/hudi/cli/DedupeSparkJob.scala |   6 +-
 hudi-client/pom.xml                                |   6 +-
 .../org/apache/hudi/CompactionAdminClient.java     |   8 +-
 .../java/org/apache/hudi/HoodieReadClient.java     |   6 +-
 .../java/org/apache/hudi/HoodieWriteClient.java    |  23 +-
 .../org/apache/hudi/client/utils/ClientUtils.java  |   3 +-
 .../org/apache/hudi/config/HoodieIndexConfig.java  |  18 +
 .../org/apache/hudi/config/HoodieWriteConfig.java  |   5 +
 .../apache/hudi/index/bloom/HoodieBloomIndex.java  |   4 +-
 .../hudi/index/bloom/HoodieGlobalBloomIndex.java   |  23 +-
 .../org/apache/hudi/index/hbase/HBaseIndex.java    |  23 +-
 .../org/apache/hudi/io/HoodieAppendHandle.java     |   9 +-
 .../java/org/apache/hudi/io/HoodieCleanHelper.java |  12 +-
 .../org/apache/hudi/io/HoodieCommitArchiveLog.java |   6 +-
 .../org/apache/hudi/io/HoodieKeyLookupHandle.java  |   4 +-
 .../java/org/apache/hudi/io/HoodieMergeHandle.java |  16 +-
 .../org/apache/hudi/io/HoodieRangeInfoHandle.java  |   4 +-
 .../java/org/apache/hudi/io/HoodieReadHandle.java  |   8 +-
 ...r.java => HoodieMergeOnReadTableCompactor.java} |  18 +-
 .../io/compact/strategy/CompactionStrategy.java    |  16 +-
 .../LogFileSizeBasedCompactionStrategy.java        |   4 +-
 .../main/java/org/apache/hudi/metrics/Metrics.java |  17 +-
 .../apache/hudi/table/HoodieCopyOnWriteTable.java  |  16 +-
 .../apache/hudi/table/HoodieMergeOnReadTable.java  |  29 +-
 .../java/org/apache/hudi/table/HoodieTable.java    |  10 +-
 .../java/org/apache/hudi/TestAsyncCompaction.java  |  63 +--
 .../src/test/java/org/apache/hudi/TestCleaner.java |  57 +--
 .../java/org/apache/hudi/TestClientRollback.java   |  24 +-
 .../org/apache/hudi/TestCompactionAdminClient.java |   4 +-
 .../java/org/apache/hudi/TestHoodieClientBase.java |   7 +-
 .../hudi/TestHoodieClientOnCopyOnWriteStorage.java |  24 +-
 .../java/org/apache/hudi/TestHoodieReadClient.java |  38 ++
 .../apache/hudi/common/HoodieClientTestUtils.java  |  10 +-
 .../hudi/common/HoodieTestDataGenerator.java       |  62 +--
 .../hudi/func/TestBoundedInMemoryExecutor.java     |   2 +-
 .../apache/hudi/func/TestBoundedInMemoryQueue.java |  18 +-
 .../hudi/index/TestHBaseQPSResourceAllocator.java  |   2 +-
 .../java/org/apache/hudi/index/TestHbaseIndex.java |   2 +-
 .../index/bloom/TestHoodieGlobalBloomIndex.java    | 140 ++++++-
 .../apache/hudi/io/TestHoodieCommitArchiveLog.java |  73 ++--
 .../org/apache/hudi/io/TestHoodieCompactor.java    |   6 +-
 .../io/storage/TestHoodieStorageWriterFactory.java |   4 +-
 ...HoodieDataFile.java => TestHoodieBaseFile.java} |  10 +-
 .../io/strategy/TestHoodieCompactionStrategy.java  |   4 +-
 .../apache/hudi/table/TestMergeOnReadTable.java    | 112 +++---
 hudi-common/pom.xml                                |   8 +-
 .../apache/hudi/avro/MercifulJsonConverter.java    |  59 ++-
 .../org/apache/hudi/common/HoodieJsonPayload.java  |   5 +-
 .../bloom/filter/InternalDynamicBloomFilter.java   |  35 +-
 .../common/bloom/filter/SimpleBloomFilter.java     |   5 +-
 .../common/io/storage/HoodieWrapperFileSystem.java |   2 +-
 .../hudi/common/model/CompactionOperation.java     |   6 +-
 .../org/apache/hudi/common/model/FileSlice.java    |  18 +-
 .../{HoodieDataFile.java => HoodieBaseFile.java}   |  10 +-
 .../apache/hudi/common/model/HoodieFileGroup.java  |  12 +-
 .../apache/hudi/common/model/HoodieTableType.java  |   3 -
 .../apache/hudi/common/model/HoodieWriteStat.java  |   3 +-
 .../hudi/common/model/TimelineLayoutVersion.java   |   5 +
 .../hudi/common/table/HoodieTableConfig.java       |  39 +-
 .../hudi/common/table/HoodieTableMetaClient.java   |  28 +-
 .../apache/hudi/common/table/HoodieTimeline.java   |  12 +
 .../hudi/common/table/SyncableFileSystemView.java  |   7 +-
 .../hudi/common/table/TableFileSystemView.java     |  32 +-
 .../apache/hudi/common/table/TimelineLayout.java   |   4 +-
 .../hudi/common/table/log/HoodieLogFileReader.java |  33 +-
 .../common/table/log/HoodieLogFormatWriter.java    |  42 +-
 .../table/timeline/HoodieActiveTimeline.java       | 141 ++-----
 .../table/timeline/HoodieArchivedTimeline.java     | 192 ++++++++--
 .../table/timeline/HoodieDefaultTimeline.java      |  83 +++-
 .../hudi/common/table/timeline/HoodieInstant.java  |   6 +-
 .../dto/{DataFileDTO.java => BaseFileDTO.java}     |  20 +-
 .../common/table/timeline/dto/FileSliceDTO.java    |   8 +-
 .../table/view/AbstractTableFileSystemView.java    | 140 +++----
 .../common/table/view/FileSystemViewManager.java   |   4 +-
 .../table/view/FileSystemViewStorageConfig.java    |  13 +-
 .../IncrementalTimelineSyncFileSystemView.java     |  10 +-
 .../table/view/PriorityBasedFileSystemView.java    |  32 +-
 .../view/RemoteHoodieTableFileSystemView.java      |  60 +--
 .../table/view/RocksDbBasedFileSystemView.java     | 198 +++++-----
 .../view/SpillableMapBasedFileSystemView.java      |   4 +-
 .../org/apache/hudi/common/util/AvroUtils.java     |   4 +-
 .../hudi/common/util/BufferedRandomAccessFile.java | 421 +++++++++++++++++++++
 .../org/apache/hudi/common/util/CleanerUtils.java  |   7 +-
 .../apache/hudi/common/util/CompactionUtils.java   |  50 ++-
 .../hudi/common/util/ConsistencyGuardConfig.java   |   5 +-
 .../java/org/apache/hudi/common/util/FSUtils.java  |  36 +-
 .../hudi/common/util/FailSafeConsistencyGuard.java |   4 +-
 .../apache/hudi/common/util/HoodieAvroUtils.java   |  10 +-
 .../common/util/HoodieRecordSizeEstimator.java     |  10 +-
 .../apache/hudi/common/util/LogReaderUtils.java    |   2 +-
 .../hudi/common/util/ObjectSizeCalculator.java     |   8 +-
 .../org/apache/hudi/common/util/RocksDBDAO.java    |  16 +-
 .../hudi/common/util/SerializationUtils.java       |   4 +-
 .../apache/hudi/common/util/SpillableMapUtils.java |   4 +-
 .../apache/hudi/common/util/TypedProperties.java   |  16 +-
 .../hudi/common/util/collection/DiskBasedMap.java  |  40 +-
 .../common/util/collection/LazyFileIterable.java   |   8 +-
 .../common/util/collection/RocksDBBasedMap.java    |   6 +-
 .../io/storage/SizeAwareDataOutputStream.java      |   5 +-
 .../common/util/queue/BoundedInMemoryQueue.java    |   4 +-
 .../hudi/exception/TableNotFoundException.java     |   3 +-
 .../hudi/common/minicluster/HdfsTestService.java   |   2 +-
 .../apache/hudi/common/model/HoodieTestUtils.java  |  48 ++-
 .../common/table/TestHoodieTableMetaClient.java    |  35 --
 .../hudi/common/table/log/TestHoodieLogFormat.java | 136 ++++++-
 .../table/log/TestHoodieLogFormatVersion.java      |  84 ++++
 .../table/string/TestHoodieActiveTimeline.java     | 308 ++++++++++++++-
 .../table/view/TestHoodieTableFileSystemView.java  | 141 +++----
 .../table/view/TestIncrementalFSViewSync.java      | 116 +++---
 .../hudi/common/util/CompactionTestUtils.java      |   8 +-
 .../hudi/common/util/TestCompactionUtils.java      |  16 +-
 .../org/apache/hudi/common/util/TestFSUtils.java   | 105 ++++-
 .../common/util/collection/TestDiskBasedMap.java   |   6 +-
 hudi-hadoop-mr/pom.xml                             |   2 +-
 .../org/apache/hudi/hadoop/HoodieHiveUtil.java     |  15 +-
 .../hudi/hadoop/HoodieParquetInputFormat.java      |  40 +-
 .../hudi/hadoop/HoodieROTablePathFilter.java       |   6 +-
 .../hadoop/hive/HoodieCombineHiveInputFormat.java  |  93 ++---
 .../realtime/AbstractRealtimeRecordReader.java     |  22 +-
 .../realtime/HoodieParquetRealtimeInputFormat.java |  45 +--
 .../hadoop/realtime/HoodieRealtimeFileSplit.java   |  20 +-
 .../realtime/HoodieRealtimeRecordReader.java       |   2 +-
 .../realtime/RealtimeCompactedRecordReader.java    |   4 +-
 .../realtime/RealtimeUnmergedRecordReader.java     |   9 +-
 .../apache/hudi/hadoop/InputFormatTestUtil.java    |  25 +-
 .../apache/hudi/hadoop/InputPathHandlerTest.java   |   2 +-
 .../hudi/hadoop/TestHoodieParquetInputFormat.java  |   6 +-
 .../realtime/TestHoodieRealtimeRecordReader.java   |  15 +-
 hudi-hive/pom.xml                                  |   2 +-
 .../java/org/apache/hudi/hive/HiveSyncConfig.java  |   3 +
 .../java/org/apache/hudi/hive/HiveSyncTool.java    |  90 +++--
 .../org/apache/hudi/hive/HoodieHiveClient.java     | 110 +++---
 .../java/org/apache/hudi/hive/util/SchemaUtil.java |  35 +-
 .../org/apache/hudi/hive/TestHiveSyncTool.java     | 102 ++---
 .../test/java/org/apache/hudi/hive/TestUtil.java   |  13 +-
 .../org/apache/hudi/hive/util/HiveTestService.java |  11 +-
 hudi-integ-test/pom.xml                            |   8 +-
 .../java/org/apache/hudi/integ/ITTestBase.java     |   2 +-
 .../org/apache/hudi/integ/ITTestHoodieDemo.java    |  77 ++--
 .../org/apache/hudi/integ/ITTestHoodieSanity.java  |  35 +-
 hudi-spark/pom.xml                                 |  37 +-
 hudi-spark/run_hoodie_app.sh                       |   2 +-
 .../main/java/org/apache/hudi/DataSourceUtils.java |  21 +-
 .../main/java/org/apache/hudi/QuickstartUtils.java |  25 +-
 .../hudi/{ => keygen}/ComplexKeyGenerator.java     |  16 +-
 .../GlobalDeleteKeyGenerator.java}                 |  49 +--
 .../org/apache/hudi/{ => keygen}/KeyGenerator.java |   2 +-
 .../{ => keygen}/NonpartitionedKeyGenerator.java   |   3 +-
 .../hudi/{ => keygen}/SimpleKeyGenerator.java      |   4 +-
 .../org/apache/hudi/AvroConversionHelper.scala     |   2 +-
 .../org/apache/hudi/AvroConversionUtils.scala      |  12 +-
 .../scala/org/apache/hudi/DataSourceOptions.scala  |  68 +++-
 .../main/scala/org/apache/hudi/DefaultSource.scala |  18 +-
 .../org/apache/hudi/HoodieSparkSqlWriter.scala     |  12 +-
 hudi-spark/src/test/java/DataSourceTestUtils.java  |   4 +-
 hudi-spark/src/test/java/HoodieJavaApp.java        |  24 +-
 .../src/test/java/HoodieJavaStreamingApp.java      |   8 +-
 .../src/test/java/TestComplexKeyGenerator.java     |   2 +-
 hudi-spark/src/test/scala/TestDataSource.scala     |  12 +-
 .../src/test/scala/TestDataSourceDefaults.scala    | 110 ++++--
 .../apache/hudi/HoodieSparkSqlWriterSuite.scala    |   4 +-
 hudi-timeline-service/pom.xml                      |   2 +-
 .../timeline/service/FileSystemViewHandler.java    |  25 +-
 .../{DataFileHandler.java => BaseFileHandler.java} |  50 +--
 hudi-utilities/pom.xml                             |  41 +-
 .../apache/hudi/utilities/HDFSParquetImporter.java |  23 +-
 .../hudi/utilities/HiveIncrementalPuller.java      |  11 +-
 .../org/apache/hudi/utilities/HoodieCleaner.java   |  19 +-
 .../hudi/utilities/HoodieCompactionAdminTool.java  |   5 +-
 .../org/apache/hudi/utilities/HoodieCompactor.java |   2 +-
 .../hudi/utilities/HoodieSnapshotCopier.java       |   8 +-
 .../hudi/utilities/HoodieWithTimelineServer.java   |  16 +-
 .../org/apache/hudi/utilities/UtilHelpers.java     |   7 +-
 .../hudi/utilities/deltastreamer/Compactor.java    |   2 +-
 .../hudi/utilities/deltastreamer/DeltaSync.java    |  43 ++-
 .../deltastreamer/HoodieDeltaStreamer.java         |  62 ++-
 .../deltastreamer/HoodieDeltaStreamerMetrics.java  |   4 +-
 .../deltastreamer/SchedulerConfGenerator.java      |   4 +-
 .../deltastreamer/SourceFormatAdapter.java         |  13 +-
 .../keygen/TimestampBasedKeyGenerator.java         |   2 +-
 .../hudi/utilities/perf/TimelineServerPerf.java    |  20 +-
 .../hudi/utilities/sources/AvroKafkaSource.java    |  17 +-
 .../hudi/utilities/sources/HoodieIncrSource.java   |  10 +-
 .../hudi/utilities/sources/JsonKafkaSource.java    |  13 +-
 .../utilities/sources/helpers/AvroConvertor.java   |   3 +-
 .../sources/helpers/IncrSourceHelper.java          |   4 +-
 .../utilities/sources/helpers/KafkaOffsetGen.java  | 151 +++-----
 .../utilities/transform/IdentityTransformer.java   |  38 --
 .../hudi/utilities/TestHDFSParquetImporter.java    |   2 +-
 .../hudi/utilities/TestHiveIncrementalPuller.java  |  18 +
 .../hudi/utilities/TestHoodieDeltaStreamer.java    |  92 ++++-
 .../hudi/utilities/TestSchedulerConfGenerator.java |   6 +-
 .../utilities/TestTimestampBasedKeyGenerator.java  |   3 +-
 .../apache/hudi/utilities/UtilitiesTestBase.java   |  11 +-
 .../utilities/sources/AbstractBaseTestSource.java  |   2 +-
 .../sources/AbstractDFSSourceTestBase.java         | 178 +++++++++
 .../hudi/utilities/sources/TestDFSSource.java      | 194 ----------
 .../hudi/utilities/sources/TestJsonDFSSource.java  |  55 +++
 .../hudi/utilities/sources/TestKafkaSource.java    |  22 +-
 .../utilities/sources/TestParquetDFSSource.java    |  32 +-
 .../delta-streamer-config/kafka-source.properties  |   4 +-
 .../resources/delta-streamer-config/source.avsc    |  15 +-
 .../resources/delta-streamer-config/target.avsc    |  15 +-
 packaging/hudi-hadoop-mr-bundle/pom.xml            |   2 +-
 packaging/hudi-hive-bundle/pom.xml                 |   2 +-
 packaging/hudi-presto-bundle/pom.xml               |   2 +-
 packaging/hudi-spark-bundle/pom.xml                |  14 +-
 packaging/hudi-timeline-server-bundle/pom.xml      |   2 +-
 packaging/hudi-utilities-bundle/pom.xml            |  20 +-
 pom.xml                                            | 124 ++++--
 scripts/release/deploy_staging_jars.sh             |  18 +-
 scripts/release/validate_staged_release.sh         |   3 +
 244 files changed, 4436 insertions(+), 2566 deletions(-)
 rename hudi-client/src/main/java/org/apache/hudi/io/compact/{HoodieRealtimeTableCompactor.java
=> HoodieMergeOnReadTableCompactor.java} (94%)
 rename hudi-client/src/test/java/org/apache/hudi/io/strategy/{TestHoodieDataFile.java =>
TestHoodieBaseFile.java} (83%)
 rename hudi-common/src/main/java/org/apache/hudi/common/model/{HoodieDataFile.java =>
HoodieBaseFile.java} (91%)
 mode change 100644 => 100755 hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTimeline.java
 rename hudi-common/src/main/java/org/apache/hudi/common/table/timeline/dto/{DataFileDTO.java
=> BaseFileDTO.java} (77%)
 create mode 100644 hudi-common/src/main/java/org/apache/hudi/common/util/BufferedRandomAccessFile.java
 mode change 100644 => 100755 hudi-common/src/test/java/org/apache/hudi/common/table/log/TestHoodieLogFormat.java
 create mode 100755 hudi-common/src/test/java/org/apache/hudi/common/table/log/TestHoodieLogFormatVersion.java
 mode change 100644 => 100755 hudi-common/src/test/java/org/apache/hudi/common/table/string/TestHoodieActiveTimeline.java
 copy hudi-spark/src/main/java/org/apache/hudi/{ => keygen}/ComplexKeyGenerator.java (87%)
 rename hudi-spark/src/main/java/org/apache/hudi/{ComplexKeyGenerator.java => keygen/GlobalDeleteKeyGenerator.java}
(55%)
 rename hudi-spark/src/main/java/org/apache/hudi/{ => keygen}/KeyGenerator.java (98%)
 rename hudi-spark/src/main/java/org/apache/hudi/{ => keygen}/NonpartitionedKeyGenerator.java
(95%)
 rename hudi-spark/src/main/java/org/apache/hudi/{ => keygen}/SimpleKeyGenerator.java (96%)
 rename hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/{DataFileHandler.java
=> BaseFileHandler.java} (51%)
 delete mode 100644 hudi-utilities/src/main/java/org/apache/hudi/utilities/transform/IdentityTransformer.java
 create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/AbstractDFSSourceTestBase.java
 delete mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestDFSSource.java
 create mode 100644 hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestJsonDFSSource.java
 copy hudi-common/src/main/java/org/apache/hudi/common/util/NoOpConsistencyGuard.java =>
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestParquetDFSSource.java (51%)


Mime
View raw message