parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ga...@apache.org
Subject [parquet-mr] branch column-indexes updated (55d791c -> c215f1f)
Date Fri, 28 Sep 2018 07:52:18 GMT
This is an automated email from the ASF dual-hosted git repository.

gabor pushed a change to branch column-indexes
in repository https://gitbox.apache.org/repos/asf/parquet-mr.git.


    from 55d791c  PARQUET-1389: Improve value skipping at page synchronization (#514)
     add a287522  PARQUET-1294: Update release scripts for the new Apache policy (#475)
     add 94a8bf6  PARQUET-1253: Support for new logical type representation (#463)
     add 345e2d5  PARQUET-1304: Release 1.10 contains breaking changes for Hive (#485)
     add aed9097  PARQUET-1311: Update README.md (#487)
     add 3fd2492  PARQUET-1317: Fix ParquetMetadataConverter throw NPE (#489)
     add a918c49  PARQUET-1317: Fix ParquetMetadataConverter throw NPE (#491)
     add 9181e1d  PARQUET-1309: Parquet Java uses incorrect stats and dictionary filter properties
(#490)
     add 74d650b  [PARQUET-1135][FOLLOW-UP] Update thrift and protoc version in README.md
(#488)
     add f2d5871  PARQUET-1321: LogicalTypeAnnotation.LogicalTypeAnnotationVisitor#visit methods
should have a return value (#493)
     add cc8bdf1  PARQUET-952: Avro union with single type fails with 'is not a group' (#459)
     add 33ee549  PARQUET-1335: Logical type names in parquet-mr are not consistent with parquet-format
(#496)
     add dc61e51  PARQUET-1336: PrimitiveComparator should implements Serializable (#497)
     add d320a45  PARQUET-1341: Fix null count stats in unsigned-sort columns. (#499)
     add 94ae6c8  PARQUET-1344: Type builders don't honor new logical types (#500)
     add e9e36cd  PARQUET-1335: Logical type names in parquet-mr are not consistent with parquet-format
(#503)
     add 55e9497  PARQUET-1371: Time/Timestamp UTC normalization parameter doesn't work (#511)
     add 45e3ce5  PARQUET-1368: ParquetFileReader should close its input stream for the failure
in constructor (#510)
     add d692ce3  PARQUET-1390: Upgrade Arrow to 0.10.0
     add 863a081  PARQUET-1381: Add merge blocks command to parquet-tools (#512)
     add b4198be  PARQUET-1410: Refactor modules to use the new logical type API (#520)
     add 1f79f9b  PARQUET-1353: Fix random data generator. (#504)
     add 93767ca  PARQUET-1417: BINARY_AS_SIGNED_INTEGER_COMPARATOR fails with IOBE for the
same arrays with the different length (#522)
     add 411e672  PARQUET-1421: InternalParquetRecordWriter logs debug messages at the INFO
level (#526)
     add 797fc6f  PARQUET-1418: Run integration tests in Travis (#524)
     add a4c107e  Updated file description.
     add cb17588  comments on the thrift file; diagram for the metadata
     add f848e45  Fix description in POM and more README cleanup
     add c619db1  Add bool description and miscellaneous cleanup.
     add 06627f0  Thrift file updates.
     add 01b195b  Update readme
     add 511b1f5  embed thrift to avoid dependency
     add dd8077e  adding utility classe to completely hide Thrift
     add efb93bf  Merge pull request #35 from Parquet/shading_jar
     add 080b953  [maven-release-plugin] prepare release parquet-format-1.0.0-t2
     add 8463ba1  [maven-release-plugin] prepare for next development iteration
     add 3848f32  undo release changes
     add 2925244  Merge pull request #46 from Parquet/alexlevenson/update-readme-rle-4bytes
     add ab5d3a1  [maven-release-plugin] prepare release parquet-format-1.0.0-t2
     add 46aa59c  revert back to SNAPSHOT
     add 32dbb4a  [maven-release-plugin] prepare release parquet-format-1.0.0
     add 97cf75f  [maven-release-plugin] prepare for next development iteration
     add d920ce2  fix plugin versions
     add dbf495c  fix plugin name
     add 0b49e8a  update mvn shade plugin instead
     add 73e3d08  Merge pull request #77 from Parquet/fix_plugin_versions
     add aa44246  [maven-release-plugin] prepare release parquet-format-2.0.0
     add 891cc3a  [maven-release-plugin] prepare for next development iteration
     add d40381a  Upgrade maven-shade-plugin to 2.1 to compile with mvn 3.1.1
     add 4b517ae  Merge pull request #80 from gerashegalov/master
     add 3f37ca2  exclude thrift source from jar
     add c54e6d5  Merge pull request #82 from Parquet/exclude_thrift_source_from_jar
     add 70a0012  [maven-release-plugin] prepare release parquet-format-2.1.0
     add 31df4fa  [maven-release-plugin] prepare for next development iteration
     add dd98249  Merge branch 'master' into field_id
     add 402749f  Merge pull request #85 from Parquet/field_id
     add 02abe09  PARQUET-11: Reduce memory pressure when reading footers
     add 97964f3  PARQUET-79: add a streaming Thrift API, to enable processing the metadata
as we read it and skipping unnecessary fields.
     add 78de104  PARQUET-72: Prepare for Apache release
     add f6608e6  PARQUET-85: add license headers
     add f8bc8d1  PARQUET-72: Update POM to use ASF parent.
     add 149ef7a  [maven-release-plugin] prepare release parquet-format-2.2.0-rc1
     add f2d88f2  [maven-release-plugin] prepare for next development iteration
     add 2963317  PARQUET-72: Fix NOTICE
     add 3633dee  PARQUET-109: Update NOTICE, add binary LICENSE.
     add 0fcfe61  PARQUET-23: Refactor parquet-format to org.apache names.
     add ec9612f  PARQUET-111: Update LICENSE and pom.
     add fc5f1f8  [maven-release-plugin] prepare release parquet-format-2.2.0
     add 6827f85  Revert "[maven-release-plugin] prepare release parquet-format-2.2.0"
     add bf8c7fd  [maven-release-plugin] prepare release apache-parquet-format-2.2.0
     add e9f2b90  [maven-release-plugin] prepare for next development iteration
     add 89f9a7a  [maven-release-plugin] prepare release apache-parquet-format-2.3.0
     add 2c23ff7  [maven-release-plugin] prepare for next development iteration
     add 2726dc5  PARQUET-185: Update release scripts and POM.
     add db35a4c  [maven-release-plugin] prepare release apache-parquet-format-2.3.0-incubating
     add 23fd736  [maven-release-plugin] prepare for next development iteration
     add 75f0715  PARQUET-265: Update POM for Parquet TLP.
     add 3052d4e  PARQUET-178: Remove SLF4J META-INF from binary artifacts.
     add d5eb863  PARQUET-369: Add shaded SLF4J NOP binding.
     add f6cd4cc  [maven-release-plugin] prepare release apache-parquet-format-2.3.1
     add 43f122f  [maven-release-plugin] prepare for next development iteration
     add 5008c5a  PARQUET-450: Fix several typos in Parquet format documentation
     add 73e936c  PARQUET-609: Add Brotli to parquet's thrift definition
     add ae52b4d  PARQUET-371: update thrift dependency to 0.9.3; do not shade slf4j
     add c48cbb1  PARQUET-1049: Make thrift version a property in pom.xml
     add e281913  PARQUET-906: Add LogicalType annotation.
     add 56c4fcd  [maven-release-plugin] prepare release apache-parquet-format-2.4.0
     add 1b8292a  [maven-release-plugin] prepare for next development iteration
     add 6cc4110  PARQUET-1144: Remove slf4j-nop.
     add 3df27ee  [maven-release-plugin] prepare release apache-parquet-format-2.4.0
     add 9eae3e8  [maven-release-plugin] prepare for next development iteration
     add 40d263f  PARQUET-1145: Add license to .gitignore
     add 8293207  PARQUET-1197: Log rat failures
     add 75f0e42  PARQUET-1201: Implement page indexes
     add ad3a8d4  PARQUET-1236: Align version of slf4j-api
     add ba5ad42  PARQUET-1258: Update scm developer connection to github (#90)
     add 06ffe24  [maven-release-plugin] prepare release apache-parquet-format-2.5.0
     add e846f7a  Revert "[maven-release-plugin] prepare release apache-parquet-format-2.5.0"
     add 59f5c72  [maven-release-plugin] prepare release apache-parquet-format-2.5.0
     add 3d8426e  [maven-release-plugin] prepare for next development iteration
     add 344b568  PARQUET-1399: Move files to the module directory
     add 412685f  Merge commit '344b56803fea37af84b9c01c9b6dcff586779683' into merge_PARQUET-1399
     add a150f24  PARQUET-1399: Move parquet-mr related code from parquet-format
     new 85e699c  Merge branch 'master' into column-indexes
     new c215f1f  PARQUET-1381: Fix missing endRecord after merging columnIndex

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .travis.yml                                        |   2 +-
 README.md                                          |  92 +-
 dev/README.md                                      |   4 +-
 dev/source-release.sh                              |   3 +-
 parquet-arrow/pom.xml                              |   2 +-
 .../parquet/arrow/schema/SchemaConverter.java      | 260 +++---
 .../parquet/arrow/schema/TestSchemaConverter.java  |  27 +-
 parquet-avro/pom.xml                               |   4 +-
 .../apache/parquet/avro/AvroSchemaConverter.java   | 176 ++--
 .../parquet/avro/TestAvroSchemaConverter.java      |  14 +-
 .../org/apache/parquet/avro/TestReadWrite.java     |  31 +
 .../parquet/cascading/convert/TupleConverter.java  |   9 +-
 .../parquet/cascading/TestParquetTBaseScheme.java  |   7 +-
 .../src/main/java/org/apache/parquet/cli/Util.java |  10 +
 .../cli/commands/ParquetMetadataCommand.java       |   4 +-
 .../cli/commands/ShowDictionaryCommand.java        |   4 +-
 .../parquet/cli/commands/ShowPagesCommand.java     |   4 +-
 .../parquet/column/impl/ColumnReadStoreImpl.java   |   5 +
 .../apache/parquet/column/values/ValuesReader.java |  70 ++
 .../values/bitpacking/BitPackingValuesReader.java  |   1 +
 .../bitpacking/ByteBitPackingValuesReader.java     |   1 +
 .../delta/DeltaBinaryPackingValuesReader.java      |   2 +
 .../values/plain/BooleanPlainValuesReader.java     |   6 +
 .../rle/RunLengthBitPackingHybridValuesReader.java |   3 +
 .../column/values/rle/ZeroIntegerValuesReader.java |   1 +
 .../parquet/filter2/predicate/ValidTypeMap.java    |   7 +-
 .../apache/parquet/schema/ConversionPatterns.java  |  28 +-
 .../java/org/apache/parquet/schema/GroupType.java  |  53 +-
 .../parquet/schema/LogicalTypeAnnotation.java      | 983 +++++++++++++++++++++
 .../org/apache/parquet/schema/MessageType.java     |   8 +-
 .../apache/parquet/schema/MessageTypeParser.java   |  55 +-
 .../org/apache/parquet/schema/OriginalType.java    |  66 +-
 .../apache/parquet/schema/PrimitiveComparator.java |  10 +-
 .../org/apache/parquet/schema/PrimitiveType.java   | 269 +++---
 .../main/java/org/apache/parquet/schema/Type.java  |  40 +-
 .../main/java/org/apache/parquet/schema/Types.java | 213 ++++-
 ...ltaBinaryPackingValuesWriterForIntegerTest.java |   8 +
 .../DeltaBinaryPackingValuesWriterForLongTest.java |   8 +
 .../column/values/dictionary/TestDictionary.java   |   5 +
 .../filter2/predicate/TestValidTypeMap.java        |   7 +-
 .../apache/parquet/parser/TestParquetParser.java   |  72 +-
 .../org/apache/parquet/schema/TestMessageType.java |   2 +-
 .../parquet/schema/TestPrimitiveComparator.java    |  19 +
 .../apache/parquet/schema/TestTypeBuilders.java    |  76 +-
 parquet-common/pom.xml                             |   4 +-
 .../parquet/bytes/ByteBufferInputStream.java       | 100 ++-
 .../java/org/apache/parquet/bytes/BytesInput.java  |  16 +
 .../parquet/bytes/MultiBufferInputStream.java      |   2 +-
 .../parquet/bytes/TestByteBufferInputStreams.java  |  14 +
 ...m.java => TestDeprecatedBufferInputStream.java} |  94 +-
 .../parquet/bytes/TestSingleBufferInputStream.java |   2 +-
 parquet-format-structures/pom.xml                  | 206 +++++
 .../apache/parquet/format/InterningProtocol.java   | 190 ++--
 .../org/apache/parquet/format/LogicalTypes.java    |  55 ++
 .../main/java/org/apache/parquet/format/Util.java  | 236 +++++
 .../org/apache/parquet/format/event/Consumers.java | 193 ++++
 .../format/event/EventBasedThriftReader.java       | 126 +++
 .../apache/parquet/format/event/FieldConsumer.java |  26 +-
 .../apache/parquet/format/event/TypedConsumer.java | 205 +++++
 .../java/org/apache/parquet/format/TestUtil.java   |  83 ++
 parquet-hadoop/pom.xml                             |   4 +-
 .../java/org/apache/parquet/HadoopReadOptions.java |   4 +-
 .../format/converter/ParquetMetadataConverter.java | 482 +++++++---
 .../parquet/hadoop/ColumnChunkPageWriteStore.java  |   5 +
 .../hadoop/InternalParquetRecordWriter.java        |   4 +-
 .../apache/parquet/hadoop/ParquetFileReader.java   | 100 ++-
 .../apache/parquet/hadoop/ParquetFileWriter.java   | 123 +++
 .../parquet/hadoop/metadata/ParquetMetadata.java   |  15 +-
 .../apache/parquet/hadoop/util/BlocksCombiner.java | 106 +++
 .../converter/TestParquetMetadataConverter.java    |  95 +-
 ...ocks.java => TestParquetWriterMergeBlocks.java} | 232 +++--
 .../apache/parquet/statistics/RandomValues.java    |   7 +-
 .../ql/io/parquet/convert/HiveSchemaConverter.java |  17 +-
 parquet-pig/pom.xml                                |   4 +-
 .../org/apache/parquet/pig/PigSchemaConverter.java | 130 +--
 .../apache/parquet/pig/convert/TupleConverter.java |  31 +-
 parquet-protobuf/pom.xml                           |  11 +
 .../parquet/proto/ProtoMessageConverter.java       |  43 +-
 .../apache/parquet/proto/ProtoSchemaConverter.java |  45 +-
 .../apache/parquet/proto/ProtoWriteSupport.java    |  29 +-
 parquet-thrift/pom.xml                             |  11 +
 .../parquet/thrift/ThriftSchemaConvertVisitor.java |  18 +-
 parquet-tools/pom.xml                              |   4 +-
 .../apache/parquet/tools/command/DumpCommand.java  |   1 -
 .../apache/parquet/tools/command/MergeCommand.java |  75 +-
 .../tools/{util => command}/MetadataUtils.java     |  93 +-
 .../parquet/tools/command/ShowMetaCommand.java     |  29 +-
 .../parquet/tools/command/ShowSchemaCommand.java   |  14 +-
 .../parquet/tools/read/SimpleRecordConverter.java  |  56 +-
 .../apache/parquet/tools/util/MetadataUtils.java   |   9 +-
 pom.xml                                            |   8 +
 91 files changed, 4771 insertions(+), 1257 deletions(-)
 create mode 100644 parquet-column/src/main/java/org/apache/parquet/schema/LogicalTypeAnnotation.java
 copy parquet-common/src/test/java/org/apache/parquet/bytes/{TestSingleBufferInputStream.java
=> TestDeprecatedBufferInputStream.java} (52%)
 create mode 100644 parquet-format-structures/pom.xml
 copy parquet-thrift/src/main/java/org/apache/parquet/thrift/ParquetProtocol.java => parquet-format-structures/src/main/java/org/apache/parquet/format/InterningProtocol.java
(58%)
 create mode 100644 parquet-format-structures/src/main/java/org/apache/parquet/format/LogicalTypes.java
 create mode 100644 parquet-format-structures/src/main/java/org/apache/parquet/format/Util.java
 create mode 100644 parquet-format-structures/src/main/java/org/apache/parquet/format/event/Consumers.java
 create mode 100644 parquet-format-structures/src/main/java/org/apache/parquet/format/event/EventBasedThriftReader.java
 copy parquet-thrift/src/main/java/org/apache/parquet/thrift/ProtocolPipe.java => parquet-format-structures/src/main/java/org/apache/parquet/format/event/FieldConsumer.java
(59%)
 create mode 100644 parquet-format-structures/src/main/java/org/apache/parquet/format/event/TypedConsumer.java
 create mode 100644 parquet-format-structures/src/test/java/org/apache/parquet/format/TestUtil.java
 create mode 100644 parquet-hadoop/src/main/java/org/apache/parquet/hadoop/util/BlocksCombiner.java
 copy parquet-hadoop/src/test/java/org/apache/parquet/hadoop/{TestParquetWriterAppendBlocks.java
=> TestParquetWriterMergeBlocks.java} (57%)
 copy parquet-tools/src/main/java/org/apache/parquet/tools/{util => command}/MetadataUtils.java
(76%)


Mime
View raw message