impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Sun (Code Review)" <ger...@cloudera.org>
Subject [RecordServiceClient-CR](master_dev) Add complex type support for client side
Date Fri, 06 May 2016 21:18:21 GMT
Chao Sun has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/2988

Change subject: Add complex type support for client side
......................................................................

Add complex type support for client side

This adds complex type support for the client side (Java). All the
implementations here conform to the Apache Arrow's standard, with the
exception of:
- struct type is different in that it doesn't have a "is_null" array
associated with. This is due to how struct is handled in Impala, and
will be changed once that is fixed.
- no support for union type.

All the work are included under package com.cloudera.recordservice.core.data.
It is also accompanied with some some tests in TestArray class. Some of these
use the functional.complextypestbl on the Impala side.

The next step is to integrate this with input formats and frameworks.

Change-Id: Ia2c14c85eb9eeb06ae30d7feebf89170136235df
---
M api/RecordService.thrift
M cpp/samples/record-service-client.cc
M cpp/samples/rs-grep.cc
M cpp/tests/client-test.cc
M cpp/tests/test-common.h
M java/avro/src/main/java/com/cloudera/recordservice/avro/GenericRecords.java
M java/avro/src/main/java/com/cloudera/recordservice/avro/KeyValueRecords.java
M java/avro/src/main/java/com/cloudera/recordservice/avro/RecordUtil.java
M java/core/src/main/java/com/cloudera/recordservice/core/Records.java
M java/core/src/main/java/com/cloudera/recordservice/core/Schema.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/AbstractArray.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/Array.java
R java/core/src/main/java/com/cloudera/recordservice/core/data/ByteArray.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/CharArray.java
R java/core/src/main/java/com/cloudera/recordservice/core/data/Decimal.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/ListArray.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/ListArrayBase.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/Map.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/MapArray.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/PrimitiveArray.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/Record.java
A java/core/src/main/java/com/cloudera/recordservice/core/data/StructArray.java
R java/core/src/main/java/com/cloudera/recordservice/core/data/TimestampNanos.java
M java/core/src/test/java/com/cloudera/recordservice/core/SampleClient.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestBase.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestBasicClient.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestFaultTolerance.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestKerberosConnection.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestS3.java
M java/core/src/test/java/com/cloudera/recordservice/core/TestSchema.java
A java/core/src/test/java/com/cloudera/recordservice/core/data/TestComplexTypes.java
R java/core/src/test/java/com/cloudera/recordservice/core/data/TestUtil.java
M java/examples/src/main/java/com/cloudera/recordservice/examples/RSCat.java
M java/examples/src/main/java/com/cloudera/recordservice/examples/SumQueryBenchmark.java
M java/examples/src/main/java/com/cloudera/recordservice/examples/terasort/RecordServiceTeraInputFormat.java
M java/hcatalog-pig-adapter/src/main/java/com/cloudera/recordservice/pig/PigHCatUtil.java
M java/mr/src/main/java/com/cloudera/recordservice/mapred/TextInputFormat.java
M java/mr/src/main/java/com/cloudera/recordservice/mapreduce/RecordServiceInputFormat.java
M java/mr/src/main/java/com/cloudera/recordservice/mapreduce/TextInputFormat.java
M java/mr/src/main/java/com/cloudera/recordservice/mr/DecimalWritable.java
M java/mr/src/main/java/com/cloudera/recordservice/mr/RecordServiceRecord.java
M java/mr/src/main/java/com/cloudera/recordservice/mr/TimestampNanosWritable.java
M java/spark/src/main/scala/com/cloudera/recordservice/spark/RecordServiceRDD.scala
M java/spark/src/main/scala/com/cloudera/recordservice/spark/RecordServiceRecordRDD.scala
M java/spark/src/main/scala/com/cloudera/recordservice/spark/RecordServiceRelation.scala
M java/spark/src/main/scala/com/cloudera/recordservice/spark/SchemaRecordServiceRDD.scala
M java/spark/src/test/scala/com/cloudera/recordservice/spark/BasicClientSuite.scala
M tests/benchmarks.py
M tests/create-tbls.sql
M tests/load-test-data.sh
50 files changed, 2,274 insertions(+), 490 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/RecordServiceClient refs/changes/88/2988/1
-- 
To view, visit http://gerrit.cloudera.org:8080/2988
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia2c14c85eb9eeb06ae30d7feebf89170136235df
Gerrit-PatchSet: 1
Gerrit-Project: RecordServiceClient
Gerrit-Branch: master_dev
Gerrit-Owner: Chao Sun <sunchao.chris@gmail.com>

Mime
View raw message