arrow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From w...@apache.org
Subject arrow git commit: ARROW-1373: Implement getBuffer() methods for ValueVector
Date Sat, 19 Aug 2017 00:10:32 GMT
Repository: arrow
Updated Branches:
  refs/heads/master 4ef7c898b -> c9805d6fd


ARROW-1373: Implement getBuffer() methods for ValueVector

cc @jacques-n , @StevenMPhillips

Patch Summary:

As part of ARROW-801, we recently added getValidityBufferAddress(), getOffsetBufferAddress(),
getDataBufferAddress() interfaces to get the virtual address of the ArrowBuf.

We now have the following new interfaces to get the corresponding ArrowBuf:

getValidityBuffer()
getDataBuffer()
getOffsetBuffer()

Background:

Currently we have getBuffer() method implemented as part of BaseDataValueVector abstract class.
As part of patch for ARROW-276, NullableValueVectors no longer extends BaseDataValueVector
-- they don't have to since they don't need the underlying data buffer  (ArrowBuf data field)
of BaseDataValueVector.

The call to getBuffer() on NullableValueVectors simply delegates the operation to getBuffer()
of underlying data/value vector.

Problem:

If a piece of code is working with ValueVector abstraction and the expected runtime type is
Nullable<something>Vector, the compiler obviously complains about doing
(v of type ValueVector).getBuffer().

Until now this worked as we kept the compiler happy by casting the ValueVector to BaseDataValueVector
and then do ((BaseDataValueVector)(v of type ValueVector)).getBuffer(). This code broke since
NullableValueVectors are no longer a subtype of BaseDataValueVector -- the inheritance hierarchy
was changed as part of ARROW-276.

Solution:

Similar to what was done in ARROW-801, we have new methods at ValueVector interface to get
the underlying buffer. ValueVector has always had the methods getBuffers(), getBufferSizeFor(),
getBufferSize(), so it makes sense to augment the ValueVector interface with new APIs.

It looks like new unit tests are not needed since the unit tests added for ARROW-801 test
the new APIs as well --> getDataBufferAddress() underneath invokes getDataBuffer() to get
the memory address of ArrowBuf so we are good.

Author: siddharth <siddharth@dremio.com>

Closes #976 from siddharthteotia/ARROW-1373 and squashes the following commits:

1ef2022e [siddharth] Fixed failures and added javadocs
e5ff0233 [siddharth] ARROW-1373: Implement getBuffer() methods for ValueVector


Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/c9805d6f
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/c9805d6f
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/c9805d6f

Branch: refs/heads/master
Commit: c9805d6fd111071b9da6533f8ddd477492ed44f6
Parents: 4ef7c89
Author: siddharth <siddharth@dremio.com>
Authored: Fri Aug 18 20:10:27 2017 -0400
Committer: Wes McKinney <wes.mckinney@twosigma.com>
Committed: Fri Aug 18 20:10:27 2017 -0400

----------------------------------------------------------------------
 .../codegen/templates/FixedValueVectors.java    | 18 +++++++++++
 .../codegen/templates/NullableValueVectors.java | 33 ++++++++++++++++++--
 .../src/main/codegen/templates/UnionVector.java | 11 ++++++-
 .../templates/VariableLengthVectors.java        | 20 +++++++++++-
 .../java/org/apache/arrow/vector/BitVector.java | 18 +++++++++++
 .../org/apache/arrow/vector/ValueVector.java    | 21 +++++++++++++
 .../org/apache/arrow/vector/ZeroVector.java     | 15 +++++++++
 .../vector/complex/FixedSizeListVector.java     | 17 +++++++++-
 .../apache/arrow/vector/complex/ListVector.java | 15 +++++++--
 .../apache/arrow/vector/complex/MapVector.java  | 17 ++++++++++
 .../arrow/vector/complex/NullableMapVector.java | 15 +++++++++
 11 files changed, 192 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/codegen/templates/FixedValueVectors.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/codegen/templates/FixedValueVectors.java b/java/vector/src/main/codegen/templates/FixedValueVectors.java
index 61164ab..9747d42 100644
--- a/java/vector/src/main/codegen/templates/FixedValueVectors.java
+++ b/java/vector/src/main/codegen/templates/FixedValueVectors.java
@@ -97,6 +97,24 @@ public final class ${className} extends BaseDataValueVector implements
FixedWidt
   }
 
   @Override
+  public ArrowBuf getValidityBuffer() {
+    /* this operation is not supported for non-nullable vectors */
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    /* we are not throwing away getBuffer() of BaseDataValueVector so use it wherever applicable
*/
+    return getBuffer();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    /* this operation is not supported for fixed-width vectors */
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
   public int getValueCapacity(){
     return (int) (data.capacity() *1.0 / ${type.width});
   }

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/codegen/templates/NullableValueVectors.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/codegen/templates/NullableValueVectors.java b/java/vector/src/main/codegen/templates/NullableValueVectors.java
index bd322ea..a431333 100644
--- a/java/vector/src/main/codegen/templates/NullableValueVectors.java
+++ b/java/vector/src/main/codegen/templates/NullableValueVectors.java
@@ -219,7 +219,7 @@ protected final static byte[] emptyByteArray = new byte[]{};
   }
 
   public ArrowBuf getBuffer() {
-    return values.getBuffer();
+    return values.getDataBuffer();
   }
 
   @Override
@@ -437,16 +437,21 @@ protected final static byte[] emptyByteArray = new byte[]{};
 
   @Override
   public long getValidityBufferAddress() {
-    return (bits.getBuffer().memoryAddress());
+    /* address of the databuffer associated with the bitVector */
+    return (bits.getDataBuffer().memoryAddress());
   }
 
   @Override
   public long getDataBufferAddress() {
-    return (values.getBuffer().memoryAddress());
+    /* address of the dataBuffer associated with the valueVector */
+    return (values.getDataBuffer().memoryAddress());
   }
 
   @Override
   public long getOffsetBufferAddress() {
+    /* address of the dataBuffer associated with the offsetVector
+     * this operation is not supported for fixed-width vector types.
+     */
     <#if type.major != "VarLen">
         throw new UnsupportedOperationException();
     <#else>
@@ -454,6 +459,28 @@ protected final static byte[] emptyByteArray = new byte[]{};
     </#if>
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() {
+    /* dataBuffer associated with the bitVector */
+    return (bits.getDataBuffer());
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    /* dataBuffer associated with the valueVector */
+    return (values.getDataBuffer());
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    /* dataBuffer associated with the offsetVector of the valueVector */
+    <#if type.major != "VarLen">
+        throw new UnsupportedOperationException();
+    <#else>
+        return (values.getOffsetBuffer());
+    </#if>
+  }
+
   public final class Accessor extends BaseDataValueVector.BaseAccessor <#if type.major
= "VarLen">implements VariableWidthVector.VariableWidthAccessor</#if> {
     final BitVector.Accessor bAccessor = bits.getAccessor();
     final ${valuesName}.Accessor vAccessor = values.getAccessor();

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/codegen/templates/UnionVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/codegen/templates/UnionVector.java b/java/vector/src/main/codegen/templates/UnionVector.java
index 8c01c4a..fe24a86 100644
--- a/java/vector/src/main/codegen/templates/UnionVector.java
+++ b/java/vector/src/main/codegen/templates/UnionVector.java
@@ -134,7 +134,7 @@ public class UnionVector implements FieldVector {
 
   @Override
   public long getValidityBufferAddress() {
-    return typeVector.getBuffer().memoryAddress();
+    return typeVector.getDataBuffer().memoryAddress();
   }
 
   @Override
@@ -147,6 +147,15 @@ public class UnionVector implements FieldVector {
     throw new UnsupportedOperationException();
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() { return typeVector.getDataBuffer(); }
+
+  @Override
+  public ArrowBuf getDataBuffer() { throw new UnsupportedOperationException(); }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() { throw new UnsupportedOperationException(); }
+
   public NullableMapVector getMap() {
     if (mapVector == null) {
       int vectorCount = internalMap.size();

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/codegen/templates/VariableLengthVectors.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/codegen/templates/VariableLengthVectors.java b/java/vector/src/main/codegen/templates/VariableLengthVectors.java
index 70c7209..c276f11 100644
--- a/java/vector/src/main/codegen/templates/VariableLengthVectors.java
+++ b/java/vector/src/main/codegen/templates/VariableLengthVectors.java
@@ -124,6 +124,24 @@ public final class ${className} extends BaseDataValueVector implements
VariableW
   }
 
   @Override
+  public ArrowBuf getValidityBuffer() {
+    /* this operation is not supported for non-nullable vectors */
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    /* we are not throwing away getBuffer() of BaseDataValueVector so use it wherever applicable
*/
+    return getBuffer();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    /* dataBuffer associated with the underlying offsetVector */
+    return offsetVector.getDataBuffer();
+  }
+
+  @Override
   public int getValueCapacity(){
     return Math.max(offsetVector.getValueCapacity() - 1, 0);
   }
@@ -170,7 +188,7 @@ public final class ${className} extends BaseDataValueVector implements
VariableW
   }
 
   public long getOffsetAddr(){
-    return offsetVector.getBuffer().memoryAddress();
+    return offsetVector.getDataBuffer().memoryAddress();
   }
 
   public UInt${type.width}Vector getOffsetVector(){

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
index e80ca82..44001d4 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/BitVector.java
@@ -110,6 +110,24 @@ public final class BitVector extends BaseDataValueVector implements FixedWidthVe
     return getSizeFromCount(valueCount);
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() {
+    /* this operation is not supported for non-nullable vectors */
+    throw new  UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    /* we are not throwing away getBuffer() of BaseDataValueVector so use it wherever applicable
*/
+    return getBuffer();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    /* this operation is not supported for fixed-width vectors */
+    throw new UnsupportedOperationException();
+  }
+
   int getSizeFromCount(int valueCount) {
     return (int) Math.ceil(valueCount / 8.0);
   }

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/ValueVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/ValueVector.java b/java/vector/src/main/java/org/apache/arrow/vector/ValueVector.java
index 0c95bcf..fb7286f 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/ValueVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/ValueVector.java
@@ -234,4 +234,25 @@ public interface ValueVector extends Closeable, Iterable<ValueVector>
{
     @Deprecated
     void generateTestData(int values);
   }
+
+  /**
+   * Gets the underlying buffer associated with validity vector
+   *
+   * @return buffer
+   */
+  public ArrowBuf getValidityBuffer();
+
+  /**
+   * Gets the underlying buffer associated with data vector
+   *
+   * @return buffer
+   */
+  public ArrowBuf getDataBuffer();
+
+  /**
+   * Gets the underlying buffer associated with offset vector
+   *
+   * @return buffer
+   */
+  public ArrowBuf getOffsetBuffer();
 }

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/ZeroVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/ZeroVector.java b/java/vector/src/main/java/org/apache/arrow/vector/ZeroVector.java
index fe224ba..b267b20 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/ZeroVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/ZeroVector.java
@@ -244,4 +244,19 @@ public class ZeroVector implements FieldVector {
   public long getOffsetBufferAddress() {
     throw new UnsupportedOperationException();
   }
+
+  @Override
+  public ArrowBuf getValidityBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    throw new UnsupportedOperationException();
+  }
 }

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
b/java/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
index aab6e57..b3be375 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/complex/FixedSizeListVector.java
@@ -295,7 +295,7 @@ public class FixedSizeListVector extends BaseValueVector implements FieldVector,
 
   @Override
   public long getValidityBufferAddress() {
-    return (bits.getBuffer().memoryAddress());
+    return (bits.getDataBuffer().memoryAddress());
   }
 
   @Override
@@ -308,6 +308,21 @@ public class FixedSizeListVector extends BaseValueVector implements FieldVector,
     throw new UnsupportedOperationException();
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() {
+    return (bits.getDataBuffer());
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
   public class Accessor extends BaseValueVector.BaseAccessor {
 
     @Override

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java b/java/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
index 4b2808d..79aca0c 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/complex/ListVector.java
@@ -180,7 +180,7 @@ public class ListVector extends BaseRepeatedValueVector implements FieldVector,
 
   @Override
   public long getValidityBufferAddress() {
-    return (bits.getBuffer().memoryAddress());
+    return (bits.getDataBuffer().memoryAddress());
   }
 
   @Override
@@ -190,9 +190,20 @@ public class ListVector extends BaseRepeatedValueVector implements FieldVector,
 
   @Override
   public long getOffsetBufferAddress() {
-    return (offsets.getBuffer().memoryAddress());
+    return (offsets.getDataBuffer().memoryAddress());
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() { return bits.getDataBuffer(); }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() { return offsets.getDataBuffer(); }
+
   private class TransferImpl implements TransferPair {
 
     ListVector to;

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java b/java/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
index d8d0964..f46635a 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/complex/MapVector.java
@@ -32,6 +32,8 @@ import com.google.common.base.Preconditions;
 import com.google.common.collect.Ordering;
 import com.google.common.primitives.Ints;
 
+import io.netty.buffer.ArrowBuf;
+
 import org.apache.arrow.memory.BufferAllocator;
 import org.apache.arrow.vector.BaseValueVector;
 import org.apache.arrow.vector.FieldVector;
@@ -130,6 +132,21 @@ public class MapVector extends AbstractMapVector {
   }
 
   @Override
+  public ArrowBuf getValidityBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
   public TransferPair getTransferPair(BufferAllocator allocator) {
     return getTransferPair(name, allocator, null);
   }

http://git-wip-us.apache.org/repos/asf/arrow/blob/c9805d6f/java/vector/src/main/java/org/apache/arrow/vector/complex/NullableMapVector.java
----------------------------------------------------------------------
diff --git a/java/vector/src/main/java/org/apache/arrow/vector/complex/NullableMapVector.java
b/java/vector/src/main/java/org/apache/arrow/vector/complex/NullableMapVector.java
index 09560c2..1c9d5aa 100644
--- a/java/vector/src/main/java/org/apache/arrow/vector/complex/NullableMapVector.java
+++ b/java/vector/src/main/java/org/apache/arrow/vector/complex/NullableMapVector.java
@@ -246,6 +246,21 @@ public class NullableMapVector extends MapVector implements FieldVector
{
     throw new UnsupportedOperationException();
   }
 
+  @Override
+  public ArrowBuf getValidityBuffer() {
+    return bits.getDataBuffer();
+  }
+
+  @Override
+  public ArrowBuf getDataBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
+  @Override
+  public ArrowBuf getOffsetBuffer() {
+    throw new UnsupportedOperationException();
+  }
+
   public final class Accessor extends MapVector.Accessor {
     final BitVector.Accessor bAccessor = bits.getAccessor();
 


Mime
View raw message