spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kiszk <...@git.apache.org>
Subject [GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...
Date Sun, 04 Mar 2018 05:43:52 GMT
Github user kiszk commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19222#discussion_r172037938
  
    --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OnHeapMemoryBlock.java
---
    @@ -0,0 +1,142 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *    http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +
    +package org.apache.spark.unsafe.memory;
    +
    +import org.apache.spark.unsafe.Platform;
    +
    +/**
    + * A consecutive block of memory with a long array on Java heap.
    + */
    +public final class OnHeapMemoryBlock extends MemoryBlock {
    +
    +  private final long[] array;
    +
    +  public OnHeapMemoryBlock(long[] obj, long offset, long size) {
    +    super(obj, offset, size);
    +    this.array = obj;
    +    assert(offset - Platform.LONG_ARRAY_OFFSET + size <= obj.length * 8L) :
    --- End diff --
    
    Thank you for your explanation. Now, we focus on the path 3. I understand what you pointed
out. How the following code should work?
    ```
    val mb = new OnHeapMemoryBlock(new long[16]), Platform.LONG_ARRAY_OFFSET, 8);
    val mb1 = mb.subBlock(0, 96);
    ```
    
    According to the name `subBlock()`, I understand that you would like to ensure `sub block's
size <= super block's size` (`not (96 <= 8)`)` logically.
    I would like to allow us to the above code since the array actually reserved enough space
(`96 <= 128`). This is because I have already seen such a code when we would use `MemoryBlock`
for `
    
    I. Add additional method `block()` (or different name) with this assertion. Keep `subBlock()`
with a new assert that you proposed.
    2. Use another name `block()` (or different name) instead of `subBlock()` with this assertion.
    
    WDYT? cc: @cloud-fan 



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message