mahout-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Trevor Grant <trevor.d.gr...@gmail.com>
Subject Re: Errors when 0.13.0 master or vcl
Date Fri, 28 Oct 2016 23:01:26 GMT
I was finally able to hit an OOM error.

I initially was running everything from the top.

When I run mvn test from the viennacl directory...


/mahout/viennacl$ mvn test
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model
for org.apache.mahout:mahout-native-viennacl_2.10:jar:0.13.0-SNAPSHOT
[WARNING] 'artifactId' contains an expression but should be a constant. @
org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version],
/home/trevor/gits/mahout/viennacl/pom.xml, line 31, column 15
[WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but
found duplicate declaration of plugin org.codehaus.mojo:exec-maven-plugin @
org.apache.mahout:mahout-native-viennacl_${scala.compat.version}:[unknown-version],
/home/trevor/gits/mahout/viennacl/pom.xml, line 181, column 15
[WARNING]
[WARNING] It is highly recommended to fix these problems because they
threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
[INFO]

[INFO]
------------------------------------------------------------------------
[INFO] Building Mahout Native VienniaCL OpenCL Bindings 0.13.0-SNAPSHOT
[INFO]
------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:add-source (add-scala-sources) @
mahout-native-viennacl_2.10 ---
[INFO] Add Source directory:
/home/trevor/gits/mahout/viennacl/src/main/scala
[INFO] Add Test Source directory:
/home/trevor/gits/mahout/viennacl/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.3:properties (default) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
mahout-native-viennacl_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.7:resources (default-resources) @
mahout-native-viennacl_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/trevor/gits/mahout/viennacl/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:exec (javacpp) @
mahout-native-viennacl_2.10 ---
Warning: Could not load platform properties for class
org.apache.mahout.viennacl.opencl.GPUMMul
Warning: Could not load platform properties for class
org.apache.mahout.viennacl.opencl.GPUMMul$
Generating
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
Compiling
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/linux-x86_64/libjniViennaCL.so
g++ -I/usr/include/viennacl -I/usr/lib/jvm/java-7-openjdk-amd64/include
-I/usr/lib/jvm/java-7-openjdk-amd64/include/linux
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
-msse3 -ffast-math -fopenmp -fpermissive -Wl,-rpath,$ORIGIN/
-Wl,-z,noexecstack -Wl,-Bsymbolic -march=native -m64 -Wall -Ofast -fPIC
-shared -s -o libjniViennaCL.so -lOpenCL
Deleting
/home/trevor/gits/mahout/viennacl/target/classes/org/apache/mahout/viennacl/opencl/javacpp/jniViennaCL.cpp
[INFO]
[INFO] --- maven-resources-plugin:2.7:testResources (default-testResources)
@ mahout-native-viennacl_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/home/trevor/gits/mahout/viennacl/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.0:testCompile (scala-test-compile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @
mahout-native-viennacl_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @
mahout-native-viennacl_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @
mahout-native-viennacl_2.10 ---
Discovery starting.
Discovery completed in 201 milliseconds.
Run starting. Expected test count is: 7
ViennaCLSuiteVCL:
- row-major viennacl::matrix
  + OCL matrix memory domain after assgn=2
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
log4j:WARN No appenders could be found for logger
(org.apache.mahout.viennacl.opencl.GPUMMul$).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
- dense vcl mmul with fast_copy
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
jvmRWRW
gpuRWCW
- mmul microbenchmark
  + Mahout multiplication time: 2630 ms.
  + ViennaCL/OpenCL multiplication time: 2072 ms.
  + ViennaCL/cpu/OpenMP multiplication time: 813 ms.
- trans
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
gpuSparseRWRW
ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'.
ViennaCL: Smaller work sizes could not solve the problem.
ViennaCL: FATAL ERROR: CL_MEM_OBJECT_ALLOCATION_FAILURE
 ViennaCL could not allocate memory on the device. Most likely the device
simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at
viennacl-support@lists.sourceforge.net and supply at least the following
information:
 * Operating System
 * Which OpenCL implementation (AMD, NVIDIA, etc.)
 * ViennaCL version
Many thanks in advance!falling back to JVM MMUL
ViennaCL: FATAL ERROR: Kernel start failed for 'spgemm_stage3'.
ViennaCL: Smaller work sizes could not solve the problem.
- sparse mmul microbenchmark *** FAILED ***
  java.lang.RuntimeException: ViennaCL: FATAL ERROR:
CL_MEM_OBJECT_ALLOCATION_FAILURE
 ViennaCL could not allocate memory on the device. Most likely the device
simply ran out of memory.
If you think that this is a bug in ViennaCL, please report it at
viennacl-support@lists.sourceforge.net and supply at least the following
information:
 * Operating System
 * Which OpenCL implementation (AMD, NVIDIA, etc.)
 * ViennaCL version
Many thanks in advance!
  at
org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.allocate(Native
Method)
  at
org.apache.mahout.viennacl.opencl.javacpp.CompressedMatrix.<init>(CompressedMatrix.scala:61)
  at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply$mcV$sp(ViennaCLSuiteVCL.scala:250)
  at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218)
  at
org.apache.mahout.viennacl.opencl.ViennaCLSuiteVCL$$anonfun$5.apply(ViennaCLSuiteVCL.scala:218)
  at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
  at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  ...
  + Mahout Sparse multiplication time: 13303 ms.
- VCL Dense Matrix %*% Dense vector
  + Mahout dense matrix %*% dense vector multiplication time: 0 ms.
  + ViennaCL/cpu/OpenMP dense matrix %*% dense vector multiplication time:
4 ms.
[INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
[INFO] Successfully created org.apache.mahout.viennacl.opencl.GPUMMul solver
gpuRWCW
- Sparse %*% Dense mmul microbenchmark
  + Mahout multiplication time: 821 ms.
  + ViennaCL/OpenCL multiplication time: 754 ms.
  + ViennaCL/cpu/OpenMP multiplication time: 1473 ms.
Run completed in 26 seconds, 944 milliseconds.
Total number of tests run: 7
Suites: completed 2, aborted 0
Tests: succeeded 6, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 45.600 s
[INFO] Finished at: 2016-10-28T17:58:45-05:00
[INFO] Final Memory: 20M/309M
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.scalatest:scalatest-maven-plugin:1.0:test (test) on project
mahout-native-viennacl_2.10: There are test failures -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException


Trevor Grant
Data Scientist
https://github.com/rawkintrevo
http://stackexchange.com/users/3002022/rawkintrevo
http://trevorgrant.org

*"Fortunate is he, who is able to know the causes of things."  -Virgil*


On Fri, Oct 28, 2016 at 5:38 PM, Andrew Musselman <
andrew.musselman@gmail.com> wrote:

> >
> > My Desktop
>
> >
> Branch: master
>
> > Build Command: mvn clean install -Phadoop2
>
> > Built and tested successfully
>
> >
> Branch: andrewpalumbo/mahout/viennacl-opmmul-a
>
> > Bulid Command: mvn clean install -Pviennacl -Phadoop2
>
> Builds successfully-
>
> > Tests still riddled with
>
> >
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
>
> > [WARN] Unable to create class GPUMMul: attempting OpenMP version
>
> > [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
>
> > [INFO] Unable to create class OMPMMul: falling back to java version
>
> >
> Also OOM
> ViennaCLSuiteOMP:
> - row-major viennacl::matrix
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> - mmul microbenchmark
>   + Mahout multiplication time: 8875 ms.
>   + ViennaCL/cpu/OpenMP multiplication time: 599 ms.
> - trans
> [INFO] Creating org.apache.mahout.viennacl.opencl.GPUMMul solver
> [WARN] Unable to create class GPUMMul: attempting OpenMP version
> [INFO] Creating org.apache.mahout.viennacl.openmp.OMPMMul solver
> [INFO] Unable to create class OMPMMul: falling back to java version
> *** RUN ABORTED ***
>   java.lang.OutOfMemoryError: Java heap space
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.rehash(
> Int2DoubleOpenHashMap.java:1059)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.insert(
> Int2DoubleOpenHashMap.java:295)
>   at
> it.unimi.dsi.fastutil.ints.Int2DoubleOpenHashMap.put(
> Int2DoubleOpenHashMap.java:301)
>   at
> org.apache.mahout.math.RandomAccessSparseVector.setQuick(
> RandomAccessSparseVector.java:130)
>   at
> org.apache.mahout.math.SparseRowMatrix.setQuick(SparseRowMatrix.java:105)
>   at org.apache.mahout.math.AbstractMatrix.assign(AbstractMatrix.java:256)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$
> colon$eq(MatrixOps.scala:192)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.cloned(MatrixOps.scala:260)
>   at
> org.apache.mahout.math.scalabindings.MatrixOps.$minus(MatrixOps.scala:66)
>   at
> org.apache.mahout.viennacl.openmp.ViennaCLSuiteOMP$$
> anonfun$4.apply$mcV$sp(ViennaCLSuiteOMP.scala:152)
>
> >
>
> Machine Info:
>
> > ============================================================
> ====================
>
> >
>  os major/minor:
>
> > $ lsb_release -a
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 16.04.1 LTS
> Release: 16.04
> Codename: xenial
>
> >
>
>  chip arch:
>
> > $ uname -a
> Linux Bob 4.4.0-34-generic #53-Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
>
> >
> $ cat /proc/cpuinfo
>
> > ...
>
> > model name : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
>
> > ...
>
> > ^^ 6 cores
>
> >
>
> GPU:
>
> > $ sudo nvidia-smi
>
> >
> +-----------------------------------------------------------
> ------------------+
> | NVIDIA-SMI 367.44                 Driver Version: 367.44
>    |
> |-------------------------------+----------------------+----
> ------------------+
> | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr.
> ECC |
> | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute
> M. |
> |===============================+======================+====
> ==================|
> |   0  GeForce GTX 750 Ti  Off  | 0000:01:00.0      On |
>  N/A |
> | 29%   25C    P8     1W /  38W |    307MiB /  1998MiB |      0%
>  Default |
> +-------------------------------+----------------------+----
> ------------------+
>
>
> +-----------------------------------------------------------
> ------------------+
> | Processes:                                                       GPU
> Memory |
> |  GPU       PID  Type  Process name                               Usage
>    |
> |===========================================================
> ==================|
> |    0      1537    G   ...C-EnableWebRtcEcdsa,WebRTC-H264WithOpenH2
>  87MiB |
> |    0      1990    G   /usr/lib/xorg/Xorg
> 101MiB |
> |    0      2319    G   /usr/bin/gnome-shell
> 104MiB |
> |    0      4360    G   /usr/lib/xorg/Xorg
>  12MiB |
> +-----------------------------------------------------------
> ------------------+
>
> >
>
>
>
>  clinfo output:
>
> > $ clinfo
> Number of platforms                               1
>   Platform Name                                   NVIDIA CUDA
>   Platform Vendor                                 NVIDIA Corporation
>   Platform Version                                OpenCL 1.2 CUDA 8.0.0
>   Platform Profile                                FULL_PROFILE
>   Platform Extensions
> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics
> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing
> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll
> cl_nv_copy_opts
>   Platform Extensions function suffix             NV
>
>   Platform Name                                   NVIDIA CUDA
> Number of devices                                 1
>   Device Name                                     GeForce GTX 750 Ti
>   Device Vendor                                   NVIDIA Corporation
>   Device Vendor ID                                0x10de
>   Device Version                                  OpenCL 1.2 CUDA
>   Driver Version                                  367.44
>   Device OpenCL C Version                         OpenCL C 1.2
>   Device Type                                     GPU
>   Device Profile                                  FULL_PROFILE
>   Device Topology (NV)                            PCI-E, 01:00.0
>   Max compute units                               5
>   Max clock frequency                             1150MHz
>   Compute Capability (NV)                         5.0
>   Device Partition                                (core)
>     Max number of sub-devices                     1
>     Supported partition types                     None
>   Max work item dimensions                        3
>   Max work item sizes                             1024x1024x64
>   Max work group size                             1024
>   Preferred work group size multiple              32
>   Warp size (NV)                                  32
>   Preferred / native vector sizes
>     char                                                 1 / 1
>     short                                                1 / 1
>     int                                                  1 / 1
>     long                                                 1 / 1
>     half                                                 0 / 0        (n/a)
>     float                                                1 / 1
>     double                                               1 / 1
>  (cl_khr_fp64)
>   Half-precision Floating-point support           (n/a)
>   Single-precision Floating-point support         (core)
>     Denormals                                     Yes
>     Infinity and NANs                             Yes
>     Round to nearest                              Yes
>     Round to zero                                 Yes
>     Round to infinity                             Yes
>     IEEE754-2008 fused multiply-add               Yes
>     Support is emulated in software               No
>     Correctly-rounded divide and sqrt operations  Yes
>   Double-precision Floating-point support         (cl_khr_fp64)
>     Denormals                                     Yes
>     Infinity and NANs                             Yes
>     Round to nearest                              Yes
>     Round to zero                                 Yes
>     Round to infinity                             Yes
>     IEEE754-2008 fused multiply-add               Yes
>     Support is emulated in software               No
>     Correctly-rounded divide and sqrt operations  No
>   Address bits                                    64, Little-Endian
>   Global memory size                              2095841280 (1.952GiB)
>   Error Correction support                        No
>   Max memory allocation                           523960320 (499.7MiB)
>   Unified memory for Host and Device              No
>   Integrated memory (NV)                          No
>   Minimum alignment for any data type             128 bytes
>   Alignment of base address                       4096 bits (512 bytes)
>   Global Memory cache type                        Read/Write
>   Global Memory cache size                        81920
>   Global Memory cache line                        128 bytes
>   Image support                                   Yes
>     Max number of samplers per kernel             32
>     Max size for 1D images from buffer            134217728 pixels
>     Max 1D or 2D image array size                 2048 images
>     Max 2D image size                             16384x16384 pixels
>     Max 3D image size                             4096x4096x4096 pixels
>     Max number of read image args                 256
>     Max number of write image args                16
>   Local memory type                               Local
>   Local memory size                               49152 (48KiB)
>   Registers per block (NV)                        65536
>   Max constant buffer size                        65536 (64KiB)
>   Max number of constant args                     9
>   Max size of kernel argument                     4352 (4.25KiB)
>   Queue properties
>     Out-of-order execution                        Yes
>     Profiling                                     Yes
>   Prefer user sync for interop                    No
>   Profiling timer resolution                      1000ns
>   Execution capabilities
>     Run OpenCL kernels                            Yes
>     Run native kernels                            No
>     Kernel execution timeout (NV)                 Yes
>   Concurrent copy and kernel execution (NV)       Yes
>     Number of async copy engines                  1
>   printf() buffer size                            1048576 (1024KiB)
>   Built-in kernels
>   Device Available                                Yes
>   Compiler Available                              Yes
>   Linker Available                                Yes
>   Device Extensions
> cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics
> cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics
> cl_khr_fp64 cl_khr_byte_addressable_store cl_khr_icd cl_khr_gl_sharing
> cl_nv_compiler_options cl_nv_device_attribute_query cl_nv_pragma_unroll
> cl_nv_copy_opts
>
> NULL platform behavior
>   clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  NVIDIA CUDA
>   clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [NV]
>   clCreateContext(NULL, ...) [default]            Success [NV]
>   clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in
> platform
>   clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  No platform
>   clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices
> found in platform
>   clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in
> platform
>   clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  No platform
>
> ICD loader properties
>   ICD loader Name                                 OpenCL ICD Loader
>   ICD loader Vendor                               OCL Icd free software
>   ICD loader Version                              2.2.8
>   ICD loader Profile                              OpenCL 1.2
> NOTE: your OpenCL library declares to support OpenCL 1.2,
> but it seems to support up to OpenCL 2.1 too.
>
> >
>
>  Java major/minor and vendor:
>
> > $ java -version
> openjdk version "1.8.0_91"
> OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-3ubuntu1~16.04.1-b14)
> OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)
>
> >
>
>   vcl version,
>
> > (Ignoring what is in /usr/include/viennacl/version.hpp)
>
> > 1.7.1
>
>
> gcc version:
>
> > $ gcc --version
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609
>
> >
>
> On Wed, Oct 26, 2016 at 11:31 PM, Andrew Palumbo <ap.dev@outlook.com>
> wrote:
>
> >
> >
> >
> > Could everybody please post any errors when stack trace that they are
> > getting on master or the vcl pr branch along with full system info?
> >
> > Ie.  Branch, os major/minor, chip arch, GPU, clinfo output, Java
> > major/minor and vendor,  vcl version, gcc version, and anything else that
> > may be useful so that we may compare?
> >
> > Thx
> >
> > Andy
> >
> >
> > Sent from my Galaxy Tab A
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message