arrow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krisztián Szűcs <szucs.kriszt...@gmail.com>
Subject Re: C++ and Python size problems with Arrow 0.13.0
Date Wed, 03 Apr 2019 08:32:10 GMT
On Wed, Apr 3, 2019 at 2:24 AM Wes McKinney <wesmckinn@gmail.com> wrote:

> hi folks,
>
> I that the arrow-cpp conda packages for Windows have ballooned in size
> to nearly 140 megabytes for RC4
>
>
> https://bintray.com/apache/arrow/python-rc/0.13.0-rc4#files/python-rc/0.13.0-rc4
>
> Looking at one of these packages it seems the Windows static libraries
> are huge -- I'm not sure why they are so big but we should probably
> investigate
>
> $ ll Library/lib/
> total 741796
> -rw-r--r-- 1 wesm wesm   1507048 Mar 27 23:34 arrow.lib
> -rw-r--r-- 1 wesm wesm     76184 Mar 27 23:35 arrow_python.lib
> -rw-r--r-- 1 wesm wesm  61322082 Mar 27 23:36 arrow_python_static.lib
> -rw-r--r-- 1 wesm wesm 328090044 Mar 27 23:37 arrow_static.lib
> drwxr-xr-x 3 wesm wesm      4096 Apr  2 19:12 cmake/
> -rw-r--r-- 1 wesm wesm    302496 Mar 27 23:38 gandiva.lib
> -rw-r--r-- 1 wesm wesm 239314018 Mar 27 23:40 gandiva_static.lib
> -rw-r--r-- 1 wesm wesm    491292 Mar 27 23:41 parquet.lib
> -rw-r--r-- 1 wesm wesm 128473780 Mar 27 23:42 parquet_static.lib
> drwxr-xr-x 2 wesm wesm      4096 Apr  2 19:12 pkgconfig/
>
> As a mitigating measure in the meantime, I would suggest that we stop
> bundling the static libraries in the arrow-cpp conda package, since
> we're just hurting release managers and users with a large package
> download when they `conda install pyarrow`. Can someone open a JIRA
> issue about this? If packaging the static libraries in conda is
> something that people need then we could create a separate
> arrow-cpp-static artifact
>
Agree, but I'm not sure what's the conda-forge policy for static libraries.

>
> The production packages in conda-forge are a bit smaller (less than
> 100 MB), but still quite large.
>
> https://anaconda.org/conda-forge/arrow-cpp/files
>
> I noticed also that the wheel Python packages on Linux have become
> quite large. The Python 3.7 wheel is 48.5 megabytes for example. The
> expected culprit is libgandiva.so, where I see
>
> -rwxr-xr-x 1 wesm wesm   131047 Apr  2 19:18 libarrow_boost_filesystem.so*
> -rwxr-xr-x 1 wesm wesm   131047 Apr  2 19:18
> libarrow_boost_filesystem.so.1.66.0*
> -rwxr-xr-x 1 wesm wesm  1253641 Apr  2 19:18 libarrow_boost_regex.so*
> -rwxr-xr-x 1 wesm wesm  1253641 Apr  2 19:18
> libarrow_boost_regex.so.1.66.0*
> -rwxr-xr-x 1 wesm wesm    30081 Apr  2 19:18 libarrow_boost_system.so*
> -rwxr-xr-x 1 wesm wesm    30081 Apr  2 19:18
> libarrow_boost_system.so.1.66.0*
> -rwxr-xr-x 1 wesm wesm  1613712 Apr  2 19:18 libarrow_python.so*
> -rwxr-xr-x 1 wesm wesm  1400561 Apr  2 19:18 libarrow_python.so.13*
> -rwxr-xr-x 1 wesm wesm 12543416 Apr  2 19:18 libarrow.so*
> -rwxr-xr-x 1 wesm wesm 11540172 Apr  2 19:18 libarrow.so.13*
> -rw-r--r-- 1 wesm wesm  6393593 Apr  2 19:18 lib.cpp
> -rwxr-xr-x 1 wesm wesm  2558504 Apr  2 19:18
> lib.cpython-37m-x86_64-linux-gnu.so*
> -rwxr-xr-x 1 wesm wesm 61260912 Apr  2 19:18 libgandiva.so*
> -rwxr-xr-x 1 wesm wesm 57342916 Apr  2 19:18 libgandiva.so.13*
> -rwxr-xr-x 1 wesm wesm  3567224 Apr  2 19:18 libparquet.so*
> -rwxr-xr-x 1 wesm wesm  3035367 Apr  2 19:18 libparquet.so.13*
> -rwxr-xr-x 1 wesm wesm   352440 Apr  2 19:18 libplasma.so*
> -rwxr-xr-x 1 wesm wesm   315802 Apr  2 19:18 libplasma.so.13*
>
> There's something very odd here, though, which is that libgandiva.so
> and libgandiva.so.13 appear to be distinct. They have different
> checksums, for example

This is true for libarrow, libparquet and libplasma as well. I've just
checked that
previous wheel ships shared libraries similarly.

>
> (pyarrow-0.13.0-py37-test) 19:19 ~/Downloads/arrow-cpp-py36-vc14 $
> sha256sum
> ~/miniconda/envs/pyarrow-0.13.0-py37-test/lib/python3.7/site-packages/pyarrow/libgandiva.so
> 8f1026d7bf476b90a0cac8239947ad334ee91cd31a944102aff6e8a67ae973e8
>
> /home/wesm/miniconda/envs/pyarrow-0.13.0-py37-test/lib/python3.7/site-packages/pyarrow/libgandiva.so
> (pyarrow-0.13.0-py37-test) 19:21 ~/Downloads/arrow-cpp-py36-vc14 $
> sha256sum
> ~/miniconda/envs/pyarrow-0.13.0-py37-test/lib/python3.7/site-packages/pyarrow/libgandiva.so.13
> 9969a50787f8e0411115c0bfffccd3a350fde5f8c2f319acd72f3cf8097365dc
>
> /home/wesm/miniconda/envs/pyarrow-0.13.0-py37-test/lib/python3.7/site-packages/pyarrow/libgandiva.so.13
>
In case of OSX wheels the checksums are equal, so I suspect auditwheel [1]
does some magic behind the curtain.

[1]
https://github.com/apache/arrow/blob/master/python/manylinux1/build_arrow.sh#L122

>
> That seems buggy to me. We might also investigate if there's a way to
> trim the binary sizes in some way.
>
> Thanks
> Wes
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message