MacにPython3とMKL+Numpyをインストール

なにも入っていないMacOSにPython3とMKL+Numpyをインストールした。brewだとかpyenvだとかvirtualenvとかもろもろ、MacとPython3のやり方諸説ありすぎて本当にカオスだと思った。あといい加減無用なpython2系をストレージ内に存在させておくのはやめてほしい。

リンクメモ（ほぼこれ）
Mac に MKL 版 numpy / scipy をインストールする - Qiita

pkgからPython3.8.2をインストール

https://www.python.org/downloads/

から普通にpkgを落としてインストールした。

curl -O https://www.python.org/ftp/python/3.8.2/python-3.8.2-macosx10.9.pkg
sudo installer -pkg python-3.8.2-macosx10.9.pkg -target /

どこからか持ってきたもの。普通にダブルクリックでやっても特に問題は起こらないはず。

$ python3 --version
Python 3.8.2

$pip3 list
Package    Version
---------- ----------
certifi    2020.4.5.1
pip        19.2.3
setuptools 41.2.0
WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip3 install --upgrade pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-20.0.2

パスは最初から通っている。最初にpip3をアップデートした。

MKLビルドのNumpy

xcode-select --install

xcode的なものをインストールした。GUIが出てきてインストールの確認ボタンみたいなのを押した。

https://software.seek.intel.com/performance-libraries

からIntel Math Kernel Libraryのインストーラをダウンロードして普通にインストールした。~/.numpy-site.cfgに次を書き込んだ。

[mkl]
library_dirs = /opt/intel/mkl/lib
include_dirs = /opt/intel/mkl/include
mkl_libs = mkl_rt

次にpipでビルドとインストールを実行する。

$ pip3 install --no-binary :all: numpy
Collecting numpy
  Downloading numpy-1.18.2.zip (5.4 MB)
     |████████████████████████████████| 5.4 MB 2.7 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: numpy
  Building wheel for numpy (PEP 517) ... done
  Created wheel for numpy: filename=numpy-1.18.2-cp38-cp38-macosx_10_14_x86_64.whl size=4509195 sha256=f01796d26c45a8988f7ac2581a2ae19fb73f41306e6d6f74e5c598d62818ec58
  Stored in directory: /Users/admin/Library/Caches/pip/wheels/19/9b/0c/54460c870ae69433c3730e09925740594be323ffd2ad5a0ae2
Successfully built numpy
Installing collected packages: numpy
Successfully installed numpy-1.18.2

ビルドしているのでかなり時間がかかった。しかし成功と出ているのでOK。

Numpyが起動しない

テンプレ通りにこの通りにやってもNumpyは起動しない。

$ python3
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/__init__.py", line 24, in <module>
    from . import multiarray
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
    from . import overrides
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
    from numpy.core._multiarray_umath import (
ImportError: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
  Reason: image not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
    from . import core
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/__init__.py", line 54, in <module>
    raise ImportError(msg)
ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed.
- Try uninstalling and reinstalling numpy.
- If you have already done that, then:
  1. Check that you expected to use Python3.8 from "/usr/local/bin/python3",
     and that you have no directories in your PATH or PYTHONPATH that can
     interfere with the Python and numpy version "1.18.2" you're trying to use.
  2. If (1) looks fine, you can open a new issue at
     https://github.com/numpy/numpy/issues.  Please include details on:
     - how you installed Python
     - how you installed numpy
     - your operating system
     - whether or not you have multiple versions of Python installed
     - if you built from source, your compiler versions and ideally a build log

- If you're working with a numpy git repository, try `git clean -xdf`
  (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on
an existing issue about this - open a new one instead.

Original error was: dlopen(/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
  Referenced from: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
  Reason: image not found

>>> quit()

ホントに元サイトと同じ。

$ install_name_tool -add_rpath /opt/intel/mkl/lib /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so

$ python3
Python 3.8.2 (v3.8.2:7b3ab5921f, Feb 24 2020, 17:52:18)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.show_config()
blas_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
blas_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
lapack_mkl_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
lapack_opt_info:
    libraries = ['mkl_rt', 'pthread']
    library_dirs = ['/opt/intel/mkl/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/opt/intel/mkl/include']
>>> quit()

若干パスは違うが言われた通りにやったら成功してちゃんとインポートできた。

Scipyはビルドできず

Scipyのビルド中にnumpyと同じようなモジュールが読み込めずにビルドがコケる。

$ pip3 install --no-binary :all: scipy
Collecting scipy
  Using cached scipy-1.4.1.tar.gz (24.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... error
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/tmpbk3ams59
         cwd: /private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-install-zlk81uk0/scipy
    Complete output (64 lines):
    setup.py:418: UserWarning: Unrecognized setuptools command ('dist_info --egg-base /private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-modern-metadata-0rlje4qe'), proceeding with generating Cython sources and expanding templates
      warnings.warn("Unrecognized setuptools command ('{}'), proceeding with "
    Traceback (most recent call last):
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/__init__.py", line 17, in <module>
        from . import multiarray
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/multiarray.py", line 14, in <module>
        from . import overrides
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/overrides.py", line 7, in <module>
        from numpy.core._multiarray_umath import (
    ImportError: dlopen(/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
      Referenced from: /private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
      Reason: image not found

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
        main()
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 158, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 249, in run_setup
        super(_BuildMetaLegacyBackend,
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
        exec(compile(code, __file__, 'exec'), locals())
      File "setup.py", line 540, in <module>
        setup_package()
      File "setup.py", line 516, in setup_package
        from numpy.distutils.core import setup
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/__init__.py", line 142, in <module>
        from . import core
      File "/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/__init__.py", line 47, in <module>
        raise ImportError(msg)
    ImportError:

    IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

    Importing the numpy c-extensions failed.
    - Try uninstalling and reinstalling numpy.
    - If you have already done that, then:
      1. Check that you expected to use Python3.8 from "/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8",
         and that you have no directories in your PATH or PYTHONPATH that can
         interfere with the Python and numpy version "1.17.3" you're trying to use.
      2. If (1) looks fine, you can open a new issue at
         https://github.com/numpy/numpy/issues.  Please include details on:
         - how you installed Python
         - how you installed numpy
         - your operating system
         - whether or not you have multiple versions of Python installed
         - if you built from source, your compiler versions and ideally a build log

    - If you're working with a numpy git repository, try `git clean -xdf`
      (removes all files not under version control) and rebuild numpy.

    Note: this error has many possible causes, so please don't comment on
    an existing issue about this - open a new one instead.

    Original error was: dlopen(/private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so, 2): Library not loaded: @rpath/libmkl_rt.dylib
      Referenced from: /private/var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/pip-build-env-kx_4xoe1/overlay/lib/python3.8/site-packages/numpy/core/_multiarray_umath.cpython-38-darwin.so
      Reason: image not found

    ----------------------------------------
ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8 /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/xw/f40tcc457klcppv72p5gjb4c0000gn/T/tmpbk3ams59 Check the logs for full command output.

同じようなアプローチでやろうとしたけどもパスがすごいことなっているしno such file or directoryになってうまくいかず。仕方なくScipyは普通にバイナリからインストールすることにした。

$ pip3 install scipy
Collecting scipy
  Using cached scipy-1.4.1-cp38-cp38-macosx_10_9_x86_64.whl (28.8 MB)
Requirement already satisfied: numpy>=1.13.3 in /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages (from scipy) (1.18.2)
Installing collected packages: scipy
Successfully installed scipy-1.4.1

なんかキャッシュが残っていたようだ。

まとめ

Unofficial Windows BinaryにはMKLビルドのnumpyバイナリが置かれているのになぁ。