本家様 https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Main_Page
MATLABが必要です.
ですが、ダウンロードしたファイルに MATLAB のRuntime版(コンパイル済みのMATLABファイルを実行できる)が入ってますので
それを使えばMATLAB(フル機能版)は不要です.
ここではダウンロードしたファイルに付属の MATLAB Runtime版でDynamoを動かす.
https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/DownloadsからLinux向けのtarファイルを取得します。
ファイル名: dynamo-v-1.1.532_MCR-9.9.0_GLNXA64_withMCR.tar (6.9GB)
開発環境と openmpi-devel を組み込んでおきます
[root@rockylinux ~]# dnf groupinstall "Development Tools"
[root@rockylinux ~]# dnf install openmpi-devel environment-modules
「environment-modules」を有効にさせるために、一度再ログインします
まずは展開します. 展開したら 7.3 GB 消費してました
[root@rockylinux ~]# mkdir -p /apps/dynamo/
[root@rockylinux ~]# cd /apps/dynamo/
[root@rockylinux dynamo]#
[root@rockylinux dynamo]# tar xf ~/dynamo-v-1.1.532_MCR-9.9.0_GLNXA64_withMCR.tar
[root@rockylinux dynamo]# du -hs .
7.3G .
[root@rockylinux dynamo]# ls -CF
C/ dynamo_activate_linux.sh* dynamo_setup_cluster.sh* matlab/ README_dynamo_installation.txt*
cuda/ dynamo_activate_linux_shipped_MCR.sh* dynamo_setup_linux.sh* MCRLinux/
ddemos/ dynamo_activate.m* examples/ mex/
doc/ dynamo_compile_mpi.sh* licenses/ mpi/
[root@rockylinux dynamo]#
次に OpenMP(OMP)プログラムをコンパイルします. いわゆる threads アプリです.
[root@rockylinux dynamo]# source dynamo_activate_linux_shipped_MCR.sh
MCR for linux has been found in location: /apps/dynamo/MCRLinux
Activating Dynamo as standalone
-------------------------------------------------------------
Dynamo Setup for standalone Linux
-------------------------------------------------------------
1- Adding execution paths
----------------------
Identifying directories of Dynamo distribution hanging from the specified Dynamo root directory:
/apps/dynamo
[OK] Done updating Linux paths.
2- Testing OpenMP
--------------
ls: cannot access '/apps/dynamo/mpi/dynamo_iteration_assemble_omp': No such file or directory <-- 無視して構わない
Binaries for OpenMP not ready: executables not found
------------------------------------
compiling OpenMP Wrappers
------------------------------------
Setting up executables of Dynamo installation on root location: /apps/dynamo
Trying to recompile the OpenMP wrappers on your system
iteration_setup compiled under OpenMP
iteration_compute compiled under OpenMP
iteration_assemble compiled under OpenMP
iteration_check compiled under OpenMP
ccmatrix_compute compiled under OpenMP
ccmatrix_assemble compiled under OpenMP
All OpenMP wrappers compiled successfully
checking functionality of OpenMP wrappers....
[OK] OpenMP tools seem functional
3- Testing MCR libraries
---------------------
[OK] MCR library seems active
------------------------------------
Dynamo is ready for standalone use in your Linux system
[root@rockylinux dynamo]#
(確認)
[root@rockylinux dynamo]# ls -l mpi/*omp
-rwxr-xr-x. 1 root 13449 18480 Oct 30 18:16 mpi/dynamo_ccmatrix_assemble_omp
-rwxr-xr-x. 1 root 13449 23040 Oct 30 18:16 mpi/dynamo_ccmatrix_compute_omp
-rwxr-xr-x. 1 root 13449 18480 Oct 30 18:16 mpi/dynamo_iteration_assemble_omp
-rwxr-xr-x. 1 root 13449 18480 Oct 30 18:16 mpi/dynamo_iteration_check_omp
-rwxr-xr-x. 1 root 13449 24968 Oct 30 18:16 mpi/dynamo_iteration_compute_omp
-rwxr-xr-x. 1 root 13449 18480 Oct 30 18:16 mpi/dynamo_iteration_setup_omp
[root@rockylinux dynamo]#
次に mpi プログラムをコンパイルします. 事前に mpi 実行/コンパイル環境をロードします
ドキュメントには「mpicc」が指定されているが「mpic++」でコンパイルされる.
[root@rockylinux dynamo]# module load mpi
[root@rockylinux dynamo]# which mpirun
/usr/lib64/openmpi/bin/mpirun
[root@rockylinux dynamo]#
[root@rockylinux dynamo]# source mpi/dynamo_compile_mpi.sh mpic++
Using selected compiler mpic++
trying to compile MPI wrappers for Dynamo standalone executables
/usr/lib64/openmpi/bin/mpic++
ok, mpic++ found, proceeding to compile MPI-wrappers....
./mpi/dynamo_iteration_setup_mpi.cc compiled
./mpi/dynamo_iteration_compute_mpi.cc compiled
./mpi/dynamo_iteration_assemble_mpi.cc compiled
./mpi/dynamo_iteration_check_mpi.cc compiled
./mpi/dynamo_ccmatrix_compute_mpi.cc compiled
./mpi/dynamo_ccmatrix_assemble_mpi.cc compiled
./mpi/dynamo_iteration_compute_mpi_gpu.cc compiled
Warnings are in principle ok
------------------------------------------------
MPI compilation SUCCESSFUL
------------------------------------------------
[root@rockylinux dynamo]#
[root@rockylinux dynamo]# ls -l mpi/*_mpi
-rwxr-xr-x. 1 root 13449 116424 Oct 30 18:25 mpi/dynamo_ccmatrix_assemble_mpi
-rwxr-xr-x. 1 root 13449 116696 Oct 30 18:25 mpi/dynamo_ccmatrix_compute_mpi
-rwxr-xr-x. 1 root 13449 116424 Oct 30 18:25 mpi/dynamo_iteration_assemble_mpi
-rwxr-xr-x. 1 root 13449 116424 Oct 30 18:25 mpi/dynamo_iteration_check_mpi
-rwxr-xr-x. 1 root 13449 121960 Oct 30 18:25 mpi/dynamo_iteration_compute_mpi
-rwxr-xr-x. 1 root 13449 116424 Oct 30 18:25 mpi/dynamo_iteration_setup_mpi
[root@rockylinux dynamo]#
最後に cuda アプリケーションをコンパイル
既にcuda-8.0で作られたバイナリーが提供されているが、ここでは cuda-11.8 でコンパイルし直します.
[root@rockylinux dynamo]# ls -l cuda/bin
total 284
-rwxrwxrwx. 1 4121 13449 290304 Aug 27 2021 dynamo_compute_iteration_spp
[root@rockylinux dynamo]# ldd cuda/bin/dynamo_compute_iteration_spp
linux-vdso.so.1 (0x00007ffd42be1000)
libcudart.so.8.0 => not found
libcufft.so.8.0 => not found
libcublas.so.8.0 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3d8dbf4000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f3d8d85f000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3d8d4dd000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f3d8d2c5000)
libc.so.6 => /lib64/libc.so.6 (0x00007f3d8cf00000)
/lib64/ld-linux-x86-64.so.2 (0x00007f3d8de14000)
[root@rockylinux dynamo]#
(退避)
[root@rockylinux dynamo]# mv cuda/bin/dynamo_compute_iteration_spp cuda/bin/dynamo_compute_iteration_spp.old
(re-compile)
[root@rockylinux dynamo]# cd cuda/
[root@rockylinux cuda]# export PATH=/usr/local/cuda/bin:$PATH
[root@rockylinux cuda]# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
[root@rockylinux cuda]# source config.sh
----------------------------------------------------------------
Configuring makefile for the CUDA installation in your system
Looking for a CUDA installation in this environment:
Found CUDA nvcc compiler at /usr/local/cuda/bin/nvcc
CUDA installation assumed to be in /usr/local/cuda
we will use /lib64 (standard procedure)
Substituting the CUDA location assignment:
CUDA_ROOT=/usr/local/cuda-8.0
with the actual value:
CUDA_ROOT=/usr/local/cuda
[ok]
you can now compile the CUDA utilities by :
make all
or :
make motors
if "make all" produces an error when trying to compile the Nvidia utility infoGPU
------------------------------------------------------------------
[root@rockylinux cuda]# make all
/usr/local/cuda/bin/nvcc -o bin/dynamo_compute_iteration_spp src/dynamo_compute_iteration_spp.cu -lcudart -lcufft -lcublas -L/usr/local/cuda/lib64 -I/usr/local/cuda/include -Isrc -Isrc/classes src/classes/gpuAngles.cpp -w
echo 'Compilation of Dynamo completed. Use make extended to include Nvidia GPU'
Compilation of Dynamo completed. Use make extended to include Nvidia GPU
[root@rockylinux cuda]#
(確認)
[root@rockylinux cuda]# ldd bin/dynamo_compute_iteration_spp
linux-vdso.so.1 (0x00007ffd273f9000)
libcudart.so.11.0 => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0 (0x00007fe22e7a0000)
libcufft.so.10 => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.10 (0x00007fe21d8c5000)
libcublas.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.11 (0x00007fe217c67000)
librt.so.1 => /lib64/librt.so.1 (0x00007fe217a5f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe21783f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fe21763b000)
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fe2172a6000)
libm.so.6 => /lib64/libm.so.6 (0x00007fe216f24000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fe216d0c000)
libc.so.6 => /lib64/libc.so.6 (0x00007fe216947000)
/lib64/ld-linux-x86-64.so.2 (0x00007fe22ea47000)
libcublasLt.so.11 => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.11 (0x00007fe1f23c1000)
[root@rockylinux cuda]#
以上でインストールは完了となる
環境設定を行います
「source /apps/Dynamo/dynamo_activate_linux_shipped_MCR.sh silent」と末尾に「silent」を入れると
メッセージの表示が抑えられる.
[illya@rockylinux ~]$ source /apps/dynamo/dynamo_activate_linux_shipped_MCR.sh silent
MCR for linux has been found in location: /apps/dynamo/MCRLinux
Activating Dynamo as standalone
[illya@rockylinux ~]$ dynamo
Initializing MATLAB Runtime version 9.9
"MCR libraries starting dynamo. Starting the Dynamo console can take some time. A good choice of MCR_CACHE_ROOT might speed this up."
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Warning: MCR_CACHE_ROOT is empty.
Initialize this variable in your shell for a quicker startup.
To know more, type:
dwiki MCR_CACHE_ROOT
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Warning: MCR_CACHE_ROOT points to non-existing location:
Correct this variable in your shell for a quicker startup.
To know more, type:
dwiki MCR_CACHE_ROOT
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Launching the console....
(use "dynamo x" for a Dynamo graphics terminal on a independent window)
Dynamo console for standalone modus
-----------------------------------
* Commands starting with the symbol "!" will be passed to the system
* Commands starting with the symbol "\" will be executed as Matlab expressions
* Strings including the symbol ":" are treated as database queries
* Graphic output: enabled
* Type "help" for a list of console control commands
Dynamo >
GUI画面を出してみる
Dynamo > dcm
Passing to MATLAB...
Resize selection
Resize selection
Dynamo >
これで下記画面が表示される
あるいは
[illya@rockylinux ~]$ source /apps/dynamo/dynamo_activate_linux_shipped_MCR.sh silent
MCR for linux has been found in location: /apps/dynamo/MCRLinux
Activating Dynamo as standalone
[illya@rockylinux ~]$ dynamo x
これで下記画面が表示されます.
https://wiki.dynamo.biozentrum.unibas.ch/w/index.php/Standalone
MATLABで動かすには Dynamo/MATLAB
「source /apps/dynamo/dynamo_activate_linux_shipped_MCR.sh」が面倒なのでmodulefilesを作ってみた
[root@rockylinux ~]# vi /apps/modulefiles/dynamo
#%Module1.0
puts stdout "source /apps/local/bin/dynamo-mkdir;"
set dynamo /apps/dynamo
setenv DYNAMO_FOLDER $dynamo
setenv MCR_CACHE_ROOT /tmp/mcr_cache_root_$env(USER)
prepend-path LD_LIBRARY_PATH $dynamo/MCRLinux/runtime/glnxa64:$dynamo/MCRLinux/bin/glnxa64:$dynamo/MCRLinux/sys/os/glnxa64
prepend-path PATH $dynamo/matlab/bin:$dynamo/matlab/src:$dynamo/cuda/bin:$dynamo/mpi:$dynamo/examples:$dynamo/doc
#prepend-path LD_PRELOAD $dynamo/MCRLinux/bin/glnxa64/glibc-2.17_shim.so
[root@rockylinux ~]#
あと「/apps/local/bin/dynamo-mkdir」を用意します
#!/bin/bash
if [ ! -d /tmp/mcr_cache_root_$USER ]; then
/bin/mkdir /tmp/mcr_cache_root_$USER
fi
この「/apps/local/bin/dynamo-mkdir」には実行権(+x)を設けます[chmod +x /apps/local/bin/dynamo-mkdir]
上記をロードすれば(module load dynamo)、「source /apps/dynamo/dynamo_activate_linux_shipped_MCR.sh」の代わりになります.
事前に「module use --append /apps/modulefiles」が必要です
「/apps/dynamo/dynamo_compile_mpi.sh」と「/apps/dynamo/mpi/dynamo_compile_mpi.sh」は同じファイル
環境変数MCR_CACHE_ROOTを ssd/nvme なデバイスに指定すると早くなるそうな. ccp4のcacheと同じ思想かな
[illya@rockylinux ~]$ mkdir /scratch/dynamo
[illya@rockylinux ~]$ export MCR_CACHE_ROOT=/scratch/dynamo
version確認は、「dynamo_version」で得られる
[illya@rockylinux ~]$ module use --append /apps/modulefiles
[illya@rockylinux ~]$ module load dynamo
[illya@rockylinux ~]$ dynamo
Initializing MATLAB Runtime version 9.9
"MCR libraries starting dynamo. Starting the Dynamo console can take some time. A good choice of MCR_CACHE_ROOT might speed this up."
Launching the console....
(use "dynamo x" for a Dynamo graphics terminal on a independent window)
Dynamo console for standalone modus
-----------------------------------
* Commands starting with the symbol "!" will be passed to the system
* Commands starting with the symbol "\" will be executed as Matlab expressions
* Strings including the symbol ":" are treated as database queries
* Graphic output: enabled
* Type "help" for a list of console control commands
Dynamo > dynamo_version
Passing order to Matlab engine:
dynamo_version
Dynamo version: v-1.1.532
ans =
'v-1.1.532'
Dynamo > quit
ok, bye!
Exiting Dynamo console.
Returning control to system.
Note after exiting the Dynamo console:
if you experience difficulties with the text display you might need to type:
stty echo
in your system shell to fix them.
[delete:pmodelpool.Manager] Deletion of model pool manager from memory.
[illya@rockylinux ~]$
AMD Ryzen CPUだとmodulefilesの末尾「prepend-path LD_PRELOAD $dynamo/MCRLinux/bin/glnxa64/glibc-2.17_shim.so」は無効にする事
#prepend-path LD_PRELOAD $dynamo/MCRLinux/bin/glnxa64/glibc-2.17_shim.so