![[PukiWiki]](image/picc.png "[PukiWiki]")
#author("2024-11-29T13:59:19+00:00;2024-06-02T07:20:04+00:00","default:sysosa","sysosa") #author("2024-11-29T14:03:51+00:00;2024-06-02T07:20:04+00:00","default:sysosa","sysosa") 本家様 [[https://github.com/rsanchezgarc/deepEMhancer>+https://github.com/rsanchezgarc/deepEMhancer]] gitには #code(nonumber){{ Simply speaking, DeepEMhancer performs a non-linear post-processing of cryo-EM maps that produces two main effects: 1. Local sharpening-like post-processing. 2. Automatic masking/denoising of cryo-EM maps. }} とある. cryoSPARCからもコールされるようになっている. インストール環境はこんな感じ #code(nonumber){{ [root@rockylinux9 ~]# cat /etc/redhat-release Rocky Linux release 9.4 (Blue Onyx) [root@rockylinux9 ~]# cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 535.179 Fri Apr 26 21:43:18 UTC 2024 GCC version: gcc version 11.4.1 20231218 (Red Hat 11.4.1-3) (GCC) [root@rockylinux9 ~]# }} 「/usr/local/cuda」は用意していないです ***インストール [#j5b5561d] DeepEMhancerはpythonアプリなので、ここでは[[crYOLO]]のようにpyenv/anacondaの環境にDeepEMhancer実行環境を用意していきます pyenv/anaconda環境がないなら下記のように敷設して #code(nonumber){{ git clone https://github.com/yyuu/pyenv.git /apps/pyenv export PYENV_ROOT=/apps/pyenv export PATH=$PYENV_ROOT/bin:$PATH pyenv install anaconda3-2024.02-1 pyenv global anaconda3-2024.02-1 source /apps/pyenv/versions/anaconda3-2024.02-1/etc/profile.d/conda.sh }} もし既にpyenv/anaconda環境があるならsourceで環境を呼び込みます. #code(nonumber){{ source /apps/pyenv/versions/anaconda3-2024.02-1/etc/profile.d/conda.sh }} っで、DeepEMhancer を入れていきます. #code(nonumber){{ [root@rockylinux9 ~]# cd /apps/ [root@rockylinux9 apps]# git clone https://github.com/rsanchezgarc/deepEMhancer [root@rockylinux9 apps]# cd deepEMhancer [root@rockylinux9 deepEMhancer]# ls -CF alternative_installation/ condaDeepEMHancer/ conda_ymls/ deepEMhancer/ deepEMhancer_env.yml deepEMhancer_tutorial.pdf LICENSE README.md setup.py [root@rockylinux deepEMhancer]# conda env create -f deepEMhancer_env.yml -n deepEMhancer [root@rockylinux deepEMhancer]# }} 作った deepEMhancer 実行環境に対して #code(nonumber){{ [root@rockylinux deepEMhancer]# conda activate deepEMhancer (deepEMhancer) [root@rockylinux9 deepEMhancer]# conda list : cudatoolkit 11.8.0 h4ba93d1_13 conda-forge : h5py 3.9.0 nompi_py39h87cadad_103 conda-forge : nvidia-cublas-cu11 11.11.3.6 pypi_0 pypi nvidia-cudnn-cu11 8.6.0.163 pypi_0 pypi : tensorflow 2.12.0 pypi_0 pypi : (deepEMhancer) [root@rockylinux9 deepEMhancer]# which python /apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/bin/python (deepEMhancer) [root@rockylinux9 deepEMhancer]# pwd /apps/deepEMhancer (deepEMhancer) [root@rockylinux9 deepEMhancer]# python -m pip install . --no-deps (deepEMhancer) [root@rockylinux9 deepEMhancer]# conda list : deepemhancer 0.16 pypi_0 pypi : (deepEMhancer) [root@rockylinux9 deepEMhancer]# }} これで一応インストールは完了です ***動作確認 [#e0d8fbf1] GPUを利用して計算速度を向上できます 実際にGPUが使えるかのテストをしてみます #code(nonumber){{ (deepEMhancer) [root@rockylinux9 deepEMhancer]# deepemhancer --version バージョン確認 0.16 (deepEMhancer) [root@rockylinux9 deepEMhancer]# python >>> >>> import tensorflow as tf 2024-06-02 02:00:18.071439: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-06-02 02:00:18.097133: I tensorflow/tsl/cuda/cudart_stub.cc:28] Could not find cuda drivers on your machine, GPU will not be used. 2024-06-02 02:00:18.097475: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-06-02 02:00:18.545798: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT /apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib/python3.9/site-packages/h5py/__init__.py:36: UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2, this may cause problems _warn(("h5py is running against HDF5 {0} when it was built against {1}, " >>> >>> }} と検証前に「UserWarning: h5py is running against HDF5 1.14.3 when it was built against 1.14.2」や「TF-TRT Warning: Could not find TensorRT」「Could not find cuda drivers on your machine, GPU will not be used」と言われる. h5pyに関しては現状 h5py 3.9.0 を h5py 3.10.0 にすることで抑えられる. TensorRTに関しては新規に tensorrt 8.6.1 と LD_LIBRARY_PATH の定義で抑えられる 「Could not find cuda drivers...」も LD_LIBRARY_PATH で抑えられる #code(nonumber){{ (deepEMhancer) [root@rockylinux9 deepEMhancer]# pip install h5py==3.10.0 tensorrt==8.6.1 (deepEMhancer) [root@rockylinux9 deepEMhancer]# export LD_LIBRARY_PATH=/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib (deepEMhancer) [root@rockylinux9 deepEMhancer]# export LD_LIBRARY_PATH=/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib/python3.9/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH (deepEMhancer) [root@rockylinux9 deepEMhancer]# export LD_LIBRARY_PATH=/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib/python3.9/site-packages/tensorrt_libs:$LD_LIBRARY_PATH (deepEMhancer) [root@rockylinux9 deepEMhancer]# python >>> >>> import tensorflow as tf 2024-06-02 02:11:02.483195: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. >>> (っでGPU確認) >>> >>> tf.test.is_gpu_available() (略) True >>> quit(); (deepEMhancer) [root@rockylinux9 deepEMhancer]# conda deactivate [root@rockylinux9 deepEMhancer]# }} ***EnvironmentModules [#r45f11d0] #code(nonumber){{ [root@rockylinux9 ~]# vi /apps/modulefiles/deepEMhancer #%Module1.0 set root /apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer prepend-path PATH $root/bin prepend-path LD_LIBRARY_PATH $root/lib:$root/lib/python3.9/site-packages/nvidia/cudnn/lib:$root/lib/python3.9/site-packages/tensorrt_libs [root@rockylinux9 ~]# }} ***つかう [#v940e17c] モデルファイルが必要ですが、/appsとかに配置しておきます #code(nonumber){{ [root@rockylinux9 ~]# module load /apps/modulefiles [root@rockylinux9 ~]# module load deepEMhancer [root@rockylinux9 ~]# deepemhancer --download /apps/deepEMhancerModels [root@rockylinux9 ~]# ls -l /apps/deepEMhancerModels/ total 0 drwxr-xr-x. 2 root root 160 Jun 2 02:27 production_checkpoints [root@rockylinux9 ~]# ls -l /apps/deepEMhancerModels/production_checkpoints/ total 904376 -rw-r--r--. 1 root root 311505312 Jun 2 02:27 deepEMhancer_highRes.hd5 -rw-r--r--. 1 root root 204777344 Jun 2 02:27 deepEMhancer_masked.hd5 -rw-r--r--. 1 root root 204886472 Jun 2 02:27 deepEMhancer_tightTarget.hd5 -rw-r--r--. 1 root root 204897936 Jun 2 02:27 deepEMhancer_wideTarget.hd5 -rw-r--r--. 1 root root 74 Jun 2 02:27 versions.txt [root@rockylinux9 ~]# cat /apps/deepEMhancerModels/production_checkpoints/versions.txt highRes: g2_3 tightTarget: v3_L_c_2 wideModel: v3t_28_c+ masked: vt2_m_28 [root@rockylinux9 ~]# }} っでテストrunです. #code(nonumber){{ [illya@rockylinux9 ~]$ module use /apps/modulefiles/ [illya@rockylinux9 ~]$ module load deepEMhancer [illya@rockylinux9 ~]$ cd test/ [illya@rockylinux9 test]$ deepemhancer -h usage: deepemhancer -i INPUTMAP -o OUTPUTMAP [-p {wideTarget,tightTarget,highRes}] [-i2 HALFMAP2] [-s SAMPLINGRATE] [--noiseStats NOISE_MEAN NOISE_STD] [-m BINARYMASK] [--deepLearningModelPath PATH_TO_MODELS_DIR] [--cleaningStrengh CLEANINGSTRENGH] [-g GPUIDS] [-b BATCH_SIZE] [--version] [-h] [--download [DOWNLOAD_DEST]] DeepEMHancer. Deep post-processing of cryo-EM maps. https://github.com/rsanchezgarc/deepEMhancer optional arguments: -h, --help show this help message and exit （略 [illya@rockylinux9 test]$ }} これが正しいか不明ですが、relionのデータを使って流してみた. #code(nonumber){{ [illya@rockylinux9 test]$ mkdir deepEMhancer [illya@rockylinux9 test]$ cd deepEMhancer/ [illya@rockylinux9 deepEMhancer]$ deepemhancer -i ../relion40_tutorial_precalculated_results/Refine3D/job029/run_it000_half1_class001.mrc -i2 \ ../relion40_tutorial_precalculated_results/Refine3D/job029/run_it000_half2_class001.mrc \ -p tightTarget -o model.mrc --deepLearningModelPath /apps/deepEMhancerModels/production_checkpoints/ -g 0 updating environment to select gpu: [0] loading model /apps/deepEMhancerModels/production_checkpoints/deepEMhancer_tightTarget.hd5 ... DONE! Automatic radial noise detected beyond 36 % of volume side DONE!. Shape at 1.00 A/voxel after padding-> (320, 320, 320) : [illya@rockylinux9 deepEMhancer]$ }} #code(nonumber){{ deepEMhancerCS.sh #!/bin/bash d=/apps/pyenv/versions/miniforge3-24.9.2-0/envs/deepEMhancer export PATH=/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/bin:$PATH export LD_LIBRARY_PATH=/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib:/apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/lib/python3.9/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH /apps/pyenv/versions/anaconda3-2024.02-1/envs/deepEMhancer/bin/deepemhancer $@ export PATH=$d/bin:$PATH export LD_LIBRARY_PATH=$d/lib:$d/lib/python3.9/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH $d/bin/deepemhancer $@ unset d }}
