本家様 https://github.com/rsanchezgarc/deepEMhancer
gitには
Simply speaking, DeepEMhancer performs a non-linear post-processing of cryo-EM maps that produces two main effects:
1. Local sharpening-like post-processing.
2. Automatic masking/denoising of cryo-EM maps.
とある.
pythonアプリなので、ここではcrYOLOのようにpyenv/anacondaの環境にDeepEMhancer 実行環境を用意していきます
pyenv/anaconda環境がないなら下記のように敷設して
git clone https://github.com/yyuu/pyenv.git /apps/pyenv
export PYENV_ROOT=/apps/pyenv
export PATH=$PYENV_ROOT/bin:$PATH
pyenv install anaconda3-2023.07-2
pyenv global anaconda3-2023.07-2
export PATH=$PYENV_ROOT/versions/anaconda3-2023.07-2/bin/:$PATH
もし既にpyenv/anaconda環境があるなら環境変数を整備して
export PYENV_ROOT=/apps/pyenv
export PATH=$PYENV_ROOT/bin:$PATH
export PATH=$PYENV_ROOT/versions/anaconda3-2023.07-2/bin/:$PATH
っで、DeepEMhancer を入れていきます.
[root@rockylinux ~]# cd /apps/
[root@rockylinux apps]# git clone https://github.com/rsanchezgarc/deepEMhancer
[root@rockylinux apps]# cd deepEMhancer
[root@rockylinux deepEMhancer]# ls
alternative_installation condaDeepEMHancer conda_ymls deepEMhancer deepEMhancer_env.yml deepEMhancer_tutorial_06_23.pdf LICENSE README.md setup.py
[root@rockylinux deepEMhancer]# conda env create -f deepEMhancer_env.yml -n deepEMhancer_env
Collecting package metadata (repodata.json): done
Solving environment: \
:
:
done
#
# To activate this environment, use
#
# $ conda activate deepEMhancer_env
#
# To deactivate an active environment, use
#
# $ conda deactivate
[root@rockylinux deepEMhancer]#
と環境構築が完了します. 引き続きその作った環境に移行して、deepEMhancerのプログラムをインストールします.
[root@rockylinux deepEMhancer]# source activate deepEMhancer_env
(deepEMhancer_env) [root@rockylinux deepEMhancer]# conda list
:
cudatoolkit 11.8.0 h4ba93d1_12 conda-forge
cudnn 8.8.0.121 h0800d71_1 conda-forge
:
tensorboard 2.12.3 pypi_0 pypi
tensorboard-data-server 0.7.1 pypi_0 pypi
tensorflow 2.12.0 pypi_0 pypi
tensorflow-estimator 2.12.0 pypi_0 pypi
tensorflow-io-gcs-filesystem 0.32.0 pypi_0 pypi
:
(deepEMhancer_env) [root@rockylinux deepEMhancer]# pwd
/apps/deepEMhancer
(deepEMhancer_env) [root@rockylinux deepEMhancer]# which python
/apps/pyenv/versions/anaconda3-2023.07-2/envs/deepEMhancer_env/bin/python
(deepEMhancer_env) [root@rockylinux deepEMhancer]#
(deepEMhancer_env) [root@rockylinux deepEMhancer]# python -m pip install . --no-deps
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Processing /apps/deepEMhancer
Preparing metadata (setup.py) ... done
Building wheels for collected packages: deepEMhancer
Building wheel for deepEMhancer (setup.py) ... done
Created wheel for deepEMhancer: filename=deepEMhancer-0.16-py3-none-any.whl size=35142 sha256=c586c01e29a4cfbc3c461f4b30c06c78afc66dd653a98b185030f17bedcf8201
Stored in directory: /tmp/pip-ephem-wheel-cache-wsj3carg/wheels/cf/c6/c8/1857699bf687d7bd754dd9dad454c30896add113827c5ed02a
Successfully built deepEMhancer
Installing collected packages: deepEMhancer
Successfully installed deepEMhancer-0.16
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(deepEMhancer_env) [root@rockylinux deepEMhancer]# conda deactivate
[root@rockylinux deepEMhancer]#
[root@rockylinux ~]# vi /apps/modulefiles/deepEMhancer
#%Module1.0
set root /apps/pyenv/versions/anaconda3-2023.03/envs/deepEMhancer_env/
prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib:$root/lib/python3.9/site-packages/nvidia/cudnn/lib
[root@rockylinux ~]#
LD_LIBRARY_PATH を定義しないと「ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found」と言われるみたい
[saber@rockylinux ~]$ module use /apps/modulefiles/
[saber@rockylinux ~]$ module load deepEMhancer
[saber@rockylinux ~]$ deepemhancer --download
DOWNLAODING MODELs from https://zenodo.org/record/7432763/files/deepEMhancerModels_tf2.zip to /home/saber/.local/share/deepEMhancerModels
It may take a while...
100%|???????????????????????????????????????????????????????????????????????????????????????????????????????????| 180462/180464 [01:50<00:00, 1630.69it/s]
Total size downloaded: 739172352.
Unzipping...
DONE!!
[saber@rockylinux ~]$ ls -lh .local/share/deepEMhancerModels/production_checkpoints/
total 884M
-rw-rw-r--. 1 saber saber 298M May 22 02:22 deepEMhancer_highRes.hd5
-rw-rw-r--. 1 saber saber 196M May 22 02:22 deepEMhancer_masked.hd5
-rw-rw-r--. 1 saber saber 196M May 22 02:22 deepEMhancer_tightTarget.hd5
-rw-rw-r--. 1 saber saber 196M May 22 02:22 deepEMhancer_wideTarget.hd5
-rw-rw-r--. 1 saber saber 74 May 22 02:22 versions.txt
[saber@rockylinux ~]$
[saber@rockylinux ~]$ cat .local/share/deepEMhancerModels/production_checkpoints/versions.txt
highRes: g2_3
tightTarget: v3_L_c_2
wideModel: v3t_28_c+
masked: vt2_m_28
[saber@rockylinux ~]$
一応「deepemhancer --download ./DeepEMhancerData」と特定の場所にダウンロード可能です
っでテストrunです.
[saber@rockylinux ~]$ deepemhancer -h
usage: deepemhancer -i INPUTMAP -o OUTPUTMAP [-p {wideTarget,tightTarget,highRes}] [-i2 HALFMAP2] [-s SAMPLINGRATE] [--noiseStats NOISE_MEAN NOISE_STD]
[-m BINARYMASK] [--deepLearningModelPath PATH_TO_MODELS_DIR] [--cleaningStrengh CLEANINGSTRENGH] [-g GPUIDS] [-b BATCH_SIZE]
[--version] [-h] [--download [DOWNLOAD_DEST]]
DeepEMHancer. Deep post-processing of cryo-EM maps. https://github.com/rsanchezgarc/deepEMhancer
options:
-h, --help show this help message and exit
--download [DOWNLOAD_DEST]
download default DeepEMhancer models. They will be saved at /home/saber/.local/share/deepEMhancerModels/production_checkpoints
if no path provided
Main options:
-i INPUTMAP, --inputMap INPUTMAP
Input map to process or half map number 1. This map should be unmasked and not sharpened (Do not use post-processed maps, only
maps directly obtained from refinement). If half map 1 used, do not forget to also provide the half map 2 using -i2
-o OUTPUTMAP, --outputMap OUTPUTMAP
Output fname where post-processed map will be saved
-p {wideTarget,tightTarget,highRes}, --processingType {wideTarget,tightTarget,highRes}
Select the deep learning model you want to use. WideTarget will produce less sharp results than tightTarget. HighRes is only
recommended for overal FSC resolution < 4 A This option is igonred if normalization mode 2 is selected
-i2 HALFMAP2, --halfMap2 HALFMAP2
(Optional) Input half map 2 to process
-s SAMPLINGRATE, --samplingRate SAMPLINGRATE
(Optional) Sampling rate (A/voxel) of the input map. If not provided, the sampling rate will be read from mrc file header
Normalization options (auto normalization is applied if no option selected):
--noiseStats NOISE_MEAN NOISE_STD
(Optional) Normalization mode 1: The statisitcs of the noise to normalize (mean and standard deviation) the input. Preferred
over binaryMask but ignored if binaryMask provided. If not --noiseStats nor --binaryMask provided, nomralization params will be
automatically estimated, although, in some rare cases, estimation may fail or be less accurate
-m BINARYMASK, --binaryMask BINARYMASK
(Optional) Normalization mode 2: A binaryMask (1 protein, 0 no protein) used to normalize the input. If no normalization mode
provided, automatic normalization will be carried out. Supresses --precomputedModel option
Alternative options:
--deepLearningModelPath PATH_TO_MODELS_DIR
(Optional) Directory where a non default deep learning model is located (model is selected using --precomputedModel) or a path
to hd5 file containing the model
--cleaningStrengh CLEANINGSTRENGH
(Optional) Post-processing step to remove small connected components (hide dust). Max relative size of connected components to
remove 0<s<1 or -1 to deactivate. Default: -1
Computing devices options:
-g GPUIDS, --gpuIds GPUIDS
The gpu(s) where the program will be executed. If more that 1, comma seppared. E.g -g 1,2,3. Set to -1 to use only cpu (very
slow). Default: 0
-b BATCH_SIZE, --batch_size BATCH_SIZE
Number of cubes to process simultaneously. Lower it if CUDA Out Of Memory error happens and increase it if low GPU performance
observed. Warning, for some inputs it may crash if --gpus > . Use only 1 gpus in that case. 1Default: 8
--version show program's version number and exit
examples:
+ Download deep learning models
deepemhancer --download
+ Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using default deep model tightTarget
deepemhancer -i path/to/inputVol.mrc -o path/to/outputVol.mrc
+ Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using high resolution deep model
deepemhancer -p highRes -i path/to/inputVol.mrc -o path/to/outputVol.mrc
+ Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using a deep learning model located in path/to/deep/learningModel
deepemhancer -c path/to/deep/learningModel -i path/to/inputVol.mrc -o path/to/outputVol.mrc
+ Post-process input map path/to/inputVol.mrc and save it at path/to/outputVol.mrc using high resolution deep model and providing normalization information (mean
and standard deviation of the noise)
deepemhancer -p highRes -i path/to/inputVol.mrc -o path/to/outputVol.mrc --noiseStats 0.12 0.03
[saber@rockylinux ~]$
これが正しいか不明ですが、relionのデータを使って流してみた.
[saber@rockylinux test]$ export LD_LIBRARY_PATH=/apps/pyenv/versions/anaconda3-2023.07-2/envs/deepEMhancer_env/lib/python3.9/site-packages/nvidia/cudnn/lib <--- これが有効でないとGPUを使わないみたい
[saber@rockylinux test]$ deepemhancer -i ../relion40_tutorial_precalculated_results/Refine3D/job029/run_it000_half1_class001.mrc -i2 ../relion40_tutorial_precalculated_results/Refine3D/job029/run_it000_half2_class001.mrc \
-p tightTarget -o model.mrc --deepLearningModelPath ~/.local/share/deepEMhancerModels/production_checkpoints/ -g 0,1,2,3
updating environment to select gpu: [0, 1, 2, 3]
loading model /apps/deepEMhancer/deepEMhancerModels/production_checkpoints/deepEMhancer_tightTarget.hd5 ... DONE!
Automatic radial noise detected beyond 36 % of volume side
DONE!. Shape at 1.00 A/voxel after padding-> (320, 320, 320)
Neural net inference
100%|?????????????????????????????????????????????????????????????????????????????????????????????????????????????????| 289/289 [4:58:14<00:00, 61.92s/it]
[saber@rockylinux test]$