かいていちゅう

本家様 https://github.com/YoshitakaMo/localcolabfold

localcolabfoldは、ColabFold様の「ColabFold: AlphaFold2 using MMseqs2」をlocal(自分の計算機)で実現できる代物でございます.
ColabFold様による「ColabFold」のご説明はこちら
https://docs.google.com/presentation/d/1mnffk23ev2QMDzGZ5w1skXEadTe54l8-Uei6ACce8eI/edit#slide=id.p

要件

Note (May 21, 2024)
Since current GPU-supported jax > 0.4.26 requires CUDA 12.1 or later and cudnn 9, please upgrade or install your CUDA driver and cudnn. CUDA 12.4 is recommended.
 
deepL先生訳
現在のGPU対応jax > 0.4.26にはCUDA 12.1以降とcudnn 9が必要なため、CUDAドライバとcudnnをアップグレードまたはインストールしてください。CUDA 12.4を推奨します。

あと「GNU compiler version is 9.0 or later」ともあります.

下準備

なので

[root@rockylinux ~]# cat /etc/redhat-release
Rocky Linux release 8.8 (Green Obsidian)
 
[root@rockylinux ~]#
[root@rockylinux ~]# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  535.161.07  Sat Feb 17 22:55:48 UTC 2024
GCC version:  gcc version 8.5.0 20210514 (Red Hat 8.5.0-18) (GCC)
 
[root@rockylinux ~]#
[root@rockylinux ~]# /usr/local/cuda/bin/nvcc --version
-bash: /usr/local/cuda/bin/nvcc: No such file or directory
 
[root@rockylinux ~]#

「/usr/local/cuda-12.4」とかのcudaライブラリは入れてません.

gccはRockylinux8では8.5.0系が規定なので要件には合わない. っで「gcc-toolset」で上げてみる

[root@rockylinux ~]# dnf install gcc-toolset-13
 :
[root@rockylinux ~]# source scl_source enable gcc-toolset-13
 
[root@rockylinux ~]# gcc --version
gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4)
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
[root@rockylinux ~]#
[root@rockylinux ~]# echo $PATH
/opt/rh/gcc-toolset-13/root/usr/bin:/usr/share/Modules/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
 
[root@rockylinux ~]# echo $LD_LIBRARY_PATH
/opt/rh/gcc-toolset-13/root/usr/lib64:/opt/rh/gcc-toolset-13/root/usr/lib
 
[root@rockylinux ~]#

ってな感じ. 一応これで作ってみる

インストール

githubからソースを取得して展開したい場所の上でインストーラーを実行します

[root@rockylinux ~]# mkdir -p /apps/src && cd /apps/src/
 
[root@rockylinux src]# git clone https://github.com/YoshitakaMo/localcolabfold
[root@rockylinux src]# cd localcolabfold/
 
[root@rockylinux localcolabfold]# git log -1
commit 06e163495b9529e14f82517102fb925d0e859754 (HEAD -> main, origin/main, origin/HEAD)
Author: Yoshitaka Moriwaki <virgospica93@gmail.com>
Date:   Tue May 21 13:26:02 2024 +0900
 
    use CUDA 12.1 or later
[root@rockylinux localcolabfold]#
 
[root@rockylinux localcolabfold]# cd /apps
 
[root@rockylinux apps]# ./src/localcolabfold/install_colabbatch_linux.sh

っでここでGPUが使えるかのテストを行ってみる.

[root@rockylinux ~]# source /apps/localcolabfold/conda/etc/profile.d/conda.sh
 
[root@rockylinux ~]# conda env list
# conda environments:
#
                         /apps/localcolabfold/colabfold-conda
base                     /apps/localcolabfold/conda
 
[root@rockylinux ~]#
[root@rockylinux ~]# conda activate /apps/localcolabfold/colabfold-conda
 
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# conda list
 :
cudatoolkit               11.8.0              h4ba93d1_13    conda-forge
 :
jax                       0.4.23                   pypi_0    pypi
jaxlib                    0.4.23+cuda11.cudnn86          pypi_0    pypi
 :
nvidia-cublas-cu11        11.11.3.6                pypi_0    pypi
nvidia-cuda-cupti-cu11    11.8.87                  pypi_0    pypi
nvidia-cuda-nvcc-cu11     11.8.89                  pypi_0    pypi
nvidia-cuda-runtime-cu11  11.8.89                  pypi_0    pypi
nvidia-cudnn-cu11         9.0.0.312                pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-cusolver-cu11      11.4.1.48                pypi_0    pypi
nvidia-cusparse-cu11      11.7.5.86                pypi_0    pypi
nvidia-nccl-cu11          2.21.5                   pypi_0    pypi
 :
tensorflow                2.16.1                   pypi_0    pypi
tensorflow-cpu            2.16.1                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.36.0                   pypi_0    pypi
 :
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# python
 
>>> from jax.lib import xla_bridge
>>> print(xla_bridge.get_backend().platform)
CUDA backend failed to initialize: Unable to load cuDNN. Is it installed? (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)
cpu
>>>

とGPUを認識してないみたい. 原因はjaxlibが「cuda11.cudnn86」なのに用意されたcudnnが「nvidia-cudnn-cu11 9.0.0.312」と「9.0.0.312」であるためっぽい.
っでこの「nvidia-cudnn-cu11」を cudnn-8.x に差し替えてみる
ここを参照するとhttps://pypi.org/project/nvidia-cudnn-cu11/#history cudnn-8.x にするには
「pip install nvidia-cudnn-cu11==8.9.6.50」
が必要みたい.
これを実施します

(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# pip install nvidia-cudnn-cu11==8.9.6.50
  
(nvidia_cudnn_cu11-8.9.6.50-py3-none-manylinux1_x86_64.whl と nvidia_cuda_nvrtc_cu11-11.8.89-py3-none-manylinux1_x86_64.whl がインストールされます)
 
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# conda list
 :
nvidia-cuda-nvrtc-cu11    11.8.89                  pypi_0    pypi     <-- 新規追加
 :
nvidia-cudnn-cu11         8.9.6.50                 pypi_0    pypi     <-- 9.0.0.312 からダウングレード
 :
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]#

っでテストしてみます

(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# python
>>> from jax.lib import xla_bridge
>>> print(xla_bridge.get_backend().platform)
gpu
>>>

これで大丈夫みたい.
他に「pip install tensorflow[and-cuda]==2.16.1」で調整もあるけど、これはcondaのnvidia関連パッケージが12系列になります
tensorflowでGPUを使うテストを行うと「GPUが見えなよ」となりますが、これでいいみたい

EnvironmentModules

[root@rockylinux ~]# vi /apps/modulefiles/localcolabfold
#%Module1.0
set          root            /apps/localcolabfold/colabfold-conda
prepend-path PATH            $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
 
[root@rockylinux ~]#

テスト

予測対象の配列は別途ファイルで用意するみたい

[saber@rockylinux test]$ module load localcolabfold
 
[saber@rockylinux test]$ colabfold_batch -h
usage: colabfold_batch [-h] [--msa-only] [--msa-mode {mmseqs2_uniref_env,mmseqs2_uniref,single_sequence}]
                       [--pair-mode {unpaired,paired,unpaired_paired}] [--pair-strategy {complete,greedy}] [--templates]
                       [--custom-template-path CUSTOM_TEMPLATE_PATH] [--pdb-hit-file PDB_HIT_FILE] [--local-pdb-path LOCAL_PDB_PATH]
                       [--num-recycle NUM_RECYCLE] [--recycle-early-stop-tolerance RECYCLE_EARLY_STOP_TOLERANCE] [--num-ensemble NUM_ENSEMBLE]
                       [--num-seeds NUM_SEEDS] [--random-seed RANDOM_SEED] [--num-models {1,2,3,4,5}]
                       [--model-type {auto,alphafold2,alphafold2_ptm,alphafold2_multimer_v1,alphafold2_multimer_v2,alphafold2_multimer_v3,deepfold_v1}]
                       [--model-order MODEL_ORDER] [--use-dropout] [--max-seq MAX_SEQ] [--max-extra-seq MAX_EXTRA_SEQ] [--max-msa MAX_MSA]
                       [--disable-cluster-profile] [--data DATA] [--amber] [--num-relax NUM_RELAX] [--relax-max-iterations RELAX_MAX_ITERATIONS]
                       [--relax-tolerance RELAX_TOLERANCE] [--relax-stiffness RELAX_STIFFNESS] [--relax-max-outer-iterations RELAX_MAX_OUTER_ITERATIONS]
                       [--use-gpu-relax] [--rank {auto,plddt,ptm,iptm,multimer}] [--stop-at-score STOP_AT_SCORE] [--jobname-prefix JOBNAME_PREFIX]
                       [--save-all] [--save-recycles] [--save-single-representations] [--save-pair-representations] [--overwrite-existing-results]
                       [--zip] [--sort-queries-by {none,length,random}] [--host-url HOST_URL] [--disable-unified-memory]
                       [--recompile-padding RECOMPILE_PADDING]
                       input results
 :
 :
[illya@rockylinux test]$ vi query.fasta
>sample
PIAQIHILEGRSDEQKETLIREVSEAISRSLDAPLTSVRVIITEMAKGHFGIGGELASK
 
 
[saber@rockylinux test]$ colabfold_batch --amber --templates --num-recycle 3 --use-gpu-relax ./query.fasta ./out
2024-03-13 01:04:06,273 Running colabfold 1.5.5 (2dca1083d6351944a32aff4f66290a961da744b0)
2024-03-13 01:04:06,350 Running on GPU
2024-03-13 01:04:06,675 Found 9 citations for tools or databases
2024-03-13 01:04:06,676 Query 1/1: sample (length 59)
 :
2024-03-13 01:04:11,775 Sequence 0 found templates: ['3mb2_C', '6bgn_C', '2fm7_B', '4fdx_A', '1otf_D', '3ry0_B', '1bjp_A', '7m59_B', (略
2024-03-13 01:04:12,162 Setting max_seq=512, max_extra_seq=5120
2024-03-13 01:05:11,276 alphafold2_ptm_model_1_seed_000 recycle=0 pLDDT=97.9 pTM=0.787
2024-03-13 01:05:12,997 alphafold2_ptm_model_1_seed_000 recycle=1 pLDDT=97.9 pTM=0.792 tol=0.109
2024-03-13 01:05:14,719 alphafold2_ptm_model_1_seed_000 recycle=2 pLDDT=97.9 pTM=0.79 tol=0.0369
2024-03-13 01:05:16,443 alphafold2_ptm_model_1_seed_000 recycle=3 pLDDT=97.8 pTM=0.788 tol=0.0257
2024-03-13 01:05:16,443 alphafold2_ptm_model_1_seed_000 took 58.4s (3 recycles)
2024-03-13 01:05:18,177 alphafold2_ptm_model_2_seed_000 recycle=0 pLDDT=97.9 pTM=0.795
2024-03-13 01:05:19,906 alphafold2_ptm_model_2_seed_000 recycle=1 pLDDT=97.9 pTM=0.802 tol=0.0662
2024-03-13 01:05:21,635 alphafold2_ptm_model_2_seed_000 recycle=2 pLDDT=97.9 pTM=0.801 tol=0.0284
2024-03-13 01:05:23,366 alphafold2_ptm_model_2_seed_000 recycle=3 pLDDT=97.8 pTM=0.8 tol=0.0196
2024-03-13 01:05:23,366 alphafold2_ptm_model_2_seed_000 took 6.9s (3 recycles)
2024-03-13 01:05:42,759 alphafold2_ptm_model_3_seed_000 recycle=0 pLDDT=97.2 pTM=0.774
2024-03-13 01:05:44,453 alphafold2_ptm_model_3_seed_000 recycle=1 pLDDT=97.4 pTM=0.783 tol=0.281
2024-03-13 01:05:46,150 alphafold2_ptm_model_3_seed_000 recycle=2 pLDDT=97.4 pTM=0.783 tol=0.0822
2024-03-13 01:05:47,846 alphafold2_ptm_model_3_seed_000 recycle=3 pLDDT=97.3 pTM=0.783 tol=0.0696
2024-03-13 01:05:47,846 alphafold2_ptm_model_3_seed_000 took 24.5s (3 recycles)
2024-03-13 01:05:49,550 alphafold2_ptm_model_4_seed_000 recycle=0 pLDDT=97.3 pTM=0.772
2024-03-13 01:05:51,251 alphafold2_ptm_model_4_seed_000 recycle=1 pLDDT=97.4 pTM=0.781 tol=0.25
2024-03-13 01:05:52,951 alphafold2_ptm_model_4_seed_000 recycle=2 pLDDT=97.2 pTM=0.78 tol=0.049
2024-03-13 01:05:54,654 alphafold2_ptm_model_4_seed_000 recycle=3 pLDDT=97 pTM=0.778 tol=0.0464
2024-03-13 01:05:54,654 alphafold2_ptm_model_4_seed_000 took 6.8s (3 recycles)
2024-03-13 01:05:56,362 alphafold2_ptm_model_5_seed_000 recycle=0 pLDDT=97.4 pTM=0.783
2024-03-13 01:05:58,068 alphafold2_ptm_model_5_seed_000 recycle=1 pLDDT=97 pTM=0.785 tol=0.246
2024-03-13 01:05:59,774 alphafold2_ptm_model_5_seed_000 recycle=2 pLDDT=96.2 pTM=0.778 tol=0.177
2024-03-13 01:06:01,482 alphafold2_ptm_model_5_seed_000 recycle=3 pLDDT=96 pTM=0.776 tol=0.128
2024-03-13 01:06:01,482 alphafold2_ptm_model_5_seed_000 took 6.8s (3 recycles)
2024-03-13 01:06:01,488 reranking models by 'plddt' metric
2024-03-13 01:06:01,665 Warning: importing 'simtk.openmm' is deprecated.  Import 'openmm' instead.
2024-03-13 01:06:06,745 Relaxation took 5.3s
2024-03-13 01:06:06,745 rank_001_alphafold2_ptm_model_2_seed_000 pLDDT=97.8 pTM=0.8
2024-03-13 01:06:07,631 Relaxation took 0.9s
2024-03-13 01:06:07,631 rank_002_alphafold2_ptm_model_1_seed_000 pLDDT=97.8 pTM=0.788
2024-03-13 01:06:08,744 Relaxation took 1.1s
2024-03-13 01:06:08,744 rank_003_alphafold2_ptm_model_3_seed_000 pLDDT=97.3 pTM=0.783
2024-03-13 01:06:09,871 Relaxation took 1.1s
2024-03-13 01:06:09,871 rank_004_alphafold2_ptm_model_4_seed_000 pLDDT=97 pTM=0.778
2024-03-13 01:06:10,959 Relaxation took 1.1s
2024-03-13 01:06:10,959 rank_005_alphafold2_ptm_model_5_seed_000 pLDDT=96 pTM=0.776
2024-03-13 01:06:11,686 Done
[saber@rockylinux test]$

メモ

nvidiaドライバが 520.56.06(CUDA 11.8) なら

[root@rockylinux ~]# cat /etc/redhat-release
Rocky Linux release 8.8 (Green Obsidian)
 
[root@rockylinux ~]# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  520.56.06  Thu Oct  6 21:38:55 UTC 2022
GCC version:  gcc version 8.5.0 20210514 (Red Hat 8.5.0-18) (GCC)
 
[root@rockylinux ~]# /usr/local/cuda/bin/nvcc --version
-bash: /usr/local/cuda/bin/nvcc: No such file or directory
[root@rockylinux ~]#
[root@rockylinux ~]# dnf install gcc-toolset-13
[root@rockylinux ~]# source scl_source enable gcc-toolset-13
[root@rockylinux ~]# gcc --version
gcc (GCC) 13.1.1 20230614 (Red Hat 13.1.1-4)
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
[root@rockylinux ~]#
[root@rockylinux ~]# mkdir -p /apps/src && cd $_
[root@rockylinux src]# git clone https://github.com/YoshitakaMo/localcolabfold
[root@rockylinux src]# cd localcolabfold/
[root@rockylinux localcolabfold]# git log -1
commit 197d3ef58191361ff7a45228013155914a6081a9 (HEAD -> main, origin/main, origin/HEAD)
Author: Yoshitaka Moriwaki <virgospica93@gmail.com>
Date:   Thu Feb 22 16:46:37 2024 +0900
 
    Do not re-download params after update
[root@rockylinux localcolabfold]# cd /apps
[root@rockylinux apps]# ./src/localcolabfold/install_colabbatch_linux.sh
 
[root@rockylinux ~]# source /apps/localcolabfold/conda/etc/profile.d/conda.sh
[root@rockylinux ~]# conda activate /apps/localcolabfold/colabfold-conda
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# conda list
 :
cudatoolkit               11.8.0              h4ba93d1_13    conda-forge
 :
jax                       0.4.23                   pypi_0    pypi
jaxlib                    0.4.23+cuda11.cudnn86          pypi_0    pypi
 :
nvidia-cublas-cu11        11.11.3.6                pypi_0    pypi
nvidia-cuda-cupti-cu11    11.8.87                  pypi_0    pypi
nvidia-cuda-nvcc-cu11     11.8.89                  pypi_0    pypi
nvidia-cuda-runtime-cu11  11.8.89                  pypi_0    pypi
nvidia-cudnn-cu11         9.0.0.312                pypi_0    pypi
nvidia-cufft-cu11         10.9.0.58                pypi_0    pypi
nvidia-cusolver-cu11      11.4.1.48                pypi_0    pypi
nvidia-cusparse-cu11      11.7.5.86                pypi_0    pypi
nvidia-nccl-cu11          2.21.5                   pypi_0    pypi
 :
tensorflow                2.16.1                   pypi_0    pypi
tensorflow-cpu            2.16.1                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.36.0                   pypi_0    pypi
 :
(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]#

一応これでもダメ.
原因は「nvidia-cudnn-cu11 9.0.0.312」である点. 「jaxlib」で「cuda11.cudnn86」を要求しているのに「cudnn」が9.0.0.312なのでGPUが使えないみたい.
っで回避方法は https://pypi.org/project/nvidia-cudnn-cu11/#history でcudnnの8.xを見つけて入れる.

(/apps/localcolabfold/colabfold-conda) [root@rockylinux ~]# pip install nvidia-cudnn-cu11==8.9.6.50

これで計算できる.

メモ その2

Could not predict dummy_sequence. Not Enough GPU memory? INTERNAL: XLA requires ptxas version 11.8 or higher」と表示されたら.
単純に ptxas コマンドが古いため.
恐らく /usr/local/cuda/bin が PATH環境変数に入っていて問題が起こった可能性がある. 回避策としては /usr/local/cuda/bin のPATHを外すか
cuda-11.8以上のcudaライブラリを入れる.
あるいは conda環境の中にある ptxas をPATHに入れる
具体的にはmodulefilesに「/apps/localcolabfold/colabfold-conda/lib/python3.10/site-packages/nvidia/cuda_nvcc/bin」を入れる

特定のGPUで計算させたい

slurmとかopenPBSでGPUのリソース管理をきちんとできていれば問題ないのですが、そうでない場合、そのまま実行するとその計算機にある全てのGPUを使って計算を行うみたい.
全てのGPUを使っているように見えるが実際には1つのGPUで処理しているみたい
2枚刺しGPUマシンで、cryoSPARCを流して、localcolabfoldを流すとたぶんcryoSPARCが使っているGPUにもそのジョブが回る。せっかく2枚あるならすみ分けてほしい.

その場合は環境変数 CUDA_VISIBLE_DEVICES を使って使用するGPUを指定することが出来るっぽい

CUDA_VISIBLE_DEVICES=1 colabfold_batch --amber --templates --num-recycle 3 --use-gpu-relax ./query.fasta ./out

CUDA_VISIBLE_DEVICESは0から始まるようで、2枚目は「1」となる。この0とか1とかは、nvidia-smiで表示される GPU 番号と同じっぽい。

メモ3

実行時に
Failed to extract font properties from /usr/share/fonts/google-noto-emoji/NotoColorEmoji.ttf: In FT2Font: Can not load face (unknown file format; error code 0x2)
と表記される。実害はないようですが、消すには下記のようにします。

source /apps/localcolabfold/conda/etc/profile.d/conda.sh
conda activate /apps/localcolabfold/colabfold-conda
 
conda list
 :
matplotlib                3.9.0                    pypi_0    pypi
 :
 
(このmatplotlib 3.9.0を 3.8.4 にする)
pip install matplotlib==3.8.4

これで上記文言は表示されなくなる

最新の60件
2024-06-17 2024-06-16
  • localcolabfold
2024-06-14 2024-06-12 2024-06-11 2024-06-10 2024-06-08 2024-06-07 2024-06-02 2024-06-01 2024-05-30 2024-05-16 2024-04-26 2024-04-18 2024-04-15 2024-04-11 2024-04-06 2024-04-05 2024-03-30 2024-03-29 2024-03-19 2024-03-17 2024-03-15 2024-03-06 2024-03-05 2024-03-03 2024-02-23 2024-02-22 2024-02-21 2024-02-20 2024-02-19 2024-02-18 2024-02-13 2024-02-12 2024-02-09 2024-02-04 2024-02-03 2024-01-31 2024-01-30 2024-01-29 2024-01-28 2024-01-22 2024-01-16

edit


トップ   編集 差分 履歴 添付 複製 名前変更 リロード   新規 一覧 検索 最終更新   ヘルプ   最終更新のRSS
Last-modified: 2024-06-16 (日) 23:33:21