![[PukiWiki]](image/picc.png "[PukiWiki]")
#author("2024-12-08T07:59:54+00:00;2024-11-23T10:34:10+00:00","default:sysosa","sysosa") #author("2024-12-21T17:52:48+00:00","default:sysosa","sysosa") 本家様 [[https://github.com/google-deepmind/alphafold3>+https://github.com/google-deepmind/alphafold3]] 12月にリリースかなと思ったのですが11月上旬にリリース. ここでは rockylinux9での構築をしてみます. AlphaFold2と同じようにdockerコンテナで実行できるようにするのがオリジナルみたい. まずは下記記載のドキュメント通りに作ってみます [[https://github.com/google-deepmind/alphafold3/blob/main/docs/installation.md>+https://github.com/google-deepmind/alphafold3/blob/main/docs/installation.md]] Toolkit for alphafold3 input and output files [[https://github.com/cddlab/alphafold3_tools>+https://github.com/cddlab/alphafold3_tools]] ***対象マシン [#jf44ec1a] インストール対象マシンは下記の感じです #code(nonumber){{ [root@rockylinux9 ~]# cat /etc/redhat-release Rocky Linux release 9.4 (Blue Onyx) [root@rockylinux9 ~]# getenforce Enforcing [root@rockylinux9 ~]# cat /proc/driver/nvidia/version NVRM version: NVIDIA UNIX x86_64 Kernel Module 550.127.05 Tue Oct 8 03:22:07 UTC 2024 GCC version: gcc version 11.4.1 20231218 (Red Hat 11.4.1-3) (GCC) [root@rockylinux9 ~]# nvidia-smi -L GPU 0: NVIDIA GeForce GTX 1070 (UUID: GPU-a49de51b-de1e-52f3-1e3f-ce704e159713) [root@rockylinux9 ~]# ls -l /usr/local/cuda ls: cannot access '/usr/local/cuda': No such file or directory [root@rockylinux9 ~]# }} ***docker構築 [#eb777c91] #code(nonumber){{ [root@rockylinux9 ~]# dnf -y install dnf-plugins-core [root@rockylinux9 ~]# dnf config-manager --add-repo https://download.docker.com/linux/rhel/docker-ce.repo Adding repo from: https://download.docker.com/linux/rhel/docker-ce.repo [root@rockylinux9 ~]# ls /etc/yum.repos.d/ docker-ce.repo rocky-addons.repo rocky-devel.repo rocky-extras.repo rocky.repo <--「docker-ce.repo」が追加される [root@rockylinux9 ~]# [root@rockylinux9 ~]# sed -i s'/enabled=1/enabled=0/' /etc/yum.repos.d/docker-ce.repo }} 使用するリポジトリは「docker-ce-stable」となる. 詳細は「dnf repolist -v docker-ce-stable」. 中身は「dnf list available --disablerepo=* --enablerepo=docker-ce-stable」で見れる. #code(nonumber){{ [root@rockylinux9 ~]# dnf --enablerepo=docker-ce-stable install docker-ce docker-ce, docker-ce-cli, docker-buildx-plugin, docker-ce-rootless-extras, docker-compose-plugin, containerd.io がインストールされる っで起動 [root@rockylinux9 ~]# systemctl enable docker --now }} 「systemctl status docker」の中身を見ると「/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock」で実行なので「/var/lib/docker」にイメージが保存されます. 別の場所に変えたいなら「--data-root」を使って変更します. 「/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --data-root /docker-images 」とかで. 次にNVIDIA Container Toolkitをインストールします #code(nonumber){{ [root@rockylinux9 ~]# curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | tee /etc/yum.repos.d/nvidia-container-toolkit.repo [root@rockylinux9 ~]# sed -i s'/enabled=1/enabled=0/' /etc/yum.repos.d/nvidia-container-toolkit.repo }} 追加したリポジトリ「nvidia-container-toolkit」の詳細は「dnf repolist -v nvidia-container-toolkit」. 中身は「dnf list available --disablerepo=* --enablerepo=nvidia-container-toolkit」で見れる っでインストールします #code(nonumber){{ [root@rockylinux9 ~]# dnf --enablerepo=nvidia-container-toolkit install nvidia-container-toolkit nvidia-container-toolkit, libnvidia-container-tools, libnvidia-container1, nvidia-container-toolkit-base がインストールされる dockerを再起動させて有効にします [root@rockylinux9 ~]# systemctl restart docker }} っでテスト #code(nonumber){{ [root@rockylinux9 ~]# nvidia-container-cli info NVRM version: 550.127.05 CUDA version: 12.4 Device Index: 0 Device Minor: 0 Model: NVIDIA GeForce GTX 1070 Brand: GeForce GPU UUID: GPU-a49de51b-de1e-52f3-1e3f-ce704e159713 Bus Location: 00000000:06:10.0 Architecture: 6.1 [root@rockylinux9 ~]# [root@rockylinux9 ~]# docker run --gpus all --rm nvidia/cuda:11.8.0-base-ubuntu22.04 nvidia-smi -L : GPU 0: NVIDIA GeForce GTX 1070 (UUID: GPU-a49de51b-de1e-52f3-1e3f-ce704e159713) [root@rockylinux9 ~]# }} これでdockerの構築が完了. そのままでは一般ユーザがdockerを使えない. なので「dockerd-rootless-setuptool.sh --skip-iptables install」でrootlessを有効にさせる. #code(nonumber){{ [saber@rockylinux9 ~]$ dockerd-rootless-setuptool.sh --skip-iptables install : [INFO] Installed docker.service successfully. [INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service` [INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger saber` : [INFO] Some applications may require the following environment variable too: export DOCKER_HOST=unix:///run/user/1000/docker.sock : [saber@rockylinux9 ~]$ ps -ef |grep docker (この段階でdockerが動いてます. ログアウトするとdockerは止まります. 再度ログインするとdockerは自動的に動きます) }} その後にrootで「/etc/nvidia-container-runtime/config.toml」にある「#no-cgroups = false」を「no-cgroups = true」に変更する. これでgpuが一般ユーザでもrootlessで使える. &size(10){systemのbootに合わせて、ログインの有無にかかわらずユーザのdockerを動かすには「sudo loginctl enable-linger saber」を有効にすればいいみたい}; っで本題へ. ***AlphaFold3構築 [#ldb5f755] せっかくdockerをrootlessで作ったので一般ユーザでalphafold3を作ってみます. #code(nonumber){{ [saber@rockylinux9 ~]$ git clone https://github.com/google-deepmind/alphafold3.git [saber@rockylinux9 ~]$ cd alphafold3/ [saber@rockylinux9 alphafold3]$ ls -CF CMakeLists.txt docs/ OUTPUT_TERMS_OF_USE.md requirements.txt src/ dev-requirements.txt fetch_databases.py pyproject.toml run_alphafold.py WEIGHTS_PROHIBITED_USE_POLICY.md docker/ LICENSE README.md run_alphafold_test.py WEIGHTS_TERMS_OF_USE.md [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker build -t alphafold3 --network host -f docker/Dockerfile . <--「apt update」が失敗するので「--network host」を追加してます : : [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alphafold3 latest 59219a3b2f9b 2 minutes ago 9.03GB [saber@rockylinux9 alphafold3]$ }} docker imageの作成に成功しました. ***データベースの準備 [#uffa3404] 以前とは違ってpythonツール「fetch_databases.py」が提供されている. 中身を読むと「storage.googleapis.com/alphafold-databases/v3.0」にあるファイルのダウンロードと解凍ですね. #code(nonumber){{ [saber@rockylinux9 alphafold3]$ python ./fetch_databases.py --help usage: fetch_databases.py [-h] [--download_destination DOWNLOAD_DESTINATION] Downloads AlphaFold databases. optional arguments: -h, --help show this help message and exit --download_destination DOWNLOAD_DESTINATION The directory to download the databases to. [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ python ./fetch_databases.py --download_destination /Public/alphafold3 : (ティータイム) : All databases have been downloaded and decompressed. [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ ls -l /Public/alphafold3 total 656822124 -rw-r--r--. 1 saber saber 18171626364 Nov 17 08:32 bfd-first_non_consensus_sequences.fasta -rw-r--r--. 1 saber saber 128579703018 Nov 17 10:08 mgy_clusters_2022_05.fa -rw-r--r--. 1 saber saber 80977012680 Nov 17 08:57 nt_rna_2023_02_23_clust_seq_id_90_cov_80_rep_seq.fasta -rw-r--r--. 1 saber saber 250266275840 Nov 17 10:05 pdb_2022_09_28_mmcif_files.tar -rw-r--r--. 1 saber saber 232899463 Nov 17 08:12 pdb_seqres_2022_09_28.fasta -rw-r--r--. 1 saber saber 228433680 Nov 17 08:12 rfam_14_9_clust_seq_id_90_cov_80_rep_seq.fasta -rw-r--r--. 1 saber saber 13860314914 Nov 17 08:20 rnacentral_active_seq_id_90_cov_80_linclust.fasta -rw-r--r--. 1 saber saber 108447942931 Nov 17 09:54 uniprot_all_2021_04.fa -rw-r--r--. 1 saber saber 71821260491 Nov 17 09:19 uniref90_2022_05.fa [saber@rockylinux9 alphafold3]$ }} ***モデルパラメータの取得 [#daac8f1d] 下記申請フォームに記載して取得します [[https://forms.gle/svvpY4u2jsHEwWYS6>+https://forms.gle/svvpY4u2jsHEwWYS6]] Google DeepMind 側の裁量があるようです. &size(10){申請しても貰えない可能性がある}; 「$HOME/af3-models」にでも配置しておきます ***実行 [#c4b7f1ba] &color(red){未実施}; #code(nonumber){{ mkdir $HOME/af_input $HOME/af_output docker run -it \ --volume $HOME/af_input:/root/af_input \ --volume $HOME/af_output:/root/af_output \ --volume $HOME/af3-models:/root/models \ --volume /Public/alphafold3:/root/public_databases \ --gpus all \ alphafold3 \ python run_alphafold.py \ --json_path=/root/af_input/fold_input.json \ --model_dir=/root/models \ --output_dir=/root/af_output }} ***Singularityへ [#oa96eb9e] 作ったdokcerイメージを Singularity image file に変換してdocker不要で実行できるようにしてみます singularityのインストール #code(nonumber){{ [root@rockylinux9 ~]# dnf install epel-release -y [root@rockylinux9 ~]# dnf install singularity-ce -y }} #code(nonumber){{ [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB [saber@rockylinux9 alphafold3]$ docker run -d -p 5000:5000 --restart=always --name registry registry:2 Unable to find image 'registry:2' locally 2: Pulling from library/registry dc0decf4841d: Pull complete 6cb0aa443e23: Pull complete 813676e291ef: Pull complete dc2fb7dcec61: Pull complete 916205650bfe: Pull complete Digest: sha256:543dade69668e02e5768d7ea2b0aa4fae6aa7384c9a5a8dbecc2be5136079ddb Status: Downloaded newer image for registry:2 3c355497cde6f5959b956e03533aa2d84fc5e760c674318c6ced5129fb2ab49d [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rootlessk 16914 saber 13u IPv4 70840 0t0 TCP *:commplex-main (LISTEN) rootlessk 16914 saber 16u IPv6 69250 0t0 TCP *:commplex-main (LISTEN) [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c355497cde6 registry:2 "/entrypoint.sh /etc…" 47 seconds ago Up 47 seconds 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp registry [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB registry 2 c18a86d35e98 13 months ago 25.4MB [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker tag alphafold3 localhost:5000/alphafold3 [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c355497cde6 registry:2 "/entrypoint.sh /etc…" 3 minutes ago Up 3 minutes 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp registry [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB localhost:5000/alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB registry 2 c18a86d35e98 13 months ago 25.4MB [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker push localhost:5000/alphafold3 Using default tag: latest The push refers to repository [localhost:5000/alphafold3] d96d8841e4b4: Pushed 9cbe0a162b7b: Pushed ef0973ada226: Pushed 5f70bf18a086: Pushed ae59d8945f10: Pushed b257f8f438cc: Pushed d0eca21f61dc: Pushed 3f00387e422f: Pushed 437449bb2387: Pushed 3b6bc3c2c74b: Pushed cc51bf61b66d: Pushed 2591292aa496: Pushed 1e9c40c384ef: Pushed 931b7ff0cb6f: Pushed latest: digest: sha256:9d3eb15f3e87f755d1ea9bdbdbae0b009f0e3b1487d739043aba6b895de595f9 size: 3274 [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c355497cde6 registry:2 "/entrypoint.sh /etc…" 9 minutes ago Up 9 minutes 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp registry [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE localhost:5000/alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB alphafold3 latest 59219a3b2f9b 13 hours ago 9.03GB registry 2 c18a86d35e98 13 months ago 25.4MB [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ lsof -i COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME rootlessk 16914 saber 13u IPv4 70840 0t0 TCP *:commplex-main (LISTEN) rootlessk 16914 saber 16u IPv6 69250 0t0 TCP *:commplex-main (LISTEN) [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ SINGULARITY_NOHTTPS=1 singularity build alphafold3.sif docker://localhost:5000/alphafold3:latest INFO: Starting build... INFO: Fetching OCI image... 82.1MiB / 82.1MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 7.1MiB / 7.1MiB [==================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 12.3MiB / 12.3MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 54.6MiB / 54.6MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 119.3MiB / 119.3MiB [==============================================================================================================================================================================] 100 % 161.3 MiB/s 0s 233.2MiB / 233.2MiB [==============================================================================================================================================================================] 100 % 161.3 MiB/s 0s 4.4MiB / 4.4MiB [==================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 11.2MiB / 11.2MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 29.0MiB / 29.0MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 99.4MiB / 99.4MiB [================================================================================================================================================================================] 100 % 161.3 MiB/s 0s 4.3GiB / 4.3GiB [==================================================================================================================================================================================] 100 % 161.3 MiB/s 0s INFO: Extracting OCI image... 2024/11/17 21:25:02 warn rootless{usr/lib/x86_64-linux-gnu/gstreamer1.0/gstreamer-1.0/gst-ptp-helper} ignoring (usually) harmless EPERM on setxattr "security.capability" INFO: Inserting Singularity configuration... INFO: Creating SIF file... INFO: Build complete: alphafold3.sif [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ ls -lh alphafold3.sif -rwxr-xr-x. 1 saber saber 4.9G Nov 17 21:28 alphafold3.sif [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE alphafold3 latest 59219a3b2f9b 14 hours ago 9.03GB localhost:5000/alphafold3 latest 59219a3b2f9b 14 hours ago 9.03GB registry 2 c18a86d35e98 13 months ago 25.4MB [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c355497cde6 registry:2 "/entrypoint.sh /etc…" 22 minutes ago Up 22 minutes 0.0.0.0:5000->5000/tcp, :::5000->5000/tcp registry [saber@rockylinux9 alphafold3]$ }} 後始末 #code(nonumber){{ [saber@rockylinux9 alphafold3]$ docker stop registry [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3c355497cde6 registry:2 "/entrypoint.sh /etc…" 23 minutes ago Exited (2) 3 seconds ago registry [saber@rockylinux9 alphafold3]$ docker rm 3c355497cde6 3c355497cde6 [saber@rockylinux9 alphafold3]$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [saber@rockylinux9 alphafold3]$ [saber@rockylinux9 alphafold3]$ docker rmi localhost:5000/alphafold3 [saber@rockylinux9 alphafold3]$ docker rmi c18a86d35e98 [saber@rockylinux9 alphafold3]$ docker rmi 59219a3b2f9b [saber@rockylinux9 alphafold3]$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE [saber@rockylinux9 alphafold3]$ }} ***job [#f5333338] 配列構造データベースが「/Public/alphafold3」にあって 取得したmodelファイルが「/Public/af3-model」にあるとして slurmな環境なら今いるディレクトリに「af_input」と「af_output」を掘って 「af_input/fold_input.json」のインプットファイルを作って 下記batchを動かす #code(nonumber){{ #!/bin/bash #SBATCH -J af3-test #SBATCH -o %j.out #SBATCH -e %j.err #SBATCH -p workq #SBATCH -n 8 #SBATCH --gres=gpu:1 cd $SLURM_SUBMIT_DIR MODEL_DIR=/Public/af3-model DB_DIR=/Public/alphafold3 singularity exec \ --nv \ --bind $SLURM_SUBMIT_DIR/af_input:/root/af_input \ --bind $SLURM_SUBMIT_DIR/af_output:/root/af_output \ --bind $MODEL_DIR:/root/models \ --bind $DB_DIR:/root/public_databases \ /apps/alphafold3.sif \ python /app/alphafold/run_alphafold.py \ --json_path=/root/af_input/fold_input.json \ --model_dir=/root/models \ --db_dir=/root/public_databases \ --output_dir=/root/af_output }}
