大佬的文章

https://blog.k8s.li/kubespray-tips.html
https://fuckcloudnative.io/posts/docker-registry-proxy/

·

docker registry 可以通过设置 remoteurl 参数将其作为远端仓库的缓存仓库，这样当你通过这个私有仓库的地址拉取镜像时，regiistry 会先将镜像缓存到本地存储，然后再提供给拉取的客户端。

我们可以选择通过 制作镜像 、环境变量的方式进行配置，详细的配置参数可参考官方文档。

·

1 前期准备

我在准备阶段白扔了几两银子，还多花了一些时间；这里介绍经过爬坑之后，个人感觉最划算的准备方式，至于我的爬坑经历就不赘述了。

• 一个个人域名：现在各大公有云厂商基本都可以买到域名，而且有很多便宜的域名。我是在新网买的(130元5年)，新网的优点是各种认证、备案通过的比较快；缺点是不提供免费的 SSL 证书。这样的话，就需要在别的地方(比如，阿里云、腾讯云)创建免费证书，然后在新网添加必要的解析。如果觉得麻烦的，可以直接在 “会魔法的服务器” 所在的公有云厂商(比如,阿里云、腾讯云)注册域名。
• 一个会魔法的服务器：我选择的是腾讯云的轻量应用服务器(中国香港区)，虽然资源配置较低，并且流量有限，但是足够个人或中小型企业使用了。具体配置信息见下图：
  

PS： 我在阿里云和新网都注册了一个域名，阿里云的好几天认证都没通过；新网的几个小时就可以使用了。

PS： 我的 SSL 证书使用的是阿里云的免费证书，当然腾讯云也有；免费证书只支持单域名；所以，每个三级域名都需要申请证书。

·

2 制作通用镜像

为了能够支持缓存 docker.io、gcr.io、k8s.gcr.io、quay.io 和 ghcr.io 等常见的公共镜像仓库，我们需要对 registry 的配置文件进行定制。Dockerfile 如下：

FROM registry:2.6
LABEL maintainer="registry-proxy Docker Maintainers https://fuckcloudnative.io"
ENV PROXY_REMOTE_URL="" \DELETE_ENABLED=""
COPY entrypoint.sh /entrypoint.sh

其中，entrypoint.sh 用来将环境变量传入配置文件：

#!/bin/shset -eCONFIG_YML=/etc/docker/registry/config.ymlif [ -n "$PROXY_REMOTE_URL" -a `grep -c "$PROXY_REMOTE_URL" $CONFIG_YML` -eq 0 ]; thenecho "proxy:" >> $CONFIG_YMLecho "  remoteurl: $PROXY_REMOTE_URL" >> $CONFIG_YMLecho "  username: $PROXY_USERNAME" >> $CONFIG_YMLecho "  password: $PROXY_PASSWORD" >> $CONFIG_YMLecho "------ Enabled proxy to remote: $PROXY_REMOTE_URL ------"
elif [ $DELETE_ENABLED = true -a `grep -c "delete:" $CONFIG_YML` -eq 0 ]; thensed -i '/rootdirectory/a\  delete:' $CONFIG_YMLsed -i '/delete/a\    enabled: true' $CONFIG_YMLecho "------ Enabled local storage delete -----"
fised -i "/headers/a\    Access-Control-Allow-Origin: ['*']" $CONFIG_YML
sed -i "/headers/a\    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']" $CONFIG_YML
sed -i "/headers/a\    Access-Control-Expose-Headers: ['Docker-Content-Digest']" $CONFIG_YMLcase "$1" in*.yaml|*.yml) set -- registry serve "$@" ;;serve|garbage-collect|help|-*) set -- registry "$@" ;;
esacexec "$@"

·

3 运行 registry-proxy，并为其添加认证功能

为了防止他人使用，可以为 registry 添加认证功能。建议通过环境变量的方式进行配置；因为在用户密码变化的时候，不用重新构建镜像。

3.1 首先，生成用户密码文件：

$ mkdir -pv /opt/auth
$ htpasswd -Bbn admin ****** > /opt/auth/htpasswd 
$ htpasswd -Bbn  panbuhei ****** >> /opt/auth/htpasswd

3.2 然后，部署 registry-proxy。我这里选择使用 docker-compose 来部署，代码如下：

$ mkdir -pv /opt/docker-compose/registry-proxy/
$ cat << "EOF" > /opt/docker-compose/registry-proxy/docker-compose.yml
version: '3'
services:k8s-gcr-registry:image: wupanfeng035/registry-proxy:v1.0container_name: k8s-gcr-registryrestart: alwaysvolumes:- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registry- /opt/auth/htpasswd:/opt/auth/htpasswdports:- 127.0.0.1:5001:5000environment:- PROXY_REMOTE_URL=https://k8s.gcr.io- REGISTRY_AUTH_HTPASSWD_REALM=basic-auth- REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswdgcr-registry:image: wupanfeng035/registry-proxy:v1.0container_name: gcr-registryrestart: alwaysvolumes:- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registry- /opt/auth/htpasswd:/opt/auth/htpasswdports:- 127.0.0.1:5002:5000environment:- PROXY_REMOTE_URL=https://gcr.io- REGISTRY_AUTH_HTPASSWD_REALM=basic-auth- REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswdhub-registry:image: wupanfeng035/registry-proxy:v1.0container_name: hub-registryrestart: always- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registry- /opt/auth/htpasswd:/opt/auth/htpasswdports:- 127.0.0.1:5003:5000environment:- PROXY_REMOTE_URL=https://registry-1.docker.io### 需要下载 dockerhub 的私有仓库时，请配置用户密码#- PROXY_USERNAME=test001#- PROXY_PASSWORD=********- REGISTRY_AUTH_HTPASSWD_REALM=basic-auth- REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswdquay-registry:image: wupanfeng035/registry-proxy:v1.0container_name: quay-registryrestart: alwaysvolumes:- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registry- /opt/auth/htpasswd:/opt/auth/htpasswdports:- 127.0.0.1:5004:5000environment:- PROXY_REMOTE_URL=https://quay.io- REGISTRY_AUTH_HTPASSWD_REALM=basic-auth- REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswdghcr-registry:image: wupanfeng035/registry-proxy:v1.0container_name: ghcr-registryrestart: alwaysvolumes:- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registry- /opt/auth/htpasswd:/opt/auth/htpasswdports:- 127.0.0.1:5005:5000environment:- PROXY_REMOTE_URL=https://ghcr.io- REGISTRY_AUTH_HTPASSWD_REALM=basic-auth- REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd
EOF

3.3 部署 registry-proxy

$ cd /opt/docker-compose/registry-proxy/
$ docker-compose up -d 

·

4 发布 registry-proxy

由于需要缓存多个公共仓库，并且都需通过 443 端口发布；但是 443 端口只有一个。所以，需要根据域名来转发请求到不同的 registry-proxy 服务。我这里选择使用简单且熟悉的 nginx 实现，配置文件如下所示：

server {listen       80;listen       443 ssl;server_name  k8s-gcr.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/k8s-gcr.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/k8s-gcr.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:5001;}
}server {listen       80;listen       443 ssl;server_name  gcr.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/gcr.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/gcr.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:5002;}
}server {listen       80;listen       443 ssl;server_name  hub.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/hub.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/hub.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:5003;}
}
server {listen       80;listen       443 ssl;server_name  quay.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/quay.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/quay.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:5004;}
}server {listen       80;listen       443 ssl;server_name  ghcr.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/ghcr.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/ghcr.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:5005;}
}

·

5 验证

root@ubuntu20:~# docker pull k8s-gcr.panbuhei.online/kube-controller-manager:v1.23.5
Error response from daemon: Head "https://k8s-gcr.panbuhei.online/v2/kube-controller-manager/manifests/v1.23.5": no basic auth credentials### 登陆
root@ubuntu20:~# docker login k8s-gcr.panbuhei.online
Username: panbuhei
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-storeLogin Succeeded### 下载
root@ubuntu20:~# docker pull k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5
v1.23.5: Pulling from kube-apiserver
2df365faf0e3: Already exists 
8c99db1114c6: Already exists 
b6a9a43f03b3: Pull complete 
Digest: sha256:ddf5bf7196eb534271f9e5d403f4da19838d5610bb5ca191001bde5f32b5492e
Status: Downloaded newer image for k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5
k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5

·

6 清理 registry-proxy 缓存

由于资源有限，所以，需要定期删除缓存到本地磁盘的部分镜像。方法也比较简单，单独再部署一个 registry，共用其他 registry-proxy 的存储，并启用 delete 功能，然后再通过 API 或者 WebUI 进行删除。这里介绍两个 Docker Registry WebUI 工具：

• docker-registry-web：由 JAVA 编写，镜像比较大，并且耗费内存资源。
• docker-registry-ui：底层通过轻量级的 nginx 发布。

为什么 docker-registry-web 比较耗费资源，还选择使用它呢？ 主要原因是：我想通过 nginx 代理它，这样就可以在系统的 nginx 上对其做一些限制操作，比如配置 TLS、限制访问 IP 等。经过测试发现 docker-registry-ui 只能通过 “ip:port” 的方式访问，不能再通过系统 nginx 代理。所以，最后选择了 docker-registry-web。

·

5.1 docker-registry-web 的 docker-compose 代码示例：

$ mkdir -pv /opt/docker-compose/clean-registry/
$ cat << "EOF" > /opt/docker-compose/clean-registry/docker-compose.yml
version: '3.2'
services:registry-local:image: registry:latestcontainer_name: registry-localrestart: alwaysvolumes:- /etc/localtime:/etc/localtime- /var/lib/registry:/var/lib/registryports:- 127.0.0.1:5000:5000environment:- REGISTRY_DELETE_ENABLED=trueregistry-web:image: hyper/docker-registry-webcontainer_name: registry-weblinks: - registry-localrestart: alwaysvolumes:- /etc/localtime:/etc/localtimeports:- 127.0.0.1:8080:8080deploy:resources:limits:cpus: '1'memory: 1Greservations:memory: 512Menvironment:- JAVA_OPTS=-Xmx1024m -Xms512m -Xss256k- REGISTRY_URL=http://registry-local:5000/v2- REGISTRY_NAME=Panbuhei Registry-proxy- REGISTRY_READONLY=false
EOF

5.2 部署

$ cd /opt/docker-compose/clean-registry/### 由于做了资源限制, 并且没有使用 swarm，所以要加上 --compatibility 参数
$ docker-compose --compatibility up -d

5.3 nginx 发布代码：

server {listen       80;server_name  clean.panbuhei.online;location / {return 301 https://$host$request_uri;}}server {listen       443 ssl;server_name  clean.panbuhei.online;ssl_certificate /usr/local/nginx/conf/cert/clean.panbuhei.online.pem;ssl_certificate_key /usr/local/nginx/conf/cert/clean.panbuhei.online.key;ssl_prefer_server_ciphers on;ssl_protocols TLSv1.2 TLSv1.3;ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";ssl_session_cache shared:SSL:10m;ssl_session_timeout 30;if ($request_method !~* GET|HEAD) {return 403;}location / {proxy_pass   http://localhost:8080;}
}

5.4 访问验证

·

这里可能会有疑问：为什么，当我删除图像的所有标签时，图像仍然在 UI 中？因为这是 docker registry 的限制，垃圾收集器(garbage-collect)不会删除空 images。如果要删除空 images，则需要删除 registry 中的文件夹。(见 garbage-collect)

下是为一个删除空 images 的脚本代码：

#!/bin/sh
# remove_nullImageDir.sh
REGISTRY="127.0.0.1:5000"
REGISTRY_NAME=registry-local
repositories=$(curl -s http://${REGISTRY}/v2/_catalog | grep -o '"[^"]*"' | tr -d '"')# docker exec $REGISTRY_NAME registry garbage-collect /etc/docker/registry/config.ymlfor i in $repositories; do[ "$i" = "repositories" ] && continuecurl -s http://${REGISTRY}/v2/${i}/tags/list | egrep '"tags":null|NAME_UNKNOWN';if [ $? -eq 0 ]; thendocker exec -it $REGISTRY_NAME rm -rf /var/lib/registry/docker/registry/v2/repositories/$iecho "delete empty repository $i"fi
done