docker容器正常运行，但不能exec

[root@VM_0_166_centos ~]# docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                    NAMES
093ce616846c        10.133.193.241:5000/v20190111-cpm-004   "/home/apache-tomc..."   21 months ago       Up 3 months         0.0.0.0:8834->8080/tcp   cpm03
edbffeca5cee        10.133.193.241:5000/v20180809-cpm-001   "/home/apache-tomc..."   2 years ago         Up 8 hours          0.0.0.0:8090->8080/tcp   cpm02
[root@VM_0_166_centos ~]# docker exec cpm02 bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "process_linux.go:75: starting setns process caused \"fork/exec /proc/self/exe: no such file or directory\""

docker 版本信息

[root@VM_0_166_centos ~]# docker version
Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      dded712/1.13.1
 Built:           Tue Jul 17 18:34:48 2018
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-68.gitdded712.el7.centos.x86_64
 Go version:      go1.9.4
 Git commit:      dded712/1.13.1
 Built:           Tue Jul 17 18:34:48 2018
 OS/Arch:         linux/amd64
 Experimental:    false

具体解决过程

1、查找出 docker 的容器 id edbffeca5cee
[root@VM_0_166_centos ~]# docker ps | grep cpm02

2、根据 docker 容器 id edbffeca5cee 找到对应的 libcontainerd 的运行pid 10209
[root@VM_0_166_centos ~]# ps -ef|grep libcontainerd | grep edbffeca5cee
root     10209  4721  0 05:48 ?        00:01:24 /usr/bin/docker-containerd-shim-current edbffeca5cee297d71f153ff2bc459042d71e317eba4996a8204fb44dc2a1225 /var/run/docker/libcontainerd/edbffeca5cee297d71f153ff2bc459042d71e317eba4996a8204fb44dc2a1225 /usr/libexec/docker/docker-runc-current

3、用 nsenter 进入 docker 容器 edbffeca5cee 的 namespace
[root@VM_0_166_centos ~]# nsenter -m -t 10209 bash

4、查看 docker 容器 edbffeca5cee 的 DeviceName
[root@VM_0_166_centos ~]# docker inspect --format='{{.GraphDriver.Data.DeviceName}}' edbffeca5cee
docker-253:1-451037-c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab

5、可以看到 docker 容器 edbffeca5cee的 /var/lib/docker/devicemapper/mnt/docker容器ID 这个目录不存在
[root@VM_0_166_centos ~]# ll /var/lib/docker/devicemapper/mnt/c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab
ls: cannot access /var/lib/docker/devicemapper/mnt/c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab: No such file or directory

6、对比正常 docker 容器 aa4416c1f1e8 的目录
[root@VM_0_166_centos ~]# ll /var/lib/docker/devicemapper/mnt/aa4416c1f1e8fb192e72b2cf60aae8507cc4bf7bbe69ef2b96d81e29640f7a4a
total 8
-rw-------  1 root root   64 Dec 11  2017 id
drwxr-xr-x 21 root root 4096 Apr  3 15:35 rootfs

7、创建 docker 容器 edbffeca5cee 对应的目录
[root@VM_0_166_centos ~]# mkdir -p /var/lib/docker/devicemapper/mnt/c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab

8、重新挂载 docker 容器 edbffeca5cee 的 /var/lib/docker/devicemapper/mnt/容器ID 目录
[root@VM_0_166_centos ~]# mount /dev/mapper/docker-253:1-451037-c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab -o rw,relatime,nouuid,attr2,inode64,sunit=512,swidth=1024,noquota -t xfs /var/lib/docker/devicemapper/mnt/c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab
[root@VM_0_166_centos ~]# ll /var/lib/docker/devicemapper/mnt/c28ea7a61159b248967e7990c4d1606fde9d39405726545f959efb5a285284ab
total 8
-rw-------  1 root root   64 Dec 21  2017 id
drwxr-xr-x 18 root root 4096 Aug  9  2018 rootfs

9、测试此 docker 容器 edbffeca5cee 可以正常执行 docker exec
[root@VM_0_166_centos ~]# docker exec -it cpm02 bash
[root@edbffeca5cee home]# ls
apache-tomcat-7.0.70  apache-tomcat-7.0.70.zip  app  jdk1.7.0_79  jdk-7u79-linux-x64.tar.gz