golang: netgo vs cgo
由于alpine很小巧, 其经常被拿来当作容器应用运行时的基础镜像, 但是稍不注意, 就会踩一个坑.
一般来讲, 如果我们想改变某个域名的地址, 除了更改DNS记录, 那就只有更改hosts文件了, 由于容器的便捷性, 我们总是会更改容器的hosts文件, 不管是通过 --add-host也好, 挂载hosts文件进去也好, 总之, 是更改这个文件.
问题发现
然鹅! 有时候竟然会出现更改hosts文件无效的情况! 明明在里面ping是可以的呀! 为什么程序就不走这个IP呢? (默认golang程序)
经过仔细的排查发现, 是golang程序的问题… 中间过程略去不表…
alpine由于是个小巧的linux系统, 里面没有 /etc/nsswitch.conf 文件, 默认编译的go程序(netgo)会首先去查找这个文件, 判断dns查找顺序, 如果找不到这个文件, 那就 “自行决断”, 去DNS服务器里查找了. 所以如果想解决这个问题, 可以在image里面把这个文件给 “补上”: 可以看这个PR
关于nsswitch.conf的详细介绍, 可以看这里 man 或者 man nsswitch.conf
实验验证
下面有个小实验:
package main
import "net"
const host = "www.google.com"
func main() {
println(host)
ips, err := net.LookupIP(host)
if err != nil {
panic(err)
}
for _, ip := range ips {
println("lookup:" + ip.String())
return
}
}
这个程序会查找google的IP地址并打印出来.
FROM golang:alpine
# remove this file
RUN rm /etc/nsswitch.conf
COPY main.go /root
RUN go build -o /bin/ncgo /root/main.go
CMD [ "/bin/ncgo" ]
# GODEBUG=netdns=go /bin/ncgo
# GODEBUG=netdns=cgo /bin/ncgo
这个dockerfile会编译这个文件, 并首先删除系统里的nsswitch.conf(alpine默认是没有的, golang后来添加的)
然后我们构建这个镜像并运行一下:
➜ netgo_cgo docker build -t netgo_cgo .
Sending build context to Docker daemon 5.408MB
Step 1/5 : FROM golang:alpine
---> 05fe62871090
Step 2/5 : RUN rm /etc/nsswitch.conf
---> Using cache
---> 3a553930831f
Step 3/5 : COPY main.go /root
---> Using cache
---> 458ea03b7b0c
Step 4/5 : RUN go build -o /bin/ncgo /root/main.go
---> Using cache
---> daa8a0fc89ae
Step 5/5 : CMD /bin/ncgo
---> Running in 49e471837516
---> 7fc1a4c16d04
Removing intermediate container 49e471837516
Successfully built 7fc1a4c16d04
Successfully tagged netgo_cgo:latest
➜ netgo_cgo docker run --rm -ti --add-host www.google.com:1.1.1.1 -e GODEBUG=netdns=go netgo_cgo
www.google.com
lookup:69.171.245.84
➜ netgo_cgo docker run --rm -ti --add-host www.google.com:1.1.1.1 -e GODEBUG=netdns=cgo netgo_cgo
www.google.com
lookup:1.1.1.1
➜ netgo_cgo
可以看到, 使用go默认的resolver会忽略hosts文件种的内容, 而使用cgo则读取了hosts文件中的内容.
这种使用环境变量去设定程序resolver的方式, 也可以在build的时候添加tag来实现:
go build --tags netcgo -o netcgo
go build --tags netgo -o netgo
那么, 这两个tag还有什么不同呢?
深入研究
我们加上调试参数来看看:
➜ netgo_cgo go build -x --tags netcgo -o netcgo
WORK=/tmp/go-build691479627
mkdir -p $WORK/net/_obj/
mkdir -p $WORK/
cd /opt/go/src/net
CGO_LDFLAGS="-g" "-O2" /opt/go/pkg/tool/linux_amd64/cgo -objdir $WORK/net/_obj/ -importpath net -- -I $WORK/net/_obj/ -g -O2 cgo_linux.go cgo_resnew.go cgo_socknew.go cgo_unix.go conf_netcgo.go
cd $WORK
gcc -fdebug-prefix-map=a=b -c trivial.c
gcc -gno-record-gcc-switches -c trivial.c
cd $WORK/net/_obj
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_export.o -c _cgo_export.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./cgo_linux.cgo2.o -c cgo_linux.cgo2.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./cgo_resnew.cgo2.o -c cgo_resnew.cgo2.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./cgo_socknew.cgo2.o -c cgo_socknew.cgo2.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./cgo_unix.cgo2.o -c cgo_unix.cgo2.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./conf_netcgo.cgo2.o -c conf_netcgo.cgo2.c
gcc -I /opt/go/src/net -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -I ./ -g -O2 -o ./_cgo_main.o -c _cgo_main.c
cd /opt/go/src/net
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/net/_obj/_cgo_.o $WORK/net/_obj/_cgo_main.o $WORK/net/_obj/_cgo_export.o $WORK/net/_obj/cgo_linux.cgo2.o $WORK/net/_obj/cgo_resnew.cgo2.o $WORK/net/_obj/cgo_socknew.cgo2.o $WORK/net/_obj/cgo_unix.cgo2.o $WORK/net/_obj/conf_netcgo.cgo2.o -g -O2
/opt/go/pkg/tool/linux_amd64/cgo -dynpackage net -dynimport $WORK/net/_obj/_cgo_.o -dynout $WORK/net/_obj/_cgo_import.go
cd $WORK
gcc -no-pie -c trivial.c
cd /opt/go/src/net
gcc -I . -fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -o $WORK/net/_obj/_all.o $WORK/net/_obj/_cgo_export.o $WORK/net/_obj/cgo_linux.cgo2.o $WORK/net/_obj/cgo_resnew.cgo2.o $WORK/net/_obj/cgo_socknew.cgo2.o $WORK/net/_obj/cgo_unix.cgo2.o $WORK/net/_obj/conf_netcgo.cgo2.o -g -O2 -Wl,-r -nostdlib -no-pie -Wl,--build-id=none
/opt/go/pkg/tool/linux_amd64/compile -o $WORK/net.a -trimpath $WORK -goversion go1.9.2 -p net -std -buildid 1d8743e51666b2f6e7dca905f790949850fff2bb -D _/opt/go/src/net -I $WORK -pack ./addrselect.go ./conf.go ./dial.go ./dnsclient.go ./dnsclient_unix.go ./dnsconfig_unix.go ./dnsmsg.go ./error_posix.go ./fd_unix.go ./file.go ./file_unix.go ./hook.go ./hook_unix.go ./hosts.go ./interface.go ./interface_linux.go ./ip.go ./iprawsock.go ./iprawsock_posix.go ./ipsock.go ./ipsock_posix.go ./lookup.go ./lookup_unix.go ./mac.go ./net.go ./nss.go ./parse.go ./pipe.go ./port.go ./port_unix.go ./rawconn.go ./sendfile_linux.go ./sock_cloexec.go ./sock_linux.go ./sock_posix.go ./sockopt_linux.go ./sockopt_posix.go ./sockoptip_linux.go ./sockoptip_posix.go ./tcpsock.go ./tcpsock_posix.go ./tcpsockopt_posix.go ./tcpsockopt_unix.go ./udpsock.go ./udpsock_posix.go ./unixsock.go ./unixsock_posix.go ./writev_unix.go $WORK/net/_obj/_cgo_gotypes.go $WORK/net/_obj/cgo_linux.cgo1.go $WORK/net/_obj/cgo_resnew.cgo1.go $WORK/net/_obj/cgo_socknew.cgo1.go $WORK/net/_obj/cgo_unix.cgo1.go $WORK/net/_obj/conf_netcgo.cgo1.go $WORK/net/_obj/_cgo_import.go
pack r $WORK/net.a $WORK/net/_obj/_all.o # internal
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/
cd /home/mr/test/go/netgo_cgo
/opt/go/pkg/tool/linux_amd64/compile -o $WORK/_/home/mr/test/go/netgo_cgo.a -trimpath $WORK -goversion go1.9.2 -p main -complete -buildid eb016ff3c1708c2bfbf31469f574aff99c045d11 -D _/home/mr/test/go/netgo_cgo -I $WORK -pack ./main.go
cd .
/opt/go/pkg/tool/linux_amd64/link -o $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out -L $WORK -extld=gcc -buildmode=exe -buildid=eb016ff3c1708c2bfbf31469f574aff99c045d11 $WORK/_/home/mr/test/go/netgo_cgo.a
cp $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out netcgo
好长一大串!
可以看到, golang使用了gcc去编译这个文件.
那么换成netgo呢?
➜ netgo_cgo go build -x --tags netgo -o netgo
WORK=/tmp/go-build857020109
mkdir -p $WORK/net/_obj/
mkdir -p $WORK/
cd /opt/go/src/net
/opt/go/pkg/tool/linux_amd64/compile -o $WORK/net.a -trimpath $WORK -goversion go1.9.2 -p net -std -buildid 12048e0b22a06eafa34fe1b04bfee87c29b39979 -D _/opt/go/src/net -I $WORK -pack ./addrselect.go ./cgo_stub.go ./conf.go ./dial.go ./dnsclient.go ./dnsclient_unix.go ./dnsconfig_unix.go ./dnsmsg.go ./error_posix.go ./fd_unix.go ./file.go ./file_unix.go ./hook.go ./hook_unix.go ./hosts.go ./interface.go ./interface_linux.go ./ip.go ./iprawsock.go ./iprawsock_posix.go ./ipsock.go ./ipsock_posix.go ./lookup.go ./lookup_unix.go ./mac.go ./net.go ./nss.go ./parse.go ./pipe.go ./port.go ./port_unix.go ./rawconn.go ./sendfile_linux.go ./sock_cloexec.go ./sock_linux.go ./sock_posix.go ./sockopt_linux.go ./sockopt_posix.go ./sockoptip_linux.go ./sockoptip_posix.go ./tcpsock.go ./tcpsock_posix.go ./tcpsockopt_posix.go ./tcpsockopt_unix.go ./udpsock.go ./udpsock_posix.go ./unixsock.go ./unixsock_posix.go ./writev_unix.go
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/
cd /home/mr/test/go/netgo_cgo
/opt/go/pkg/tool/linux_amd64/compile -o $WORK/_/home/mr/test/go/netgo_cgo.a -trimpath $WORK -goversion go1.9.2 -p main -complete -buildid a0e2e67cde44dfa98c4ccaf123bd7ff6ae1370f8 -D _/home/mr/test/go/netgo_cgo -I $WORK -pack ./main.go
cd .
/opt/go/pkg/tool/linux_amd64/link -o $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out -L $WORK -extld=gcc -buildmode=exe -buildid=a0e2e67cde44dfa98c4ccaf123bd7ff6ae1370f8 $WORK/_/home/mr/test/go/netgo_cgo.a
cp $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out netgo
➜ netgo_cgo
就短多了, 可以看到用的是 /opt/go/pkg/tool/linux_amd64/compile
在来看一下默认的build:
➜ netgo_cgo go build -x
WORK=/tmp/go-build285258876
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/
mkdir -p $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/
cd /home/mr/test/go/netgo_cgo
/opt/go/pkg/tool/linux_amd64/compile -o $WORK/_/home/mr/test/go/netgo_cgo.a -trimpath $WORK -goversion go1.9.2 -p main -complete -buildid 137ba24345f4e3075192f2afe0e3b5aad76f10b7 -D _/home/mr/test/go/netgo_cgo -I $WORK -pack ./main.go
cd .
/opt/go/pkg/tool/linux_amd64/link -o $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out -L $WORK -extld=gcc -buildmode=exe -buildid=137ba24345f4e3075192f2afe0e3b5aad76f10b7 $WORK/_/home/mr/test/go/netgo_cgo.a
cp $WORK/_/home/mr/test/go/netgo_cgo/_obj/exe/a.out netgo_cgo
➜ netgo_cgo
然后我们对比一下这些二进制文件:
➜ netgo_cgo ll
total 8004
drwxrwxr-x 2 mr mr 4096 Jul 31 23:31 ./
drwxr-xr-x 46 mr mr 12288 Jul 31 22:19 ../
-rw-r--r-- 1 mr mr 206 Jul 31 23:18 Dockerfile
-rw-rw-r-- 1 mr mr 229 Jul 31 23:17 main.go
-rwxr-xr-x 1 mr mr 2726957 Jul 31 23:30 netcgo*
-rwxr-xr-x 1 mr mr 2672063 Jul 31 23:31 netgo*
-rwxr-xr-x 1 mr mr 2726922 Jul 31 23:31 netgo_cgo*
➜ netgo_cgo ls
Dockerfile main.go netcgo netgo netgo_cgo
➜ netgo_cgo ldd net*
netcgo:
linux-vdso.so.1 (0x00007ffe718f9000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007efd89078000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efd88c87000)
/lib64/ld-linux-x86-64.so.2 (0x00007efd89297000)
netgo:
not a dynamic executable
netgo_cgo:
linux-vdso.so.1 (0x00007fff56818000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4d7ad12000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4d7a921000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4d7af31000)
➜ netgo_cgo
在容器内部可以看到:
➜ netgo_cgo docker run --rm -ti netgo_cgo ldd /bin/ncgo
/lib/ld-musl-x86_64.so.1 (0x7f0567c59000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f0567c59000)
➜ netgo_cgo
然后再看一下其他信息:
➜ netgo_cgo readelf -d netcgo
Dynamic section at offset 0x1a9120 contains 19 entries:
Tag Type Name/Value
0x0000000000000004 (HASH) 0x535060
0x0000000000000006 (SYMTAB) 0x535500
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000005 (STRTAB) 0x5352a0
0x000000000000000a (STRSZ) 579 (bytes)
0x0000000000000007 (RELA) 0x534c40
0x0000000000000008 (RELASZ) 24 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x0000000000000003 (PLTGOT) 0x5a9000
0x0000000000000015 (DEBUG) 0x0
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000006ffffffe (VERNEED) 0x534fe0
0x000000006fffffff (VERNEEDNUM) 2
0x000000006ffffff0 (VERSYM) 0x534f80
0x0000000000000014 (PLTREL) RELA
0x0000000000000002 (PLTRELSZ) 792 (bytes)
0x0000000000000017 (JMPREL) 0x534c58
0x0000000000000000 (NULL) 0x0
➜ netgo_cgo readelf -d netgo_cgo
Dynamic section at offset 0x1a9120 contains 19 entries:
Tag Type Name/Value
0x0000000000000004 (HASH) 0x535060
0x0000000000000006 (SYMTAB) 0x535500
0x000000000000000b (SYMENT) 24 (bytes)
0x0000000000000005 (STRTAB) 0x5352a0
0x000000000000000a (STRSZ) 579 (bytes)
0x0000000000000007 (RELA) 0x534c40
0x0000000000000008 (RELASZ) 24 (bytes)
0x0000000000000009 (RELAENT) 24 (bytes)
0x0000000000000003 (PLTGOT) 0x5a9000
0x0000000000000015 (DEBUG) 0x0
0x0000000000000001 (NEEDED) Shared library: [libpthread.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
0x000000006ffffffe (VERNEED) 0x534fe0
0x000000006fffffff (VERNEEDNUM) 2
0x000000006ffffff0 (VERSYM) 0x534f80
0x0000000000000014 (PLTREL) RELA
0x0000000000000002 (PLTRELSZ) 792 (bytes)
0x0000000000000017 (JMPREL) 0x534c58
0x0000000000000000 (NULL) 0x0
➜ netgo_cgo file net*
netcgo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, with debug_info, not stripped
netgo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, with debug_info, not stripped
netgo_cgo: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, with debug_info, not stripped
再来strace一下, 找一找关键的点:
… cogo的太长了, 感兴趣的自己对比吧, 大体意思就是, netcgo的调了各种lib,open这个那个的, 而netgo的是静态文件, 用了几个我不认识的调用就完事儿了:
...
...
futex(0xc420032490, FUTEX_WAKE, 1) = 1
epoll_wait(4, [], 128, 0) = 0
futex(0x5b5b50, FUTEX_WAIT, 0, NULL) = 0
epoll_wait(4, [{EPOLLIN|EPOLLOUT, {u32=4131852144, u64=140363663085424}}], 128, 0) = 1
read(3, "M\335\201\200\0\1\0\1\0\0\0\0\3www\6google\3com\0\0\34\0\1"..., 512) = 60
epoll_ctl(4, EPOLL_CTL_DEL, 3, 0xc420045b14) = 0
close(3) = 0
rt_sigprocmask(SIG_SETMASK, ~[], [], 8) = 0
clone(lookup:69.171.245.84
<unfinished ...>) = ?
+++ exited with 0 +++
所以, 得出的结论是, 使用cgo编译的程序(或者运行时为cgo), 运行的时候会调用系统的libc, 在alpine中, 虽然它的libc是精简过的, 但人家好歹也是读了hosts文件, 所以会达到预期的效果.
解决方案
- 在alpine镜像中添加
/etc/nsswitch.conf文件 - 使用netcgo编译
- 增加一个GODEBUG环境变量, 强制使用netcgo