Fetching Private Go Modules During Docker Build

5 min readJun 5, 2019

After many hours of struggling with getting my go application to build within the building of my docker image, I decided this information could be useful to others. The problem I experienced, and likely why you are reading this, is that in order to fetch go dependencies from a private repository you need to setup git credentials within the docker image build.

Here are some errors that I was seeing through my struggles:

# ERROR 1
go: github.com/jwenz723/privatepackage@v1.0.0: unknown revision v1.0.0
go: error loading module requirements# ERROR 2
github.com/inContact/orch-entity-contact@v0.0.0-20190530222558-8a8073e6b7d7: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /go/pkg/mod/cache/vcs/72c960bac0ed57f7476f8d5540fea5f31c278ae0591ac68bee5481c69cc5a5ef: exit status 128:
	fatal: could not read Username for 'https://github.com': terminal prompts disabled

These problems occur because during the docker build process the git cli does not have credentials to access any private git repositories. You may say “But, I have an ssh key setup?” Your SSH keys and HTTPS credentials are on your host system and are not accessible during the docker build process (unless you mount them into the docker build).

Create a Personal Access Token

Before proceeding with the solutions listed below, you will need a personal access token that has ‘repo’ access to your private git repositories that contain your go dependencies. For example, if you use GitHub, then after logging in browse here. Click the button to create a new token, check the ‘repo’ option (see screenshot below), then click Generate Token. Copy the personal access token that is generated and save it somewhere for later.

Solution 1

The easy route is to create a .netrc file in a directory near your Dockerfile. The .netrc file should contain your git user and personal access token that you generated in the previous step. For example, if my git user was mygituser and my personal access token was 12345678901234567890 then my .netrc file would look like this:

machine github.com
login mygituser
password 12345678901234567890machine api.github.com
login mygituser
password 12345678901234567890

In my project I named the file netrc rather than .netrc so that it would not be a hidden file on my host system.

Now you need to create a Docker file that will copy in the netrc file at build time. Since the netrc file contains credentials, it is important that you don’t keep this file in the final built docker image. This is solved through using a multi-stage docker build. Here is my Dockerfile:

# First stage: build the executable.
FROM golang:1.12.5-alpine AS builder# git is required to fetch go dependencies
RUN apk add --no-cache ca-certificates git# Create the user and group files that will be used in the running 
# container to run the process as an unprivileged user.
RUN mkdir /user && \
    echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
    echo 'nobody:x:65534:' > /user/group# Copy the predefined netrc file into the location that git depends on
COPY ./netrc /root/.netrc
RUN chmod 600 /root/.netrc# Set the working directory outside $GOPATH to enable the support for modules.
WORKDIR /src# Fetch dependencies first; they are less susceptible to change on every build
# and will therefore be cached for speeding up the next build
COPY ./go.mod ./go.sum ./
RUN go mod download# Import the code from the context.
COPY . .# Build the executable to `/app`. Mark the build as statically linked.
RUN CGO_ENABLED=0 go build \
    -installsuffix 'static' \
    -o /app .# Final stage: the running container.
FROM scratch AS final# Import the user and group files from the first stage.
COPY --from=builder /user/group /user/passwd /etc/# Import the Certificate-Authority certificates for enabling HTTPS.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/# Import the compiled executable from the first stage.
COPY --from=builder /app /app# Perform any further action as an unprivileged user.
USER nobody:nobody# Run the compiled binary.
ENTRYPOINT ["/app"]

Now I can simply run docker build -t myimage . from the directory containing my Dockerfile and netrc file and I have an image that was built using private dependencies! The only problem with this solution is that it would be very bad practice to commit the netrc file that you created into a git repository. Read on…

Solution 2

Since we can’t commit the netrc file into git, then we should make the docker build create the file at build time. This can be accomplished using the following Dockerfile (the only difference between this file and the last is that the netrc file is being created by the printf RUN command using user specified build-args):

# First stage: build the executable.
FROM golang:1.12.5-alpine AS builder# It is important that these ARG's are defined after the FROM statement
ARG ACCESS_TOKEN_USR="nothing"
ARG ACCESS_TOKEN_PWD="nothing"# git is required to fetch go dependencies
RUN apk add --no-cache ca-certificates git# Create the user and group files that will be used in the running 
# container to run the process as an unprivileged user.
RUN mkdir /user && \
    echo 'nobody:x:65534:65534:nobody:/:' > /user/passwd && \
    echo 'nobody:x:65534:' > /user/group# Create a netrc file using the credentials specified using --build-arg
RUN printf "machine github.com\n\
    login ${ACCESS_TOKEN_USR}\n\
    password ${ACCESS_TOKEN_PWD}\n\
    \n\
    machine api.github.com\n\
    login ${ACCESS_TOKEN_USR}\n\
    password ${ACCESS_TOKEN_PWD}\n"\
    >> /root/.netrc
RUN chmod 600 /root/.netrc# Set the working directory outside $GOPATH to enable the support for modules.
WORKDIR /src# Fetch dependencies first; they are less susceptible to change on every build
# and will therefore be cached for speeding up the next build
COPY ./go.mod ./go.sum ./
RUN go mod download# Import the code from the context.
COPY . .# Build the executable to `/app`. Mark the build as statically linked.
RUN CGO_ENABLED=0 go build \
    -installsuffix 'static' \
    -o /app .# Final stage: the running container.
FROM scratch AS final# Import the user and group files from the first stage.
COPY --from=builder /user/group /user/passwd /etc/# Import the Certificate-Authority certificates for enabling HTTPS.
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/# Import the compiled executable from the first stage.
COPY --from=builder /app /app# Perform any further action as an unprivileged user.
USER nobody:nobody# Run the compiled binary.
ENTRYPOINT ["/app"]

Now to build the docker image using this Docker file you will specify your git user and personal access token as build-args when you execute the docker build command:

docker build -f Dockerfile.buildargs -t test --build-arg ACCESS_TOKEN_USR=mygituser --build-arg ACCESS_TOKEN_PWD=12345678901234567890 .

In a CI tool like Jenkins you can store the git user and personal access token as credentials that are then specified in your pipeline as environment variables. For example:

docker build -f Dockerfile.buildargs -t test --build-arg ACCESS_TOKEN_USR=${ACCESS_TOKEN_USR} --build-arg ACCESS_TOKEN_PWD=${ACCESS_TOKEN_PWD} .

This method of specifying credentials seems to work well for my use case. I know it is possible to also handle this using SSH keys, but I didn’t take the time to figure that out. If anyone has an example using SSH keys, please share in the comments.

You can see all these files in my GitHub repo here.

Fetching Private Go Modules During Docker Build

Create a Personal Access Token

Solution 1

Solution 2

Written by Jeff Wenzbauer