Separating Kubernetes Kubeconfigs per Environment
The idea of having a singular Kubernetes config, authenticated to all your production, staging and dev environments alike, should fill you with trepidation. An accidental mis-selection from your shell history of a potentially destructive command could end up being enacted against the wrong cluster. In any case, a stark visual indicator of the current environment is desirable. Folks do this with browser configs, terminal sessions or IDE-database connections being colour coded to an environment, so we want to achieve a similar result here.
##
Tooling
We’re going to make use of Starship and direnv. Go ahead and get those installed now.
##
Setup
###
Starship config
There are two things we need to do to setup Starship.
- Starship to register a directory as containing Kubernetes related things by having it look for the presence of a
kubeconfigor ak8sfile. The latter is included so we can trigger Starship from any directory we wish and have it use the default kubeconfig. - Configure the prompt format based on keywords or regex.
[kubernetes]
disabled = false
symbol = '☸'
detect_files = ['k8s', 'kubeconfig']
style = "bold green"
format = '[$symbol CTX $context( NS \($namespace\))]($style) '
[[kubernetes.contexts]]
context_pattern = "prod(-.*)?|production|prod-.*|.*-prod"
style = "bold red"
[[kubernetes.contexts]]
context_pattern = "staging|stg(-.*)?|stage-.*|stg-.*"
style = "bold yellow"
###
Directory structure
My preferred setup is for the default Kubeconfig (in the home directory) to cover local dev (kind) so that the local cluster can be enacted on from anywhere, and per environment or per project configs to be loaded from a particular directory. You can do as much or as little as you like here.
You may end up with something along these lines:
tree -a ~/projects/kubernetes
/home/ben/projects/kubernetes
├── ankh-morpork
│ ├── production
│ │ ├── .envrc
│ │ └── kubeconfig
│ └── staging
│ ├── .envrc
│ └── kubeconfig
├── lancre
│ ├── production
│ │ ├── .envrc
│ │ └── kubeconfig
│ └── staging
│ ├── .envrc
│ └── kubeconfig
├── quirm
│ ├── production
│ │ ├── .envrc
│ │ └── kubeconfig
│ └── staging
│ ├── .envrc
│ └── kubeconfig
└── README.md
####
.envrc
.envrc
Each .envrc is identical, containing:
export KUBECONFIG=./kubeconfig
##
How it works
cding into a directory activatesdirenvautomatically.direnvloads the given Kubeconfig.- Starship picks up the new Kubeconfig and determines the appropriate prompt.
This results in:
☸ CTX am-staging kubernetes/ankh-morpork/staging
Changing up the parent kubernetes directory, which contains a file named k8s thus results in:
☸ CTX kind-local-dev projects/kubernetes
You’ll have to take my word for it, as the above is text pasted into Markdown, but they are colour coded and we have thus achieved our goals.
It also works fine with kubectx for when you may have multiple clusters in the one environment, depending on how your setup works.
Finally, this plays nicely with Atuin especially in directory-first mode, in that any commands you have run contextually to a given directory (project and/or cluster in our case) will be there at your fingertips.
You can also make this entire directory structure into a Git repository. There should be nothing sensitive in your Kubeconfigs as auth details are in your home directory. There may also be nothing specific to your user, such that you can share the repo with your team.