This is the Way – My K8s Learning Journey, Part 2: Installing kubectl is a multi-round fight (Round 1)

This post is one of a series about my journey in learning Kubernetes from the perspective of a total n00b. Feel free to suggest topics in the comments. For more posts in this series, click this link.

In my previous post, I had a decision to make – what should I use to install my first Kubernetes cluster?

I got some helpful feedback and decided on using straight up kubectl over something like minikube to install and configure the cluster. My plan is to read the docs, but to read them like I suspect most people read them – skim them, use the parts you need at the time you’re using them and miss important details. And it’s no surprise that I ran into issues by doing it that way.

And that’s OK!

That’s what this exercise is for; realism, learning and discovery. But, fair warning: this post will have twists and turns and you’ll start to feel like Charlie from Always Sunny…

So, I followed the guidance for installing kubectl on my CentOS 8.4 system, which can be found here:

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

I ran through the first few steps, and it worked like a charm. TL;DR warning, though – this blog does *not* result in success. But we did get some valuable lessons.

# curl -LO "https://dl.k8s.io/release/$(curl -L -s https:// dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 154 100 154 0 0 29 0 0:00:05 0:00:05 --:--:-- 32
100 44.4M 100 44.4M 0 0 6519k 0 0:00:06 0:00:06 --:--:-- 32.9M

# curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.i o/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 154 100 154 0 0 29 0 0:00:05 0:00:05 --:--:-- 30
100 64 100 64 0 0 9 0 0:00:07 0:00:06 0:00:01 47

# echo "$(<kubectl.sha256) kubectl" | sha256sum --check
kubectl: OK

# sudo install -o root -g root -m 0755 kubectl /usr/local/b in/kubectl

# kubectl version --client
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCom mit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate: "2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd 64"}

Then, I skipped past the “Install using package management” because I had already installed it and ran the next step of “Verify kubectl configuration.” Rather than read the text right below it, I copied/pasted the next command into my client:

# kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Wait. You mean I don’t have a cluster right now???

I read up a bit and saw this very helpful line:

In order for kubectl to find and access a Kubernetes cluster, it needs a kubeconfig file, which is created automatically when you create a cluster using kube-up.sh or successfully deploy a Minikube cluster.

Oops.

# find / -name kube-up.sh
#

That script doesn’t seem to exist when I run the above, so I have to copy/paste it from the GitHib repo it links to or manually run through kubectl config to create a config file. If I’m an admin, I’m probably copy and pasting a script, so YOLO!

# vi /tmp/kube-up.sh
# chmod 777 /tmp/kube-up.sh
# ./kube-up.sh
./kube-up.sh: line 33: ./../cluster/kube-util.sh: No such file or directory

Ok… what did I do wrong *THIS* time??

From the looks of the error, there’s a kube-util.sh file that it can’t find. And apparently, neither can I!

# find / -name kube-util.sh
#

Now, I didn’t see *any* mention of where to put the script, nor did it mention this other script. So I check out the README on the GitHub, and… it’s less than useful.

When I click on Getting started, it just takes me back to the setup docs for K8s, with a link back to “Install tools including kubectl”… So it seems I am now in a time loop.

But I think I can work my way out of this. On the GitHub repo, there is a kube-util.sh script. Now, it was last updated 2 years ago, so YMMV.

Since the error referenced /cluster, I went ahead and created that directory and moved the kube-up.sh script. Then I created kube-util.sh and re-ran kube-up.sh and….

# ./kube-up.sh
./../cluster/kube-util.sh: line 23: ./../cluster/../cluster/skeleton/util.sh: No such file or directory

Well shucks. Now it’s referencing *another* script. And it looks like it’s a folder (skeleton) in the same repo. So it seems like I need to copy the entire thing.

The easiest way to do that is to use GitHub to clone it to my local node.

https://docs.github.com/en/repositories/creating-and-managing-repositories/cloning-a-repository

So I install git and try to clone the repo.

# yum install -y git
# mkdir kubernetes
# cd kubernetes/
# git clone https://github.com/kubernetes/kubernetes.git
Cloning into 'kubernetes'…
remote: Enumerating objects: 1294436, done.
remote: Counting objects: 100% (179/179), done.
remote: Compressing objects: 100% (118/118), done.
remote: Total 1294436 (delta 80), reused 62 (delta 61), pack-reused 1294257
Receiving objects: 100% (1294436/1294436), 803.12 MiB | 29.92 MiB/s, done.
Resolving deltas: 100% (932833/932833), done.
Updating files: 100% (23342/23342), done.

So I created a “kubernetes” folder, but it turns out I didn’t need to. But that’s ok, it should still work. I see all the files and folders I’d expect.

# ls -la
total 4
drwxr-xr-x. 3 root root 24 Dec 16 14:37 .
dr-xr-xr-x. 18 root root 242 Dec 16 14:37 ..
drwxr-xr-x. 19 root root 4096 Dec 16 14:38 kubernetes
# cd kubernetes/
# ls -la
total 204
drwxr-xr-x. 19 root root 4096 Dec 16 14:38 .
drwxr-xr-x. 3 root root 24 Dec 16 14:37 ..
drwxr-xr-x. 4 root root 57 Dec 16 14:38 api
drwxr-xr-x. 7 root root 4096 Dec 16 14:38 build
drwxr-xr-x. 2 root root 4096 Dec 16 14:38 CHANGELOG
lrwxrwxrwx. 1 root root 19 Dec 16 14:38 CHANGELOG.md -> CHANGELOG/README.md
drwxr-xr-x. 9 root root 4096 Dec 16 14:38 cluster
drwxr-xr-x. 25 root root 4096 Dec 16 14:38 cmd
-rw-r--r--. 1 root root 148 Dec 16 14:38 code-of-conduct.md
-rw-r--r--. 1 root root 525 Dec 16 14:38 CONTRIBUTING.md
drwxr-xr-x. 2 root root 38 Dec 16 14:38 docs
-rw-r--r--. 1 root root 766 Dec 16 14:38 .generated_files
drwxr-xr-x. 8 root root 163 Dec 16 14:38 .git
-rw-r--r--. 1 root root 381 Dec 16 14:38 .gitattributes
drwxr-xr-x. 3 root root 93 Dec 16 14:38 .github
-rw-r--r--. 1 root root 2634 Dec 16 14:38 .gitignore
-rw-r--r--. 1 root root 1112 Dec 16 14:38 .golangci.yaml
-rw-r--r--. 1 root root 34923 Dec 16 14:38 go.mod
-rw-r--r--. 1 root root 58959 Dec 16 14:38 go.sum
drwxr-xr-x. 12 root root 8192 Dec 16 14:38 hack
-rw-r--r--. 1 root root 11358 Dec 16 14:38 LICENSE
drwxr-xr-x. 4 root root 68 Dec 16 14:38 LICENSES
drwxr-xr-x. 2 root root 4096 Dec 16 14:38 logo
lrwxrwxrwx. 1 root root 19 Dec 16 14:38 Makefile -> build/root/Makefile
lrwxrwxrwx. 1 root root 35 Dec 16 14:38 Makefile.generated_files -> build/root/Makefile.generated_files
-rw-r--r--. 1 root root 782 Dec 16 14:38 OWNERS
-rw-r--r--. 1 root root 10612 Dec 16 14:38 OWNERS_ALIASES
drwxr-xr-x. 32 root root 4096 Dec 16 14:38 pkg
drwxr-xr-x. 3 root root 31 Dec 16 14:38 plugin
-rw-r--r--. 1 root root 3387 Dec 16 14:38 README.md
-rw-r--r--. 1 root root 563 Dec 16 14:38 SECURITY_CONTACTS
drwxr-xr-x. 4 root root 66 Dec 16 14:38 staging
-rw-r--r--. 1 root root 1110 Dec 16 14:38 SUPPORT.md
drwxr-xr-x. 17 root root 250 Dec 16 14:38 test
drwxr-xr-x. 5 root root 85 Dec 16 14:38 third_party
drwxr-xr-x. 16 root root 4096 Dec 16 14:38 vendor
# pwd
/kubernetes/kubernetes

So, let’s try it!

Here we go…

# cd cluster
# ./kube-up.sh
… Starting cluster in us-central1-b using provider gce
… calling verify-prereqs
/usr/bin/which: no gcloud in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
Can't find gcloud in PATH, please fix and retry. The Google Cloud
SDK can be downloaded from https://cloud.google.com/sdk/.

Well… that’s progress, right?

Installing Google Cloud SDK

To install the Google Cloud SDK, go here (I google so you don’t have to!):

https://cloud.google.com/sdk/docs/install

First, you need Python2 and Python3.

# yum install -y python2 python3

Then, download one of the archive files. I grabbed the one for Linux 64-bit:

# curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-367.0.0-linux-x86_64.tar.gz

Then extract. I extracted to my homedir:

# tar -xvf google-cloud-sdk-367.0.0-linux-x86_64.tar.gz --directory ~

Then I run the included install.sh script:

# ~/google-cloud-sdk/install.sh

I went ahead and let it set the PATH variable and then I run init:

# ~/google-cloud-sdk/bin/gcloud init

There’s an auth link/verification code process, but other than that, pretty straightforward.

When I run the kube-up.sh again, I get this:

# ./kube-up.sh
... Starting cluster in us-central1-b using provider gce
... calling verify-prereqs
/usr/bin/which: no gcloud in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
Can't find gcloud in PATH, please fix and retry. The Google Cloud
SDK can be downloaded from https://cloud.google.com/sdk/.

That’s because the PATH doesn’t work until the .bash profile is re-read. Just run a source and try it again.

# source ~/.bash_profile
# ./kube-up.sh
… Starting cluster in us-central1-b using provider gce
… calling verify-prereqs
missing required gcloud component "alpha"
Try running $(gcloud components install alpha)
missing required gcloud component "beta"
Try running $(gcloud components install beta)

ARGGGGH

Installing gcloud alpha and beta…

# gcloud components install alpha
# gcloud components install beta

So I re-run it and…

!!! kubectl appears to be broken or missing
Required release artifacts appear to be missing. Do you wish to download them? [Y/n]
y
Can't determine Kubernetes release.
/kubernetes/kubernetes/cluster/get-kube-binaries.sh should only be run from a prebuilt Kubernetes release.
Did you mean to use get-kube.sh instead?

Sigh. Maybe? Let’s try get-kube.sh… But I’m not getting a ton of confidence from the K8s docs.

It looks like get-kube.sh simply repeats the steps from the first few commands as we began this blog. I’m beginning to think I’ve just installed Kubernetes twice in different places…

Downloading kubernetes release v1.23.1
from https://dl.k8s.io/v1.23.1/kubernetes.tar.gz
to /kubernetes/kubernetes/cluster/kubernetes.tar.gz
Is this ok? [Y]/n

So it finishes running and I get this error during bucket creation:

Creating gs://kubernetes-staging-afea54f323/...
AccessDeniedException: 403 The project to be billed is associated with a closed billing account.

The project it uses is the same one I defined when I first set up gcloud.

Add '/kubernetes/kubernetes/cluster/kubernetes/client/bin' to your PATH to use newly-installed binaries.
Creating a kubernetes on gce…
… Starting cluster in us-central1-b using provider gce
… calling verify-prereqs
… calling verify-kube-binaries
… calling verify-release-tars
… calling kube-up
Project: dark-garden-252515
Network Project: dark-garden-252515

This is what I did when I set up gcloud:

You are logged in as: [whyistheinternetbroken@gmail.com].
Pick cloud project to use:
[1] dark-garden-252515
[2] wise-dispatcher-252515
[3] Create a new project
Please enter numeric choice or text value (must exactly match list item): 1

Your current project has been set to: [dark-garden-252515].

So I guess because there’s no billing account associated, that’s why this fails?

I set up the billing account and re-run it the script. And it fails with the same error:

Creating gs://kubernetes-staging-afea54f323/…
AccessDeniedException: 403 The project to be billed is associated with a closed billing account.

So maybe the account that hosts the project is closed? So, I dig into the script that creates the “Can’t determine Kubernetes release” error (get-kube-binaries.sh) to see why that error gets generated. At this point, it feels like I’ll be reverting the snapshot on this VM and re-doing the installation once I work through all the errors.

Here’s where that error occurs:

function detect_kube_release() {
  if [[ -n "${KUBE_VERSION:-}" ]]; then
    return 0  # Allow caller to explicitly set version
  fi

  if [[ ! -e "${KUBE_ROOT}/version" ]]; then
    echo "Can't determine Kubernetes release." >&2
    echo "${BASH_SOURCE[0]} should only be run from a prebuilt Kubernetes release." >&2
    echo "Did you mean to use get-kube.sh instead?" >&2
    exit 1
  fi

So it tries to find KUBE_VERSION and then if it can’t, the error is generated. That is a variable that is set here:

KUBE_VERSION=$(cat "${KUBE_ROOT}/version")

KUBE_ROOT is this:

KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)

So it seems to be trying to cat a file named “version” and uses a path determined by KUBE_ROOT. With a “find” I was able to get a file with that info:

# find /kubernetes/kubernetes/ -name version
/kubernetes/kubernetes/cluster/kubernetes/version

So I went in and changed the script to make KUBE_ROOT the /kubernetes/kubernetes/cluster/kubernetes/ path. And that gets me past the “can’t determine Kubernetes version, but…

# ./kube-up.sh
... Starting cluster in us-central1-b using provider gce
... calling verify-prereqs
... calling verify-kube-binaries
... calling verify-release-tars
... calling kube-up
Project: dark-garden-252515
Network Project: dark-garden-252515
Zone: us-central1-b
BucketNotFoundException: 404 gs://kubernetes-staging-afea54f323 bucket does not exist.
Creating gs://kubernetes-staging-afea54f323
Creating gs://kubernetes-staging-afea54f323/...
AccessDeniedException: 403 The project to be billed is associated with a closed billing account. <<<<<WTF

So it’s looking like this method *may* be a dead end. At this point, I’m going to start over and when I get prompted for a cloud project to use, I’m selecting this one:

[3] Create a new project 

Current working theory is that the person who created that Git repo doesn’t maintain it anymore and whoever created those projects has let them expire.

Lessons learned in this attempt:

• Read everything first. Then try.
• Just because you read everything doesn’t mean it will work; sometimes the documentation sucks.
• Make sure you are prepared to tear it all down and start over.
• Knowing shell scripting is a useful tool to have and don’t be afraid to modify shell scripts – especially if they’re old.
• Everything has dependencies. In this case, we seem to be dependent on a now-defunct cloud bucket.
• Know Linux basics like “find,” “ls,” “chmod,” etc.
• Know that shell scripts require special permissions to run them.

We’ll learn more lessons in the upcoming post…

Advertisement