This is the Way – My K8s Learning Journey, Part 3: Installing kubectl is a multi-round fight (Round 2)

This post is one of a series about my journey in learning Kubernetes from the perspective of a total n00b. Feel free to suggest topics in the comments. For more posts in this series, click this link.

In my previous post called “This is the Way – My K8s Learning Journey, Part 2: Installing kubectl is a multi-round fight (Round 1)”, I failed miserably in my attempt to install kubectl.

And that’s OK!

That’s what this exercise is for; realism, learning and discovery.

Previously, my cluster build failed here:

# ./kube-up.sh 
... Starting cluster in us-central1-b using provider gce 
... calling verify-prereqs 
... calling verify-kube-binaries 
... calling verify-release-tars 
... calling kube-up 
Project: dark-garden-252515 
Network Project: dark-garden-252515 
Zone: us-central1-b BucketNotFoundException: 404 gs://kubernetes-staging-afea54f323 bucket does not exist. 
Creating gs://kubernetes-staging-afea54f323 
Creating gs://kubernetes-staging-afea54f323/... AccessDeniedException: 403 The project to be billed is associated with a closed billing account.

Using my superpower of deduction (and, well, reading comprehension), I figured the issue was probably related to the project I selected when I first tried to set up gcloud during the kube-up.sh script setup.

Inspector Gadget 80's T-Shirt

In that, I selected a pre-existing project.

You are logged in as: [whyistheinternetbroken@gmail.com]. 
Pick cloud project to use: 
[1] dark-garden-252515 
[2] wise-dispatcher-252515 
[3] Create a new project 
Please enter numeric choice or text value (must exactly match list item): 1 

This time, I selected “Create a new project.”

Pick cloud project to use:
[1] cv-solution-architect-lab
[2] dark-garden-252515
[3] versatile-hash-335321
[4] wise-dispatcher-252515
[5] Create a new project
Please enter numeric choice or text value (must exactly match list item): 5

Enter a Project ID. Note that a Project ID CANNOT be changed later.
Project IDs must be 6-30 characters (lowercase ASCII, digits, or
hyphens) in length and start with a lowercase letter. this-is-the-way-witib
Waiting for [operations/cp.8622655766389923424] to finish…done.
Your current project has been set to: [this-is-the-way-witib].

The kube-up.sh script, if you recall, wasn’t the right script. Instead, get-kube.sh is.

# ./kube-up.sh 
… Starting cluster in us-central1-b using provider gce
… calling verify-prereqs
… calling verify-kube-binaries
!!! kubectl appears to be broken or missing
Required release artifacts appear to be missing. Do you wish to download them? [Y/n]
y
Can't determine Kubernetes release.
/k8s/kubernetes/cluster/get-kube-binaries.sh should only be run from a prebuilt Kubernetes release.
Did you mean to use get-kube.sh instead?

Now, when I run through get-kube.sh again… it still fails. But with a different error:

Project: this-is-the-way-witib
Network Project: this-is-the-way-witib
Zone: us-central1-b
BucketNotFoundException: 404 gs://kubernetes-staging-91a201ac9a bucket does not exist.
Creating gs://kubernetes-staging-91a201ac9a
Creating gs://kubernetes-staging-91a201ac9a/…
AccessDeniedException: 403 The project to be billed is associated with an absent billing account.

Maybe that means I need to associate the project?

I went to my Google cloud project and clicked “Billing” and maybe that’s where the issue is:

I link the billing account:

Now, I re-run the script:

# ./get-kube.sh
'kubernetes' directory already exist. Should we skip download step and start to create cluster based on it? [Y]/n
y
Skipping download step.
Creating a kubernetes on gce…
… Starting cluster in us-central1-b using provider gce
… calling verify-prereqs
… calling verify-kube-binaries
… calling verify-release-tars
… calling kube-up
Project: this-is-the-way-witib
Network Project: this-is-the-way-witib
Zone: us-central1-b
BucketNotFoundException: 404 gs://kubernetes-staging-91a201ac9a bucket does not exist.
Creating gs://kubernetes-staging-91a201ac9a
Creating gs://kubernetes-staging-91a201ac9a/…
+++ Staging tars to Google Storage: gs://kubernetes-staging-91a201ac9a/kubernetes-devel
+++ kubernetes-server-linux-amd64.tar.gz uploaded (sha512 = be1b895d3b6e7d36d6e8b9415cb9b52bb47d34a3cb289029a8bce8a26933c6cc5c0fe2e9760c3e02b20942aa8b0633b543715ccf629119946c3f2970245f6446)
+++ kubernetes-manifests.tar.gz uploaded (sha512 = 26c653dd65408db5abe61a97d6654d313d4b70bf0136e8bcee2fc1afb19a3e2d26367b57816ec162a721b10dd699fddd4b28211866853b808a434660ddf5e860)

Hey, that was it! But I’m wondering how much $$ this is going to cost me, because stuff is getting created and downloaded…

API [compute.googleapis.com] not enabled on project [493331638462]. Would you like to enable and retry (this will take a few minutes)? (y/N)? y

Enabling service [compute.googleapis.com] on project [493331638462]…
Operation "operations/acf.p2-493331638462-3027f3b3-c93e-4a03-bdcb-f7767fa0406a" finished successfully.
...
Creating firewall...
Creating firewall...
IP aliases are disabled.
Creating firewall...
Found subnet for region us-central1 in network default: default
Starting master and configuring firewalls
Configuring firewall for apiserver konnectivity server
Creating firewall...
Creating firewall...
Created [https://www.googleapis.com/compute/v1/projects/this-is-the-way-witib/zones/us-central1-b/disks/kubernetes-master-pd].
..Created [https://www.googleapis.com/compute/v1/projects/this-is-the-way-witib/global/firewalls/kubernetes-default-internal-master].
NAME                  ZONE           SIZE_GB  TYPE    STATUS
kubernetes-master-pd  us-central1-b  20       pd-ssd  READY

Luckily, I still have my free trial!

Now, I have a public IP

Looking for address 'kubernetes-master-ip'
Using master: kubernetes-master (external IP: x.x.x.x; internal IP: (not set))
Waiting up to 300 seconds for cluster initialization.

This will continually check to see if the API for kubernetes is reachable.
This may time out if there was some uncaught error during start up.

……….Kubernetes cluster created.
Cluster "this-is-the-way-witib_kubernetes" set.
User "this-is-the-way-witib_kubernetes" set.
Context "this-is-the-way-witib_kubernetes" created.
Switched to context "this-is-the-way-witib_kubernetes".
User "this-is-the-way-witib_kubernetes-basic-auth" set.
Wrote config for this-is-the-way-witib_kubernetes to /root/.kube/config

Kubernetes cluster is running. The master is running at:
https://x.x.x.x

The user name and password to use is located in /root/.kube/config.

And the nodes are all validated and the installation succeeds!

Validating gce cluster, MULTIZONE=
Project: this-is-the-way-witib
Network Project: this-is-the-way-witib
Zone: us-central1-b
No resources found
Waiting for 4 ready nodes. 0 ready nodes, 0 registered. Retrying.
No resources found
Waiting for 4 ready nodes. 0 ready nodes, 0 registered. Retrying.
Waiting for 4 ready nodes. 0 ready nodes, 2 registered. Retrying.
Found 4 node(s).
NAME                           STATUS                     ROLES    AGE   VERSION
kubernetes-master              Ready,SchedulingDisabled   <none>   24s   v1.23.1
kubernetes-minion-group-fmrn   Ready                      <none>   12s   v1.23.1
kubernetes-minion-group-g7l6   Ready                      <none>   21s   v1.23.1
kubernetes-minion-group-hf4j   Ready                      <none>   16s   v1.23.1
Warning: v1 ComponentStatus is deprecated in v1.19+
Validate output:
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE                         ERROR
etcd-1               Healthy   {"health":"true","reason":""}
etcd-0               Healthy   {"health":"true","reason":""}
controller-manager   Healthy   ok
scheduler            Healthy   ok
Cluster validation succeeded
Done, listing cluster services:

Kubernetes control plane is running at https://x.x.x.x
GLBCDefaultBackend is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
CoreDNS is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Kubernetes binaries at /k8s/kubernetes/cluster/kubernetes/cluster/
You may want to add this directory to your PATH in $HOME/.profile
Installation successful!

One thing I don’t like is that I think I installed kubernetes twice. I’m guessing I needed to navigate to a different place when I ran the script (such as to /k8s, where the kubernetes directory would have been found and no need to re-download). This may be why I had to run get-kube.sh instead of kube-up.sh.

But I don’t think it’s a big enough problem to start over – yet – so I’ll keep going.

kubectl cluster-info runs successfully!

# kubectl cluster-info
Kubernetes control plane is running at https://x.x.x.x
GLBCDefaultBackend is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
CoreDNS is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://x.x.x.x/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

When I try to navigate to the control plane, however:

kind	"Status"
apiVersion	"v1"
metadata	{}
status	"Failure"
message	"forbidden: User \"system:anonymous\" cannot get path \"/\""
reason	"Forbidden"
details	{}
code	403

Going back to when the script ran successfully, I saw this:

The user name and password to use is located in /root/.kube/config.

So, I cat that file and see this:

  user:
    password: #########
    username: admin

Step by step

Here are the commands I ran on an initial installation of a fresh CentOS8 VM.

Now, the question is… how do I log in using that info? Or do I even need to?

# yum update -y
# curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
# curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
# echo "$(<kubectl.sha256) kubectl" | sha256sum --check
# sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
# kubectl version --client
# yum install -y git
# mkdir /k8s
# cd /k8s
# git clone https://github.com/kubernetes/kubernetes.git
# yum install -y python2 python3
# curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-367.0.0-linux-x86_64.tar.gz
# tar -xvf google-cloud-sdk-367.0.0-linux-x86_64.tar.gz --directory ~
# ~/google-cloud-sdk/install.sh
# ~/google-cloud-sdk/bin/gcloud init
# gcloud components install alpha
# gcloud components install beta
# cd cluster/
# ./get-kube.sh

I started to Google the error, but this looks like a deeper dive and the initial point of this post was to successfully install kubectl – which it looks like I did!

So, next post will be… how to get this working?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s