Blog article category for blog articles on this site covering the areas of DevOps, Cloud Infrastructure, Site Reliability, Technical Writing, Project Management and Commerical Writing along with Event Management and associated areas. 

Certified Kubernetes Administrator Exam

Certified Kubernetes Administrator Exam

Prepping for the CKA Exam...

This week is a good week, I sat my long-awaited Certified Kubernetes Exam with the Linux Foundation and passed with a comfortable margin on 1st try on Thursday 22nd November 2021. The examination was solely practical with a split Chrome GUI split vertically showing the questions and controls with a command-line emulator on the other side. It was done in Kubernetes v1.22 over 8 clusters (I think) and kubeadm was the build used.

I have to say, having Linux skills for this course and exam is required, which I think will fail you if you don't have basic command-line skills on Linux servers. If you are a new IT grad to the industry or not using Linux commands regularly, doing a Linux basics course with any provider will be a great first step to becoming a certified Kubernetes administrator. For those with great exam technique making up any gap in preparation, be warned as this exam is fully practical. Unless you effectively over prepare for the exam; you will be under time pressure risking failure in this highly sought after qualification. The good news is that if you prepare for it assuming you feel you are ready like I did, the Linux Foundation certainly helps you with a free resit and two mock exam simulators from I did (and strongly recommended) the CKA course @aCloudGuru by William Boyd. William does a great job of explaining Kubernetes, the exam topics and lays out great labs on real servers along with 10 exam questions, which I found very beneficial up to the day before exam day.

When I went to register for the CKA exam I purchased via the Linux Foundation portal, I was met with a shock on my readiness. I got as far as the exam and decided to do the first of two simulations open for 36 hours per simulation. They did in fairness warn me these are at the of the 'hard' question type in the CKA exam and will be present in fewer numbers in the real exam, and I can now confirm to be the case. I also noted in the actual exam that the question count in the 2-hour exam was augmented to allow for 'hard' questions, which is helpful. I got 17 questions to do in 2 hours. This time to question ratio should not be confused with a time surplus to complete questions. There is no excess room for getting stuck, which happens in all exams, especially practical only ones like this. I took this as a given and in exam prep, time trialled myself on 'normal' exam prep questions reducing my overall time to completion time for each aCloudGuru exam prep question by c.40% in the 4-6 weeks of part-time prep before going into 72-84 hours of full-time preparation before the exam. This muscle memory on the keyboard, familiarisation with file paths that configure key Kubernetes objects (e.g. /etc/kubernetes/manifests/pki/, /var/lib/kubelet/pki/), and comfort with 'hard' as much as 'normal' questions is extremely valuable in the exam and for our continuing journey with Kubernetes in production.

Despite all of this good preparation, exam strategy is important. I had a bad webcam and it took nearly 40 mins to get registered and start the exam due to ID validation issues. They (and now I) recommend you practice showing clear legible IDs to your webcam as a test using zoom or some app that displays the webcam output. This tested my nerves pre-starting a complex practical exam and I got stuck in the actual exam in 3 of the first 5 questions but did not panic. I had planned a strategy for this pre-exam and just updated my notepad with those numbers in 'Exam Controls/Notepad' and moved on to 'normal' questions with enough time to revisit and complete these 'hard' questions I got stuck on. By that time, I had completed other 'hard' and 'normal' questions and was in the zone allowing me to reread the questions, debunking my first impression that they were all too long to complete. Two of the 3 had shorter paths to completion with the 3rd being meaty in its path to completion. I completed them all with 5 mins to go to exam completion. Completion of questions is the goal, getting as much right as you go. If you 'cat /filepath/answer.txt' to check your answer is actually there, it's time well spent allowing you to confirm that you are done with that question. 

Here is my advice for anyone committed to doing and passing the Certified Kubernetes Administrator course. The following is based upon my own experience of doing the course and passing the exam by a comfortable margin:

a) Study

- Linux: Be sure you have the Linux command line know how to navigate the command line and perform basic functions on it.
- Do a Course: Do a good course dedicated to the Certified Kubernetes Administrator (aka CKA) exam, not any other Kubernetes exam type. I can recommend the aCloudGuru course, and I believe the course with Linux Foundation is good also.
- Production experience compliments, rather than substitutes: I have production support experience with Kubernetes, this does not substitute for a structured learning program covering exam topics. If you have design experience and many years of production support experience, it's conceivable but not advised to skip the course material.
- Study the content taking notes on each video/chapter that summarises it concisely. Test yourself the next day by reading the nodes and seeing if you can picture the flow of events in its content along with the objects used, etc. Spend at least 1 hour every 2/3 days with a multi-hour session if possible during your days off.
- Labs & Exam Questions: Only do a course that provides labs on each topic along with exam prep questions that are well rated. A big part of my comfort with the 'normal' question type was getting used to it (including the kubectl config use-context) via exam prep questions on the aCloudGuru course I took.
- If you break as I did in the course to take on a demanding project like launching my freelancing business, you will lose your learning curve on the course. Well taken notes are highly valuable in picking right up where you left off. It worked as a great time-saving refresher activity, as I got back onto the learning path. Do note the labs were equally as important to get rethreaded on the course path.

b) Exam Preparation

- Finish the Course: Taking shortcuts on the course by just watching the videos and doing labs/exam questions one time only diminishes your chances of success. It's an all practical exam, which can go easily wrong in the exam environment leaving you stuck and having to spend the time you don't have, even if you fully comprehend what is been asked of you. Finish the course material revising as you go per the above points and be comfortable understanding as much as doing the course's practice questions and labs. I find it a good idea to talk yourself through what's been asked and how you are going to go achieve it commenting on your progress as you go. 
- Docs are your Friend: in particular is your friend. Do exam questions and practice having a web browser or console filling circa 75% of your screen space to mimic the exam command-line emulator on Google Chrome. In the exam, you have one tab for the exam questions and a command-line emulator with a screen split of 25%/75% so mimicking this from the start will help get comfortable with your environment. This aids in practice as you search for Kubernetes objects in the search bar on and apply them to your yml solution.
- Declarative Commands: Cutting and pasting is allowed in the exam from as it is an open exam but its time consuming to make sure your YAML white spacing is correct. 'vi config' and 'i' for INSERT is a better idea with correct web formatting of the configuration you are pasting been carried forward into the file. When you have it in, 'ESC' and ':wq' out of the file to convert it into a .yml by 'mv config config.yml'. I found this to be a real time-saver and handy for production also.
- Imperative commands: Commands like 'kubectl run pod --image=nginx --dry-run=client -o yaml > pod.yml' is a very good tool for generating a pod configuration. There are about 3-4 imperative commands I used comfortably because I drilled with them relentlessly through the circa 4 weeks of preparation. Whilst there are more, practice with what you found comfortable to use and 'practice practice practice using the learnt commands and techniques. The same applies to the point above around using the 'Docs' URL allowed in the exam to create declarative commands i.e. 'kubectl create -f pod.yml'. Practice is key to success.
- Time Trials: you will easily find yourself taking the time you don't have in the exam doing normal questions. Train for this by doing practice exam questions in sequence, recording your time to the second when doing them and redoing them with a view to reducing your time to completion. Only start this when you understand what you are doing and why as you may get tricky wording in exam questions on the day. The objective of time trails is to become competent at process flows in Kubernetes admin that may factor in part or in full into questions on exam day. I found this to be extremely useful preparation benefiting me on exam day.
- Exam registration: This should form part of your exam preparation phase. Creating an account with or logging into your Linux Foundation account will allow you to browse the catalogue and purchase the CKA exam. Scheduling the exam via your Linux Foundation account is a separate task. Once you purchase (not schedule) the exam, you will be able to test your computer for compatibility in the exam, and most importantly do the exam simulators x2. My exam purchase cost me $375 (Eur328)
- 1st exam simulator: Do the first exam simulator about a week before your intended date for the exam and expect to do poorly in it. Your purpose here is to familiarise yourself with the format and layout, which is a near-perfect match to the exam layout and try the 2-hour test to see how you do on 'hard' questions. When the two hours is complete, check your score to see what tasks were completed successfully and what failed. Go through the uncompleted ones in your own time and check the score again to see how you did. This will give you an idea of where you are at regarding the hardest type of questions in the exam. Take a break from the simulator and try your course exam preparation (normal) questions to clear your mind of it and keep your normal question time proficiency intact for normal questions. You should then be in a position to set/schedule your exam date
- 2rd exam simulator: Use this after much normal question practice and revision of any grey areas of understanding via your notes to test your exam readiness. If you achieve 70/75+ correct tasks of the total 100+ tasks, then you should be ready for the real thing. Take it 2 days before your scheduled exam so you finish with the simulator the day before the exam
- Exam strategy: After doing the exam simulators, finalise your strategy and approach for the day including what you will do if you get a question you are stuck on, and what if you fail. I chose to record the question in the Exam Controls/Notepad text pane and move on to the next. I got early 'hard' questions, which could have sunk my exam timing goal of 7 mins a question on 'normal' questions but taking note of the question numbers and completing the others was a strategy that saved me. I was able to be more relaxed, more accurate and come back with 40 minutes to spare to complete the hard ones I skipped. I also recorded questions I was dubious about yet completed noting them in a separate line on the exam Notepad allowing me to make a revisit of them time permitting. Double-checking all answers to make they are recorded on x file and/or answer the question was also done to raise my performance bar. Slow is smooth and when it comes to trickily worded questions and there are plenty of them in the CKA.
- Fail to plan, plan to fail: I reserved time slots in my calendar for the free repeat if I did not pass it the first time around. It's important to keep the time gap short in days to retaking it because your accumulated command-line skills are ramped up for it. Anything later will see you lose your edge making a repeat more difficult than it has to be.
- Good exam conditions: They are important as you will be locked into the exam in deep thought for 2 hours. An ergonomic desk and chair are important but make sure you do the exam on a 27-inch monitor plus. I did the first practice exam simulator on my 16-inch mac screen and it was a disaster. I am short-sighted and had to squint to see the command line with glasses on the 75% screen coverage the command line emulator takes. I tried CMD + and I lost the end of the screen display, which is also an issue as it ties you up in knots trying to fix it. It's also a little laggy, which happens with web emulators especially if your broadband is not top tier. These issues are mitigated with a strategy pivot as I did to resetting up one of my 27-inch monitors and allow myself the luxury of space. I drilled that way for the final 2 days to exam day in exam recreation format in the 2nd exam simulation.
- Test your exam strategy using the exams provided: I augmented my exam strategy twice due to the insights I gained from the exam preparation simulators. Think your strategy through and be comfortable it works in line with your preparation, so you can deal with adversity in the exam.
- Linux Foundation documentation: The links you get in the registration process via Candidate Handbook are a must-read for candidates as they inform you of the exam, its requirements and expectations

C) Exam Day Strategy


- Well-rested and ready: Be well-rested and relaxed on the day of your exam and reasonably do not be needing to use the bathroom. You can request a toilet break in the exam controls in the exam but your exam timer keeps on ticking, while you are away. It's time you really do not have given the challenge of covering all exam questions.
- Test your webcam: I was nearly 40 minutes getting registered as my old webcam was not up to the task of clearly showing my drivers licence, it was ok with the less reflective surface of my passport. These problems led to panic starting the exam in a very nervous state. Pre-exam, test your ID placing it up to your webcam using Zoom/MS Teams/Skype/etc and if it cannot focus enough, it will not do. My passport got me out of the jam as it was not as reflective as my driver's licence, which I initially used. I have a new webcam now, so best to make sure your hardware is up to the task before scheduling your exam.
- Step into the groove: Do an exercise like a question to build a cluster or upgrade a cluster from your exam prep questions. Idea is to get into the Kubernetes world calling your knowledge into your mind's foreground around 1 hour before you start
- 15 mins early: You can log into the exam 15 mins early, which will give you time to get your registration completed, which can be lengthy as I found out.
- Lateness will forfeit your exam fee: Make sure you log into the exam and launch it no later than 15 minutes after your scheduled date or you will forfeit your exam and your right to a resit. You will have to fork up a new exam fee with a 2nd CKA exam purchase. It's in the rules and should not be overlooked.
- Chrome Exam GUI: Your exam preparation simulations do a great job of preparing you for the format of the exam. However, lean into the exam instructions on launching the exam after it's released by the proctor and take time to note where your question dropdown navigator, Exam Controls/Notepad and question flag is before you clear the introduction part and the timer starts. They are the three main controls I used during the exam.
- Proctor Instructions: Proctor instructions were crystal clear and augment your pre-exam your knowledge of what to do if something goes wrong, read them carefully via live chat.
- Question reading: Read and reread every question, if you don't know what you are been asked, note the number in your notepad and move on. Nerves at the beginning will make comprehension harder, so keep recording those questions numbers in the exam notepad and move on to a question you have read/reread, which triggers an "aha, I know this one" in your head. Make sure to pay close attention to keywords e.g. Daemonset V Deployment and instructions such as the use of context at the beginning to allow your answer to be graded. Not all questions will require 'kubectl config get-context blahblah' but most do.
- Reduce typographical errors with aliases: I used 'alias k=kubectl', which was a great help. Others use Linux variables, which I did not trust given context was changing on most questions. I did not bother repeating the alias in root user, as my need to be in it was minimal. I was mindful though that I had to use the full kubectl when I 'sudo -i' into the root. As I practised with typing '--dry-run=client -o yaml >' instead of exporting it to a variable 'd', I was ok with not using variables.
- Check questions: When you get a question complete, run over what you did in your head, reread the question to make sure you did as requested and cat any files you echo'd answers into. I caught two misses on echos using this technique and the peace of mind knowing I have done all I can do without killing my time discipline leaving me more relaxed and confident heading into the next question.
- kubectl config use-context: If you don't follow the instructions and use it before answering a question where instructed to, you will not get due credit for your solution in the exam score. Read all questions carefully.
- If you lag behind: Even with this technique, you can lag behind and not complete all questions. Try not to panic and think at least the completion of 13 questions will leave you with a fighting chance of passing. If you don't pass, learn from the questions that snagged you and solve them to a point where you fully understand the task and can do it super quick. Not passing is not a failure, it's a learning opportunity.
- Resit: if you don't pass, try to schedule a resit within days and keep your exam takeaways to incorporate them along with any necessary exam strategy changes into your preparation for your next exam. Keeping it to a measure of days is demanding, but it keeps your first exam preparation skill up and builds upon it for a successful resit.

Finally, if you are sitting the CKA, I hope you find this article useful and best of luck in your quest to become a Certified Kubernetes Administrator. 

Stay tuned for more on Writing in this blog along with articles on other areas of interest in the Infrastructure and DevOps arenas. To not miss out on any updates on my availability, tips on related areas or anything of interest to all, sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards


Why Technical Writing is a Customer Bridge

Why Technical Writing is a Customer Bridge

Technical Writing connects the dots for your customer...

In times past, technical writing has been deemed a nuisance, an afterthought for many engineering teams as they drive their code through QA under enormous pressure and out into production. The subsequent rise of technical writing has been one of necessity, yet adoption has been slow considering the ramifications of not embracing it by widening a team's skill profile to include a technical writer. Let's face it, software developers don't have the time or the willingness to embrace technical writing as a primary task. Project and product managers have the process knowledge as they understand it from software developers, yet may lack the skills of a technical writer to create, review/refine and push to a production-based platform a technical document for the customer. Bearing in mind our customer user experience is a revenue affecting priority, what can a technical writer bring to the team that bridges the knowledge gap between the customer and the product?

Convention - For starters, a good technical writer will know commercial 'styles' of such a the Chicago Manual of Style are great aides for in-house conventions by their employer and thus simplify what goes where in a technical document. Whilst the presentation of documents to the customer becomes somewhat uniform, it all becomes redundant if the technical repository upwards is made on an ad-hoc basis. Those companies who do not implement an industry-wide convention like Darwin Information Typing Architecture (DITA) may likely lose information access ease, version control, and extendability of their repository for their customer, and/or internal users alike. Standardisation of what and where people can access knowledge regarding your product is key to achieving end-user operating competency, and satisfaction with the product.

Use Case Adoption - Software Developers by far are the most expert in the composition and process flow of the product they work on, but can often be poor technical writers for the same reason. They often lack the understanding of being in the customer's shoes operating the product, which can lead to serious oversights in technical process flows in their documentation based on what they think a customer knows and does not know. It's too easy to raise the bar to the standard of an engineer, when engineers may not use the documentation. 

Silos - Company culture may be somewhat hierarchical in nature and technical documentation tasking often gets offloaded by a software development team to other teams notably marketing, product management or even a PMO function. This creates an incomplete knowledge transfer to professionals in other areas, who do not possess the technical understanding of the process flows, nor be able to identify the gaps that need to be filled upon the initial knowledge transfer. Like a DevOps Engineer, who bridges the gap between development and operations, a good technical writer can bridge the gap between business needs such as the target market for the documentation and the technical process flows. The technical writer will also be versed in creating technical documentation in a manner that is version controlled, auditable, easy to access and easy to understand for the end-user.

Software Development, DevOps and/or Operations teams who use a technical writer will achieve a more credible technical documentation source for their target market, which will increase customer/end-user satisfaction. By adopting technical writing into your development teams, you increase your product effectiveness and associated returns, which can only be described as a win-win for you, your customer and your product's overall success.  

Stay tuned for more on Writing in this blog along with articles on other areas of interest in the Infrastructure and DevOps arenas. To not miss out on any updates on my availability, tips on related areas or anything of interest to all, sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards


Backing Up Etcd Database in Kubernetes

Backing Up Etcd Database in Kubernetes

Why backing up a cluster's Etcd database is so important...

Kubernetes as an orchestration tool of choice has made quite the footprint from its humble origins in the open-source community, mainly thanks to the devotion and development of the community. This along with the subsequent investment by leading tech companies has made it a champion of orchestration automation and high availability (HA).

In prepping for my Certified Kubernetes Administrator exam and from past experience, I can advise without reservation the following:

Unless your Kubernetes enviornment is immutable (e.g. AWS EKS/Azure AKS), you need to backup your Etcd Database in additon to your existing disaster recovery plan

Here are the reasons why:

- Etcd database (and yes, there are many talented engineers out there who don't know what it does) stores your Kube-API server transactions. This means it stores the state of your Kube cluster as the Kube-API server is the entry point for the cluster.

- If you use Kubernetes in a stacked or external etcd architectural pattern, these HA patterns will have the latest state data on the etcd database, making a backup every x minutes by cron job worthwhile versus retrieving your assets from an external source (or region if you are in the cloud), should both the primary and backup cluster go down at once into an unrecoverable state. Your time to mitigate on a destroyed cluster in production will be greatly reduced as a result.

- You can recreate a cluster with your original deployment.yaml file and just retrieve your etcd-backup.db from backup to restore its state. This means you can with the knowledge or an automated process in place, create a new cluster as was before, then restore the state to the etcd database from your last etcd database backup. This becomes useful in restoring the cluster, in a disaster recovery context for a mutable Kubernetes environment where all related clusters were destroyed in the major incident.

image of cluster etcd database backup command

Let's see what's involved in the etcd database backup and restore. To save your snapshot noting the image above is not a production console output, and the file path for the snapshot save will be your s3/file storage URL or path as detailed in the example below. So, a cron job could point to a script that creates a new backup DB by timestamp and may destroy the oldest database file in the storage folder as an administrative action that will achieve the snapshot save and store objective. Let's check out the following command from the etcd node or control plane server with etcdctl installed:

ETCDCTL_API = 3 etcdctl snapshot save /filepath/you/wish/to/save/to/etcd-backup.db \

--endpoints= \

--cacert=/filepath/to/cacert/etcd-ca.pem \

--cert=/filepath/to/cert/etcd-cert.crt \


With one command, can have taken a snapshot of your database! Hopefully, you will never have to use it but if disaster strikes, swift mitigation of your major incident is paramount. After the recreation of the cluster, you can have your cluster's state restored with the backup restore command.

image of etcd database restore command

 The command is similar to the backup command and can also be scripted for automation as a recovery tool. The script should be based on the following command:

ETCDCTL_API = 3 etcdctl snapshot restore /path/to/backup/etcd-backup.db \

--initial-cluster etcd-restore= \

--initial-advertise-peer-urls \

--name etcd-restore \

--data-dir /var/lib/etcd/

Note both commands in a pinch can be run from the command line to backup and restore your etcd database and thus your cluster's state. However, I would advise automation and sewing this into your disaster recovery process (and runbooks) for your Kubernetes cluster.

Stay tuned for more on DevOps in this blog along with articles on other areas of interest in the Infrastructure and Writing arenas. To not miss out on any updates on my availability, tips in related areas or anything of interest to all, why not sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards