Blog

Blogs

Blog article category for blog articles on this site covering the areas of DevOps, Cloud Infrastructure, Site Reliability, Technical Writing, Project Management and Commerical Writing along with Event Management and associated areas. 

Databricks Course in the Lakehouse

Databricks Course in the Lakehouse

As any Data practitioner in the data science, analytics or engineering areas will tell you, the industry is moving forward with technological developments at a lightning pace with algorithmic development leading the charge in what the industry can do in the digital era. If anything, the COVID pandemic has accelerated the pace of development as business migrates to the digital space and Databricks is one of many hyper scaling vendors riding the crest of this data industry boom. As an Infrastructure and DevOps Engineer, I can tell you that Databricks is in an exciting phase of growth and maturing at different rates depending on where you work in the stack. 

Databricks for Azure is more mature than Databricks for AWS thanks to the focus and investment Microsoft has given in developing its integration for Databricks. That said, Databricks for AWS was of more interest given its capabilities, which led me to the Databricks Lakehouse Essentials Accreditation when completing the Databricks Platform administrator course via Databricks Academy. I recommend it for all entering the data industry noting the content is basic but solid. The delivery however is somewhat rough, which I am sure will resolve over time given the newly minted nature of this course. The nice part about it was the very wide breadth of content covering all things Databricks starting with the problems around cost in mining traditional data warehouses versus technical drawbacks around the cheaper data lake assuming it does not turn into a data swamp with easy to make errors around hierarchies and indexing. The managerial aspects of the Databricks abstraction focus on Delta Lake, alongside the impressive cluster related features that integrate well with AWS, were strongly featured. Databricks is deployable via Databricks account user interface, API or CLI along with having a range of machine learning features and libraries that are nicely spun into this managed service. These are just some of the areas the platform administration course covers briefly noting the deeper dives are in other courses at Databricks Academy. The governance and data integrity aspects via Delta Lake over the Data Lake were covered in somewhat more detail, which is understandable given the problems it solves with that single layer of technology in the platform.

As someone who has worked on Databricks in the past, I can tell you that it is a versatile ETL tool using parquet as a default data type and integrates nicely with the likes of TensorFlow, MLFlow, Tableau, Power BI and other key tools for the data industry. It also works well with many cloud IdPs for SSO and SCIMs with AD for example. If you are in the industry and looking for an ACID guaranteed Data Lakehouse ETL/Analytics solution, I would advise stopping by Databricks and checking out what they have to offer. Stay tuned for more on infrastructure in this blog along with articles on other areas of interest in the writing and DevOps arenas. To not miss out on any updates on my availability, tips on related areas or anything of interest to all, sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards

John 

Openshift and Kubernetes

Openshift and Kubernetes

Why Openshift adds value to Kubernetes...

We have all heard the headlines around the orchestration renaissance as we migrated our applications to containers running on static clusters, then onto Kubernetes via Kubeadm builds. Industrious product development by AWS led to EKS, an immutable version of Kubernetes ADM build with Azure following up with AKS as its own product offering. Whilst both products are awesome by practically guaranteeing high availability and performance, the products are platform-centric despite their similarities. Each has its own command syntax, abstracted functionality, and its own HA patterns in terms of subnet deployments, etc. It's also correct to say that each has its own learning curve for the supporting engineer.

IBM Redhat took a different approach as a software provider rather than a service provider by developing Openshift. Like AKS/ECS, Openshift has its own commands, lower level immutability and its own functionality abstracted from the KubeAdm build of Kubernetes. This lower level of immutability is not as extensive however allowing Openshifts command syntax to line up near perfectly with the Kubernetes 'kubectl' client for container and cluster administration. The commands are impressively aligned even down to shortcuts such as 'kubectl get pods (or po) -n prod -o wide' is executable via the Openshift client 'oc get pods (or po) -n prod -o wide', which would return pods (containers) created in the namespace (called a project in Openshift) and associated meta details such as a pod's network assignment. This is an example of the versatility of Openshift, noting its console offering is as impressive a view of Kubernetes as you can get scoping all major areas of the platform into the graphical user interface from the command line. Kubernetes base builds via 'kubeadm' can be error-prone if not configured properly, especially around network mapping and security. The allow-deny paradigm is present in any new Kubernetes cluster and needs to be secured via network policies. Openshift presents a highly available option for cloud deployments that is preconfigured for optimal security features such as Redhat's service mesh and global encryption that has optional extras for high-security requirements such as FEDRAMP, etc. It's also cloud-agnostic, already integrable with AWS, Azure, GCP plus more including on-prem bare-metal deployments. So, it's very securable, has a highly available subnet configuration baked into the software and can be deployed via Jenkins plus other popular CICD applications.

So, that's some Openshift highlights in review. It's closely abstracted from Kubernetes with pre-configured features that reduce errors and increase availability along with being deployable on-prem and the cloud. For those who require a securable orchestration platform for containers, which is deployable across multiple cloud platforms and/or on-prem, the above is just a taste of the many reasons why Openshift will meet your requirements. 

Stay tuned for more on DevOps in this blog along with articles on other areas of interest in the Infrastructure and Writing arenas. To not miss out on any updates on my availability, tips on related areas or anything of interest to all, sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards

John 

 

Global NOG Alliance - Help Keep Ukraine Connected

Global NOG Alliance - Help Keep Ukraine Connected

We have all seen and are horrified by the brutal invasion by Russia into Ukraine, which has seen the country pounded by the Russian military without any care for international law or common decency from what I can see. Needless to say, we all have done what we can to help the Ukrainian people ranging from donations to signing up for the Ukrainian Foreign Legion plus more. After donating, I was moved to write this article after I got an email distribution from iNOG (Irish Network Operators Group) on the global Networker initiative to help Ukraine stay connected during these brutal times.

They are asking for help in funding and also technology donations of routers and switches along with financial contributions to supply Ukraine with the necessary infrastructure to stay connected during this brutal war. The field engineers installing and maintaining this equipment are literally targets for the Russian military for just doing their job. Imagine our on-call shift as Engineers that suddenly becomes lethal after a hostile power decides they want to take away our people's right to self-determination. I did and after donating myself am now sharing the link to the Global NOG Alliance initiate to keep Ukraine connected as https://nogalliance.org/our-task-forces/keep-ukraine-connected/.

I would encourage companies to contribute any of the listed equipment and the rest of us to chip by donating some money to this worthy network-related cause. It's amazing what we take for granted in peaceful lands, the connectivity and freedom we have to our voice heard, learn about the world around us and explore opportunities to grow in peace. Ukraine just wants the same and we free peoples in free lands are ideally positioned to make a difference in helping Ukraine. We are helping them to make a stand against a fierce and unjust invasion of their lands, their lives and their freedom by just staying connected. 

Stay tuned for more on writing in this blog along with articles on other areas of interest in the infrastructure and DevOps arenas. To not miss out on any updates on my availability, tips on related areas or anything of interest to all, sign up for one of my newsletters in the footer of any page on Maolte. I look forward to us becoming pen pals!

Best Regards

John