Azure - CosmosDB Infrastructure IaC Demo

Title: Azure DevOps, GitHub and Azure - Project CosmosDB

Completion Date: October 2022

Category: DevOps

Demo Project Overview

This project uses Github to deploy Azure Resource Template (ARM) IaC code to Azure in a 2-stage deployment over Azure DevOps using reusable IaC templates. The solution includes a CosmosDB account, a database and a single container housing a partition key, an indexing policy and a sample stored procedure for the DocumentDB database container.

Project Description

This is the second project in the Azure-based demo project range where infrastructure rather than applications are deployed in a highly available pattern using IaC. The workflow is an infrastructure provisioning deployment. The first stage deploys a V-Net and 2 subnets to the Azure resource group at the resource group level. The second stage deploys a highly available solution for CosmosDB between 'westeurope' and 'northeurope' on Azure. The configuration of CosmosDB to the container level enables a highly available solution that can be easily used for applications requiring a NoSQL solution via DocumentDB. The base configuration of this managed service is deployed using Azure best security practices. Post-deployment configuration of CosmosDB using this IaC code can be customised for a multitude of production use cases requiring a managed NoSQL instance of DocumentDB via the Azure portal. It is noteworthy, that ARM API calls have a host of options available, which were excluded. A more detailed deployment can be coded up with specific requirements in the development phase of a project. CosmosDB is recommended for project requirements that have unpredictable data payloads, thus downtime periods of note are expected.

Project Workflow

The project workflow is as follows:

• A GitHub repo was created to work on the IaC development of the template and pipeline code. This repo enables version control, reuse plus more at https://github.com/johnmlhll/azure-ha-cosmosdb.
• Local coding up of the IaC deployment templates plus associated parameter files in JSON was done for the V-Net and then for CosmosDB.
• Azure DevOps project set up in the Maolte Technical Solutions Limited organization. We then set up an ARM service connection to our target Azure subscription. We did not set up an AZ Boards ticket for this project as it is detailed in the first Azure demo project and is also an auxiliary feature of Azure DevOps.
• We set up an (Azure Key Vault) secret for use in the project on Azure and imported the key vault into the Azure DevOps project library's variable group area.
• YAML code for azure-pipelines.yaml was then drafted and pushed to the GitHub repo. After the initial pipeline testing was done, the YAML was split into two stages and re-pushed to the repo.
• The pipeline at this stage is authorised for its connection to our Azure subscription along with the role assignment of the service principal to the RBAC 'Contributor' role. 
• From this step on, the changes in commits to commit 23 were micro in nature correcting typos, trying different options and features to experiment with the new ARM apiVersion 2022-05-15 plus more.
• During this debugging and exploratory stage, one of the Azure errors corrected was the initial registration at the subscription level for the provider Microsoft.DocumentDB for this deployment and future deployments of the NoSQL database on our targeted subscription.
• We developed the IaC template code by trying different features and correcting typos until we had a base deployment pattern we liked for a successful deployment noting requirements for the demo project are as broad as they get. We note this demo project was also a good opportunity to explore the ARM template versions we were using in IaC code development.
• When deployed successfully in a safe pattern, we documented our user story in video format and posted it to YouTube (see below).
• We recorded the project to its success via build notes on our kanban board.
• The final parts of the project were to deploy this demo project write-up and record learned lessons, which are based on the below project problems recording in our build notes.
• We then created a kanban project card for the next Azure-based project and placed it in the backlog list.

Project Problems

There were some problems in the project involving typos, which a JSON linter fixed. Others were more complex yet educational in nature. They are as follows:

• Linter in MS Code does not recognise Azure ARM apiVersion '2022-05-15' and suggests earlier API versions to fix it. I found this via exploratory debugging to not impact the API calls made but suspect the newness of this API version was the reason why "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers" type call failed the container creation stage of CosmosDB on multiple runs. This looks like an Azure API bug as I fixed it by adding a tags member, and a location member to the container IaC code block only then to remove the location member to see the error disappear over type member pre-pipeline update and run. The container deployed successfully updating the CosmosDB account DB.
• The first time using a new Azure product requires registration via the Azure resource group (storage account) command line. This was forgotten and it failed a deployment. It was fixed with 'az provider --register Microsoft.DocumentDB' on the bash command line via resource group storage account.
• Tagging via IaC failed the V-Net deployment. Exploratory debugging determined it was not supported at the subnet level with the API call used. A parent-child approach in the V-Net code block was not explored as tags were fully removed from the V-Net deployment, noting it for follow-up in future demo projects.
• The Azure system property 'resourceGroup().location' cannot be used in a template.parameters.json file. This will trigger a pipeline validation failure if not removed from the parameters file.
• If you after deleting your CosmosDB account via the Azure portal, are looking to deploy a fresh CosmosDB account, make sure you rename your resource parameter values in your deployment.parameters.json file. The delete process takes a considerable period of time to process before it is removed from Azure's managed indexes.
• Partition keys used for the deployment's development stage went to 13 characters long. Partition keys greater than 101 bytes require explicit enablement so character length is to be kept in mind when you are coding it in your IaC template. Do note longer character lengths in partition keys carry overhead so performance at scale is best served with smaller partition keys (other than 'id').
• CosmosDB container failures were a challenge on this project. They were debugged using code reviews, event log reviews and the Azure portal to inspect and assess what resources deployed successfully. The event logs in Azure DevOps are often non-descript as to the file location of the error. They only give a relative position in a JSON object that can be hundreds of lines long. Many error messages are non-descript and also require exploratory debugging. Also of note, detailed error messages posted in the event log output via URL posting were not always given in the event long error stack. This detailed pointer is required for more efficient debugging as less exploratory debugging is required.
• For cosmos DB instances of DocumentDB, a minimum throughput is 400ru is required noting auto scaleable instances are possible from 1000ru upwards (and also chargeable beyond the free tier). Throughput is only scaleable in manual increments of 100ru. If setting database throughput manually for the CosmosDB database as we did in this demo project, make sure it is greater than the container configured throughput. The provisioned database ru figure will require more throughput planning in a multi-container deployment as one database will service multiple containers.
• If you are deploying a multi-region CosmosDB IaC solution, you must set an indexing policy for the container(s).

Project Story

The project story as described above can be told in this short video: Project CosmosDB

Next Steps

Next steps in the Azure series is a demo project to deploy v-net, subnet x2 infrastructure via IaC templated stage along with a database provisioned and then an application. This project will be the most detailed in the Azure series and will be carded in our demo project board for action. The immediate next steps in our demo project series will once again move back to AWS.