Aged Relic CI/CD Process Update: Teamcity

kosar

This is the second article in the series on updating CI/CD processes. Up until now, we’ve been preparing to set up new tools: planning, daily meetings, resolving disagreements - basically, all the essentials to build an efficient workflow. Now that all questions are settled, we’re confident we can proceed with development and that our code won’t disappear into a black hole at the start of summer.

Here’s a recap of the article series contents and a brief summary of the previous part.

• Part 1: What we have, why it’s a problem, planning, and a bit of bash.
• Part 2: TeamCity.
• Part 3: Octopus Deploy.
• Part 4: Behind the scenes. Pain points, future plans, and perhaps an FAQ. Semi-technical.

We provided the client with a proof of concept for the new update delivery system, outlined the shortcomings of the existing system, chose a foundation for the new one, developed a transition plan, and converted our Mercurial repository to Git. In the previous part, we also described project features that will influence the entire subsequent process.

As previously mentioned, the current solution didn’t meet modern requirements. Originally, there was likely only one build configured in ccnet. The very first build that started it all (like the Big Bang, right?). Then, it seems the person who set it all up pressed ctrl+c, ctrl+v, tweaked a few lines, and voilà - a new configuration. Over time, this was done enough times that the source files on the build server grew to over 60GB. And that’s just the source files! There are also logs, build artifacts, temporary files, and so on. This alone made one pause. Looking at the old system, you’d see that each module had a build step. The same core was effectively rebuilt repeatedly, and this build was stored for each module (along with the sources) on the server. Build results (90% identical) were pushed to local repositories (git deployment), which also naturally took up space. This was something we could and needed to - avoid.

General setup

At this stage, it’s assumed that TeamCity and Octopus are already configured. I’ll skip over the installation and setup details. For convenience, I define the Octopus URL and API token in TeamCity’s environment variables, which, for the root project, looks like this:

• env.apiUserName and env.apiUserPassword are the credentials for the TeamCity API user (needed later)
• env.buildPath and env.sourcePath are the default paths for build files and sources respectively
• env.octoApiKey and env.octoUrl are the Octopus token and URL, along with env.teamcityUrl
• Lastly, env.tt.exe is the path to the text transform utility, which generates all configurations

The only external plugin installed is Octopus Deploy integration.

To avoid adding even more text, I’ll cover only the steps with noteworthy details. Everything else should be understandable from the information provided and the reader’s knowledge. If you have questions - feel free to ask.

Project breakdown, versioning, naming conventions

The resulting project structure looks like this:

• Modules
  • Modules dev
    • Build
    • Deployment configurations
  • Modules test (configurations similar to modules-dev)
  • Modules staging (configurations similar to modules-dev)
  • Modules production (configurations similar to modules-dev)
• Company website
  • Build and deploy dev
  • Build and deploy staging
  • Build and deploy prod
• Special module (not like the others. That is, not even a separate module, but completely different, not related to the main ones.)
  • Build and deploy dev
  • Build and deploy staging
  • Build and deploy production

We’ll version packages as follows: major.minor.patch, where major is currently 1, minor is the build counter from the Build configuration, and patch is the build counter for the Deploy configuration (not needed for Build, so it’s always 0).

Note: This versioning method is old. We’re currently reworking it to base it on Git tags.

For example, here, 1 is the major version, 95 is the minor, and 1 is the patch.

In the new process, modules will share a single build that will then be deployed to each module. In other words, sources are stored as a single instance for each environment, as is the build result since it’s shared among all modules. Unique configurations and libraries are packaged individually during deployment. This saves disk space and reduces delivery time for each package.

During preparation, we held a short meeting on naming conventions for configuration files. Previously, there was a convention (kind of), but it wasn’t consistent, and configuration names didn’t always contain enough information. In the end, all configuration names followed this format: ConfigurationType.Environment.ModName.config, where ConfigurationType can be AppSettings, Web, etc., Environment is development, test, staging, production, and ModName is the unique module name.

Modules

All Modules projects work on the same principle.

Each environment has a Build configuration that:

1. Runs NuGet restore (NuGet installer, NuGet restore)
2. Generates configurations (Command line, Text transform)
3. Builds the solution (Visual Studio sln)
4. Collects files from env.buildPath into a zip and pushes it to Octopus (Octopus deploy: push packages)
5. Creates a release (without deploying) (Octopus deploy create release)
6. Resets the patch version (powershell)

There are also deploy configurations based on a template that do the following:

1. Pull the configurations generated during the build phase (step 2) (Octopus deploy push packages)
2. Deploy the core release (created in step 5 of the build) (Octopus deploy: deploy release)
3. Deploy the individual module release (created in step 1 of this configuration) (Octopus deploy: deploy release)
4. Display the list of all domains for the given module in the log (a step for tester and developer convenience) (Powershell).

Each deploy configuration depends on the build configuration. This allows for automatically triggering a build when deploying if it’s outdated and resetting the patch version.

The Deploy_all configuration is essentially a build chain where we first run the build if it’s outdated and then deploy all modules in parallel.

It looks like this:

Let’s go over some of the steps in more detail.

Build.General settings

Only the build number format is unusual here. It’ll be explained later.

Build.env

Key variables:

• env.coreProjectName: set for the entire project, used to identify the project in Octopus.
• env.Environment: set for the Modules subproject, used to select the correct configs based on the environment.
• env.octoChannel: the Octopus channel for the package, matching env.Environment.
• env.serviceName: the solution name, primarily added for visual clarity.
• env.tenantRoleTag: the tenant tag in Octopus. More details in the Octopus section.

Build.4

Packages the build output from env.buildPath into a zip named %env.serviceName%.%build.number%.zip, excluding configurations and bin/native DLLs. These DLLs were manually added to the web servers and will stay there without updates. Updating them will also be a manual process (we might write a separate script for this, but let’s be honest). Excluding these libraries from deployment allows us to skip stopping the IIS pool, reducing downtime.

Build.5

This is mostly straightforward, except for the empty Environment field. Octopus determines the target environment based on the pre-release tag (set in channels). This tag is part of %build.number% (the -staging in the Build.General screenshot). This was more convenient in this case. Also, the checkbox “Show deployment process” is noteworthy. If unchecked, TeamCity considers the build successful once Octopus starts (but doesn’t finish!) its process. If something goes wrong during deployment, we won’t know without checking Octopus directly.

Build.6

Nothing unusual, just a PowerShell script and details from the TeamCity documentation.

$pair = "%env.apiUserName%:%env.apiUserPassword%"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
$basicAuthValue = "Basic $encodedCreds"
$depUri="%env.teamcityUrl%/app/rest/buildTypes?locator=snapshotDependency:(from:(build:(id:%teamcity.build.id%)),includeInitial:false)"
# Get all dependencies of main build
$result = (Invoke-RestMethod -Headers @{'Origin'='http://localhost:8090'; "Authorization"="$basicAuthValue"} -Uri $depUri)
foreach ($i in $result.buildTypes.buildType.Count) { $buildId += $result.buildTypes.buildType.id }
$buildId = $buildId | Where-Object { $_ -ne 'Module_DeployAll' }
# Reset all child build counters to 1. This build counters are patch versions in every build. (1.build.patch)
for ($i=0; $i -lt $buildId.Count; $i++) {
  $buildCounterUri = "%env.teamcityUrl%/httpAuth/app/rest/buildTypes/"+$buildId[$i]+"/settings/buildNumberCounter"
  Invoke-RestMethod -Method Put -Headers @{'accept'='text/plain'; 'Origin'='http://localhost:8090'; "Authorization"="$basicAuthValue"} -Uri $buildCounterUri -ContentType "text/plain" -Body 1
}

Here, we need a TeamCity API user (for which the environment variables were created). We authenticate, get all configurations dependent on Build, and reset their build counter to 1. Essentially, for each deployment, the patch version reflects the number of times the same build has been deployed for a specific module. If it’s not 1, it means something went wrong during deployment or with the module on this version and requires attention.

Deploy.General

Version format: 1.%dep.Module_Build.build.counter%.%build.counter%-staging, where %dep.Module_Build.build.counter% is a TeamCity system variable corresponding to the build counter for the Build configuration (which must be added as a dependency). Here, 1 is the major version, %dep.Module_Build.build.counter% is the minor, and %build.counter% is the patch. staging is the pre-release tag.

Deploy.Env

When adding a new module, only the deploy configuration needs to be added with the env.modName variable. All other variables inherit from parent projects. The env.tenantModTag variable for Octopus is generated from env.modName.

Deploy.2

Core package deployment.

• Release number latest: uses the latest release available for this environment.
• Tenant tag: client/organization tag. More on this in the Octopus section.
• Additional CLI parameters: here, we specify the channel for our package and additional parameters. This will also be covered in the Octopus part.

Deploy.3

Similar to Deploy.2 but deploys the module’s individual package.

Deploy.4

This step will also be detailed later as it retrieves data from Octopus.

The configurations for Company website and Special module follow the same principle, and the main points are the same, so I won’t go into them in as much detail. They’re uniform, don’t need multiple deployments, and essentially perform nuget restore, text transform, msbuild, octopus push, octopus create release + deploy.

More on unique configurations: some modules lack certain environments (e.g., staging), need to be deployed to different servers, or require separate client ID setup (by taking a base config and modifying specific fields with PowerShell). In other words, they operate the same as the main modules. Their uniqueness is in the added/modified steps or deployment server differences, making them unsuitable for the template, so they take a bit more time to set up. Fortunately, there are only a few of these.

TeamCity: Results

Implementing TeamCity allowed us to streamline the application build process. Now it takes much less time: instead of rebuilding the same code repeatedly, we build it ownce. We also use disk space, network traffic, and compute time more efficiently. Previously, the number of deploy packages (artifact repositories) equaled the number of modules, with each package weighing around 100MB compressed. Now we have one more package than modules: one package is around 100MB, while the others are about 500KB. Plus, we have a more user-friendly interface, convenient logs, and most importantly, reduced time for maintaining the build system and adding configurations. Adding a new module now takes 15-30 minutes.

Templates deserve special mention since they save us time on creating and maintaining new configurations. All Octopus interaction configurations (deploy) are based on a single template. First, it standardizes and simplifies the process, and second, as mentioned, it saves time on adding new configurations: just attach the template, change one variable, and it’s ready. Most importantly, templates simplify maintenance. Any changes made in the template propagate to configurations based on it. So, if we need to add a step across all configurations, we don’t have to edit them one by one—just update the template.

Naturally, the TeamCity setup process ran alongside the Octopus setup. It’s just impossible to describe both simultaneously without creating confusion, so the description is split, and this part only covers CI setup steps. The Octopus setup process will be covered in the next part.