One of the biggest challenges in software development is keeping track of the dependencies’ version releases and following the most up-to-date versions as soon as possible. While this is critical toward reducing technical debt in the long term, it can raise some challenging moments and a lot of effort to check, upgrade, merge, test, and release. Those dependencies can have different levels of criticality and can impact the build or run phase of software.
Some typical dependencies are:
Libraries and Frameworks
Container Images
Runtimes
Importance and necessity
It seems important, on paper, to keep everything up to date all the time, but in reality, the importance relies on the following axis:
Criticality: How Critical is that update?
Life Cycle: How frequently are new versions released?
Effort: How many projects shall be changed?
Let’s explore a bit more. Some dependencies are used in many projects, such as @types/node, but it is used in the build phase, and the new version release rate is low enough. It is rarely released as a major version upgrade and follows the node runtime’s version, but minor and patch versions are released regularly. Another example is @aws-sdk\client-dynamodb, while the rate of releases is lower, this impacts the runtime, and once a change is definitively required, the whole organization will put lots of effort into updating. Another important note is how critical updates are rolled out for any package. Does it respond to any security issue or not?.
Enterprise Level Ecosystem
An ecosystem represents the overall leading system to any level of revenue. That system runs on top of many components that collaborate to achieve the final asset, and all those components communicate via defined interfaces called contracts. Not talking about contracts here, but the released assets that are settled on two sides of any single line.
At first glance, those boxes are some autonomous services communicating together, but what a service means to every engineer counts a lot. Some consider a service a dynamic component with a database residing on a separate isolated tier with some business logic, and some others consider a service can have different levels of granularity and flexibility. There are many options. It might be only some complex business logic without any database, protocol, or memory, such as a consumed library at runtime to help extract some metadata from a web URL path, or it can be a container running as a gateway proxy to some partner services.
Looking at a single box and imagining the potential dependencies it may have can be a fun part of the game. The following figure tries to explode a single box to as many pieces as possible.
The above list is not an exclusive list of all dependencies but ones I suffered in experience with. Let’s explore the challenges each one brings
Build time:
Execution Container: These are the application Dockerfiles built from base images such as public.ecr.aws/docker/library/node:22.13.1-slim. The base images for runtime are less evolved than base images such as public.ecr.aws/awsguru/aws-lambda-adapter:0.9.0, which evolve more frequently.
CDK: The CDKs are used at development time and evolve frequently as they encapsulate various services and infrastructure. However, they also release versions if any security issue is reported.
Delivery Dependencies: They evolve less frequently (Such as GitHub Actions) but can be hard to track and keep up to date as they are often declarative and out of sight.
Compiler & Bundler: In reality, bundlers and compilers are often once configured and keep working for years, but the rate of Bundler releases is high enough that it can never be tracked to define the real upgrade necessity.
Test Runners and Linters: Are always some packages/libraries that are at a normal change rate; they never impact the real environment. However, it is nice to keep them updated.
L3 constructs: These are custom and internal CDK constructs developed internally, and they might have a higher or lower release rate based on the problem they solve. The tricky part is when they rely on official CDKs, so not keeping them updated gives some dependency version conflicts.
Run time:
Execution Runtime: The release of new runtimes is not as often as other options, but EOL must be tracked, and necessary action and upgrades must be done.
Libraries: Keeping them up-to-date in the JS ecosystem is a must. The JS ecosystem is big enough and often driven by the community, which means more people with different knowledge and culture can contribute, and hidden issues can be produced in terms of functionality or security. But the interesting point is that the resolution of those issues is often fast enough to cover the produced issue.
SDKs: They provide some simple-to-use interfaces to abstract the real complexity. These are similar to libraries but are often maintained in a structured way and under an enterprise ownership. Often, the rate of releases is low.
Frameworks: It is hard to name Frameworks as runtime dependencies as they are transpired to JS, however, they can impact the runtime environment.
Database Engines: This category of engines is the hardest part per experience, as often the upgrades need a full regression testing. Some challenging Engines are PG-SQL, My-SQL, and OpenSearch. The rate of major releases is often low, such as once or twice yearly. The minor and patches are more frequently released. There are options to auto upgrade patch or minors while using managed services, but some instability can be experienced based on the engine and other factors.
💡
In this article we try to explore the automation possibilities of the above list except the Database Engines and Execution Runtime that need to be applied with control and planification.
GitHub Dependabot
One of the best tools to simplify the process and keep dependencies up to date is Dependabot. It is a free tool maintained by GitHub. It covers a variety of package managers and bundlers. For dependencies, it covers most known such as npm, pnpm, and yarn but also docker.
💡
The abovementioned dependencies will be explored in this article.
The configuration gives enough flexibility to manage a variety of scenarios such as single or mono-repo, private or public dependencies, development or production dependencies , etc.
💡
UPDATE:
Andres Moreno raised an important point related to dependabot access to Environment Variables and Secrets, The Only Secret that Dependabot has access to , is GITHUB_TOKEN, for all other needs, the variables and secrets must be separately configured for dependabot as it has no access to repository Secrets or Env Variables
Npm Dependencies
To manage npm dependencies the dependabot looks at repository and detect the package manager by presence of package-lock.json and looks at any single package version on registry to find if a new version available. Following example shows a simple dependabot config.
version: 2
updates:
- package-ecosystem: 'npm'
directory: '/'
schedule:
interval: 'weekly'
commit-message:
prefix: "chore"
include: "scope"
Mono Repo dependencies
Dependabot will determine the pnpm process if finds the pnpm-lock.yaml , the package-ecosystem for pnpm has the same as npm , both must have the npm value. By default using the above config it will run into all modules and will raise pullrequests for upgrading packages. But if any grouping is configured those are applied correctly to root level package.json but all modules will have single PRs per dependency which is not ideal.
version: 2
updates:
- package-ecosystem: 'npm'
directories:
- 'packages/**'
- '/'
schedule:
interval: 'weekly'
commit-message:
prefix: "chore"
include: "scope"
Grouping Dependencies
By default Dependabot creates a Pull Request per package that can become cumbersome and hard to manage and need to deal with conflicts , etc. Grouping allows to raise single Pull Request for a configured set of dependencies as shown below.
version: 2
updates:
- package-ecosystem: 'npm'
directories:
- 'packages/**'
- '/'
schedule:
interval: 'weekly'
groups:
dependencies:
update-types:
- 'patch'
- 'minor'
- 'major'
commit-message:
prefix: "chore"
include: "scope"
Dev and Prod Dependencies
To separate the build and run dependencies, you can give those dependencies different grouping and have separate PR per group, this allows to better take actions and test based on the group PR.
version: 2
updates:
- package-ecosystem: 'npm'
directories:
- 'packages/**'
- '/'
schedule:
interval: 'weekly'
groups:
prod-dependencies:
dependency-type: 'production'
dev-dependencies:
dependency-type: 'development'
commit-message:
prefix: "chore"
include: "scope"
The above config group development ad production dependencies in different processes and raise separate PRs for each group.
Isolating Major, Minor and Patch versions
It is a good practice to separate the production dependencies ( those impacting runtime ) from development ones, however, in each category the Major versions are the most fearful and separating them from the minor and path version updates will save the time , energy and mental health.
version: 2
updates:
- package-ecosystem: 'npm'
directories:
- 'packages/**'
- '/'
schedule:
interval: 'weekly'
groups:
prod-dependencies:
dependency-type: 'production'
update-types:
- "minor"
- "patch"
dev-dependencies:
dependency-type: 'development'
update-types:
- "minor"
- "patch"
ignore:
- dependency-name: "*"
update-types: [ "version-update:semver-major" ]
commit-message:
prefix: "chore"
include: "scope"
- package-ecosystem: "npm"
directories:
- 'packages/**'
- '/'
target-branch: "prod"
schedule:
interval: "monthly"
groups:
prod-dependencies:
dependency-type: "production"
update-types:
- "major"
dev-dependencies:
dependency-type: "development"
update-types:
- "major"
ignore:
- dependency-name: "*"
update-types: [
"version-update:semver-minor",
"version-update:semver-patch"
]
commit-message:
prefix: "chore"
include: "scope"
The above config ignores all semver major and only looks for minor and patch updates.
Docker Dependencies
Dependabot upgrade Dockerfiles by looking at stages ( FROM ) and apply updates as configured. In following example the major versions are excluded imperatively as the example relies on nodejs base image, but, a Major config ( excluding minor and patch ) can be added if the base image Major releases are desired.
version: 2
updates:
- package-ecosystem: "docker"
directories:
- 'packages/**'
schedule:
interval: "weekly"
ignore:
- dependency-name: "*"
update-types: [ "version-update:semver-major" ]
commit-message:
prefix: "chore"
include: "scope"
Private and Public Registries
By default the public registers are verified such as https://registery.npmjs.com/ , but private registers can be considered by adding them in dependabot registries section. When using pnpm adding a .npmrc file helps to eliminates the dependabot confusions and looking at both public and private dependencies and applying the fallback pattern if package is not found.
version: 2
registries:
npm-github:
type: npm-registry
url: https://npm.pkg.github.com
token: ${{secrets.MY_GITHUB_TOKEN}}
updates:
- package-ecosystem: 'npm'
directories:
- 'packages/**'
- '/'
schedule:
interval: 'weekly'
registries:
- npm-github
ignore:
- dependency-name: "*"
update-types: [ "version-update:semver-major" ]
groups:
prod-dependencies:
dependency-type: 'production'
update-types:
- "minor"
- "patch"
dev-dependencies:
dependency-type: 'development'
update-types:
- "minor"
- "patch"
commit-message:
prefix: "chore"
include: "scope"
The .npmrc file must be like the following example
registry=https://registry.npmjs.org
@xaaxaax:registry=https://npm.pkg.github.com
This helps to lead the Dependabot to recognize the scoped packages ( starting with @xaaxaax ) from GitHub packages and others from public registry.
Other Docker image registries
In scenarios when the base images are from other registries such as public.ecr.aws , this can be configured as a registry. Dependabot force having a username and password for docker registeries but some of public registries dont need that, however providing fake values for these two parameters is the solution.
version: 2
registries:
ecr-publichub:
type: "docker-registry"
url: "https://gallery.ecr.aws"
username: "fakeit"
password: "fakeit"
updates:
- package-ecosystem: "docker"
directories:
- 'packages/**'
schedule:
interval: "weekly"
registries:
- ecr-publichub
ignore:
- dependency-name: "*"
update-types: [ "version-update:semver-major" ]
commit-message:
prefix: "chore"
include: "scope"
Automating Pull Request Actions
If the project has enough barriers to validate the viability of dependencies’ upgrades some last parts of the process can be automated, such as approving, and merging pull requests. This can be achieved using the GitHub workflows, The process looks at pull request metadata, validates some details, approves, and merges the Pr.
name: Dependabot Automated Pull Requests
on:
pull_request:
types: [opened, reopened]
permissions:
pull-requests: write
contents: write
issues: write
repository-projects: write
env:
DEPS_SCOPE: 'production'
MAJOR_UPDATE: 'false'
jobs:
dependabot:
runs-on: ubuntu-latest
if: github.event.pull_request.user.login == 'dependabot[bot]'
steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
- name: Install pnpm
uses: pnpm/action-setup@v4
with:
version: latest
run_install: false
- name: Install dependencies
run: pnpm install --no-frozen-lockfile
- name: Build
run: pnpm run build:all
- name: Synth
run: pnpm run synth:all
- name: Test
run: pnpm run test:all
- name: Dependabot metadata
id: metadata
uses: dependabot/fetch-metadata@v2
with:
github-token: "${{ secrets.GITHUB_TOKEN }}"
- name: Define Dependencies scope
run: |
echo "DEPS_SCOPE=${{ steps.metadata.outputs.dependency-type == 'direct:development' && 'development' || 'production' }}" >> $GITHUB_ENV
echo "MAJOR_UPDATE=${{ steps.metadata.outputs.update-type == 'version-update:semver-major' && 'true' || 'false' }}" >> $GITHUB_ENV
- name: Create ${{ env.DEPS_SCOPE }} label
continue-on-error: true
run: gh label create ${{ env.DEPS_SCOPE }}
env:
GH_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Add a label for all ${{ env.DEPS_SCOPE }} dependencies
continue-on-error: true
run: gh pr edit "$PR_URL" --add-label ${{ env.DEPS_SCOPE }}
env:
PR_URL: ${{github.event.pull_request.html_url}}
GH_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Approve a PR
run: gh pr review --approve "$PR_URL"
env:
PR_URL: ${{github.event.pull_request.html_url}}
GH_TOKEN: ${{secrets.GITHUB_TOKEN}}
- name: Enable auto-merge for Dependabot PRs
if: ${{ env.MAJOR_UPDATE != 'true' }}
run: gh pr merge --auto --rebase --delete-branch "$PR_URL"
env:
PR_URL: ${{github.event.pull_request.html_url}}
GH_TOKEN: ${{secrets.GITHUB_TOKEN}}
The above workflow will be triggered each time a Pull Request is opened or closed by dependabot[bot] user. This is the Dependabot default user. The Pull Requests are build and tested and in case of not a major version bump , they are automatically merged.
Conclusion
While software development benefits from internally shared or community-driven dependencies, they can become an obstacle when an upgrade is required. The decision is based on many factors that lead to consider a dependency upgrade necessary or not.
While the factors are important, they often make things more complex than upgrading the dependencies frequently. Dependency upgrades just need to be categorized and automated as much as possible. Dependency upgrades can force some software changes or not; whatever the case, small changes are easier than looking at the whole implementation, so keeping track of versions and upgrading gives more chance to keep the minimum of effort and mental health.
