Drunk on Monads RSS Feed

Useful Azure DevOps Boards queries

Sat, 28 Dec 2024 00:00:00 GMT

Useful Azure DevOps Boards Queries

A collection of some useful Azure DevOps Boards queries.

Grooming queries

You will likely want to update the filters on these query to suite your needs, for example by filtering on specific work item types, state etc. What is saved and shown here are just the barebone queries.

View unparented items

View bugs parented under user stories/epics

⚠️This is a violation of the expected Epic -> Feature -> User story/Bug -> Task hierarchy

View all work items in the current iteration

You can also time travel across multiple iteration by adding +/- count in front of the @CurrentIteration

View active work that is not assigned to the sprint

This would likely indicate some previous work that has not been updated or has potentially fallen through the cracks.

View work items logged in the last x days

View ill defined work items

The default looks at items with no descriptions, or not acceptance criteria or no story points.

View work items pending review

View unestimated work items

View blocked work items

View work items that may need to be broken down This is determined by high estimates and you can decide on the threshold.

View stale work items You can decide on the time period for this to be considered stale.

View unassigned items in the current iteration

View work items that are due soon

View overdue work items

Other queries

View items assigned to me

View items that I am following

View items where I have recently been mentioned

Instead of constantly querying the grooming type queries, you can assign them to a dashboard as charts or as tile widgets so you always have a good view.

💡 In addition to placing charts and other visualization on a dashboard, you can only place these on the query itself. For example you can create a pivot table) to visualize the data from the query.

Here are recommendations of some great things to visualize

Bug trends over time
Task burn down
Test case pass/fail rate

Additional tips for working with queries

Utilize the predefined variables like @Me, @Today, @CurrentIterations etc to your advantage.
Use tags wisely. These can help filter work items, but if used without disciple they can go out of hand fast. As a general rule, if you have no use case to query by a given tag, you should not be using it. This will keep your tags list clean.

File Signature Checks in .NET

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Jason Mavrommatis on Unsplash

File Signature Checks in .NET

If you have an application that receives an file uploads, it is best practice to validate these files. It is common to see validation on extensions and mime types, although this is not fool proof and leaves lots of room for malicious files to be uploaded.

A more solid way to validate binary files is to check the unique header that each file type has, know as the magic number, that can be used to identify the type. For example if allowing PDF, you can verify that the signature matches that of a PDF. Depending on the file type, some will have more than one potential file signature.

In .NET FileSignatures is a really good library for this and has a very simple API for identifying files. It covers a lot of files that you are most likely to encounter, like popular image and video formats, PDF, Excel/Word/PowerPoint/Open Office, Outlook messages, Zip files and more.

Having started off reinventing the wheel using public domain information to verify signatures, it proved a bit hard as a lot of the sources were not complete. For example the information on Wikipedia states that a PDF will have the signature %PDF- or HEX 25 50 44 46 2D which generally is correct as the first bytes will be %PDF-[version number], except some PDF variants have no version and would just be %PDF. or HEX 25 50 44 46 This was one of many false negatives I was getting and on collating more information I came up with better checks, which happened to match what FileSignatures had when I stumbled upon it, giving confidence it was sufficiently battle tested. This was also a decent source of reliable signatures.

If you are not working with .NET taking a look at the implementation and seeing what signatures are used can be a great start to hand roll something your self.

Usage example

Think clean architecture and avoid the dependency leaking into your Domain and Use Cases by wrapping it in infrastructure.

Example Port:

public interface IFileFormat
{
    /// <summary>
    /// Returns true if the file is a PDF by checking the file signature
    /// </summary>
    /// <param name="stream"></param>
    /// <returns></returns>
    bool IsPdf(Stream stream);
    /// <summary>
    /// Returns true if the file is an image by checking the file signature
    /// GIF, BMP, JPEG, PNG, WEBP, TIFF
    /// </summary>
    /// <param name="stream"></param>
    bool IsImage(Stream stream);
}

Example Infra:

public class FileFormat : IFileFormat
{
    private readonly IFileFormatInspector _fileFormatInspector;

    public FileFormat(IFileFormatInspector fileFormatInspector) => _fileFormatInspector = fileFormatInspector;

    /// <summary>
    /// Returns true if the file is a PDF by checking the file signature
    /// </summary>
    /// <param name="stream"></param>
    /// <returns></returns>
    public bool IsPdf(Stream stream) => _fileFormatInspector.DetermineFileFormat(stream) is Pdf;

    /// <summary>
    /// Returns true if the file is an image by checking the file signature
    /// GIF, BMP, JPEG, PNG, WEBP, TIFF
    /// </summary>
    /// <param name="stream"></param>
    /// <returns></returns>
    public bool IsImage(Stream stream) => _fileFormatInspector.DetermineFileFormat(stream) is Image;
}

public static class FileSignatures
{
    /// <summary>
    /// Registers the file signatures service.
    /// </summary>
    /// <param name="services"></param>
    /// <returns></returns>
    public static IServiceCollection AddFileSignatures(this IServiceCollection services)
    {
        services.AddSingleton<IFileFormatInspector>(new FileFormatInspector());
        services.AddTransient<IFileFormat, FileFormat>();
        return services;
    }
}

// Registration in host
services.AddFileSignatures();

Pro Tip

When working with File Signature checks you risk getting false negative for some file types as there may be so many variations in the wild. Do make sure to log any failures and include at least the first 10 bytes of that file, so that you do not have to follow up asking for files to be shared with you which can be annoying for user and at times hard do the sharing of sensitive information using insecure mediums.

Example

    public override async Task<bool> Handle(ValidateFileUploadRequest request, CancellationToken cancellationToken)
    {
        using var __ = _logger.BeginScope(new Dictionary<string, object>
        {
            ["FileExtension"] = fileExtension,
            ["ContentType"] = request.ContentType,
            ["Header"] = GetFileHeader(request) // in case we get false negatives, we can investigate the file header and see if it's a valid file
        });
...
     if (!isValid)
         _logger.LogWarning("Uploaded file failed all validation.");
    }

    private static string GetFileHeader(ValidateFileUploadRequest request)
    {
        byte[] fileHeader;
        using (var reader = new BinaryReader(request.Stream, System.Text.Encoding.UTF8, leaveOpen: true))
        {
            // for investigation purposes, we read the first 10 bytes of the file and add to log context
            fileHeader = reader.ReadBytes(10); // safe, returns fewer bytes if the end of the stream is reached
            request.Stream.Position = 0;
        }

        return string.Join(", ", fileHeader.Select(b => $"0x{b:X2}"));
    }

OWASP ZAP API scan automation with Azure Pipelines

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Scott Webb on Unsplash

OWASP ZAP API scan automation with Azure Pipelines

[[TOC]]

OWASP ZAP) is a free, feature-filled web app scanner. Being under the OWASP banner, you can be sure that it is backed by a lot of industry experts and security best practices. This tool is particularly great for those who are new to security testing and have limited knowledge and skills, but would still like to ensure that their web apps are secure.

You can make use of the desktop application, which is available for download here). This article however goes over how to automate API scans using the docker images shipped by ZAP. Docker is the easiest way to get started with ZAP automation and offers a lot of flexibility, there are however other ways to automate ZAP).

Short intro to ZAP

At its core, ZAP is what is known as a man-in-the-middle proxy. It stands between the tester’s browser and the web application so that it can intercept and inspect messages sent between the browser and web application, modify the contents if needed, and then forward those packets to the destination. It can be used as a stand-alone application and as a daemon process.

ZAP ships out multiple docker images that can be used for scanning web apps, of interest ghcr.io/zaproxy/zaproxy:stable which is used in this article.

All of the docker images (apart from the ‘bare’ one) provide a set of packaged scan scripts:

Baseline Scan which runs the ZAP spider against the target for (by default) 1 minute followed by an optional ajax spider scan before reporting the results of the passive scanning.
Full Scan which runs the ZAP spider against the target (by default with no time limit) followed by an optional Ajax spider scan and then a full active scan before reporting the results.
API Scan which performs an active scan against APIs defined by OpenAPI, or GraphQL (post 2.9.0) via either a local file or a URL. - This is the one we shall be working with

ZAP API scan

The ZAP API scan is a script that is available in the ZAP Docker images. It is tuned for performing scans against APIs defined by OpenAPI, SOAP, or GraphQL via either a local file or a URL. It imports the definition that you specify and then runs an Active Scan against the URLs found. The Active Scan is tuned to APIs, so it doesn’t bother looking for things like XSSs.

It also includes 2 scripts:

Raise alerts for any HTTP Server Error response codes
Raise alerts for any URLs that return content types that are not usually associated with APIs

For our example, we are going to run the container with this configuration

🖱️ Click to expand

# Run a Docker container using the specified image
docker run \
# Mount the current directory to /zap/wrk/ in the container with read/write access
-v $(pwd):/zap/wrk/:rw \
# Allocate the container
-t ghcr.io/zaproxy/zaproxy:stable zap-api-scan.py \
# Target URL for the API scan, dynamically specifies the JSON file from Swagger
-t '$(ZapApiTarget)/$(apiSuffix)/swagger.json' \
# Specify HTML report file name
-r report.html \
# Output in JSON format
-j \
# Ignore minor warnings
-I \
# Set the alert threshold level (minimum level to consider)
-l INFO \
# Run all active scans
-a \
# Use specified config file for the scan
-c api-scan.conf \
# Use specified context file
-n api.context \
# Set the input format as OpenAPI
-f openapi \
# Set default level for additional rules
-i WARN

This will start the container to target a site URL we provide outputting an HTML report of the scan results.

The configuration file provides the rules that should be used in the scan and the desired response. A default configuration file can be created using the ‘-g’ parameter

We will make use of the following configuration

🖱️ Click to expand

# zap-api-scan rule configuration file
# Change WARN to IGNORE to ignore rule or FAIL to fail if rule matches
# Active scan rules set to IGNORE will not be run which will speed up the scan
# Only the rule identifiers are used - the names are just for info
# You can add your own messages to each rule by appending them after a tab on each line.
0	WARN	(Directory Browsing - Active/release)
10003	WARN	(Vulnerable JS Library - Passive/release)
10009	WARN	(In Page Banner Information Leak - Passive/alpha)
10010	WARN	(Cookie No HttpOnly Flag - Passive/release)
10011	WARN	(Cookie Without Secure Flag - Passive/release)
10012	WARN	(Password Autocomplete in Browser - Passive/release)
10015	WARN	(Re-examine Cache-control Directives - Passive/release)
10017	WARN	(Cross-Domain JavaScript Source File Inclusion - Passive/release)
10019	WARN	(Content-Type Header Missing - Passive/release)
10020	WARN	(Anti-clickjacking Header - Passive/release)
10021	WARN	(X-Content-Type-Options Header Missing - Passive/release)
10023	WARN	(Information Disclosure - Debug Error Messages - Passive/release)
10024	WARN	(Information Disclosure - Sensitive Information in URL - Passive/release)
10025	WARN	(Information Disclosure - Sensitive Information in HTTP Referrer Header - Passive/release)
10026	WARN	(HTTP Parameter Override - Passive/beta)
10027	WARN	(Information Disclosure - Suspicious Comments - Passive/release)
10028	WARN	(Open Redirect - Passive/beta)
10029	WARN	(Cookie Poisoning - Passive/beta)
10030	WARN	(User Controllable Charset - Passive/beta)
10031	WARN	(User Controllable HTML Element Attribute (Potential XSS) - Passive/beta)
10032	WARN	(Viewstate - Passive/release)
10033	WARN	(Directory Browsing - Passive/beta)
10034	WARN	(Heartbleed OpenSSL Vulnerability (Indicative) - Passive/beta)
10035	WARN	(Strict-Transport-Security Header - Passive/beta)
10036	WARN	(HTTP Server Response Header - Passive/beta)
10037	WARN	(Server Leaks Information via "X-Powered-By" HTTP Response Header Field(s) - Passive/release)
10038	WARN	(Content Security Policy (CSP) Header Not Set - Passive/release)
10039	WARN	(X-Backend-Server Header Information Leak - Passive/beta)
10040	WARN	(Secure Pages Include Mixed Content - Passive/release)
10041	WARN	(HTTP to HTTPS Insecure Transition in Form Post - Passive/beta)
10042	WARN	(HTTPS to HTTP Insecure Transition in Form Post - Passive/beta)
10043	WARN	(User Controllable JavaScript Event (XSS) - Passive/beta)
10044	WARN	(Big Redirect Detected (Potential Sensitive Information Leak) - Passive/beta)
10045	WARN	(Source Code Disclosure - /WEB-INF folder - Active/release)
10048	WARN	(Remote Code Execution - Shell Shock - Active/beta)
10049	WARN	(Content Cacheability - Passive/alpha)
10050	WARN	(Retrieved from Cache - Passive/beta)
10052	WARN	(X-ChromeLogger-Data (XCOLD) Header Information Leak - Passive/beta)
10054	WARN	(Cookie without SameSite Attribute - Passive/release)
10055	WARN	(CSP - Passive/release)
10056	WARN	(X-Debug-Token Information Leak - Passive/release)
10057	WARN	(Username Hash Found - Passive/release)
10061	WARN	(X-AspNet-Version Response Header - Passive/release)
10062	WARN	(PII Disclosure - Passive/beta)
10063	WARN	(Permissions Policy Header Not Set - Passive/alpha)
10094	WARN	(Base64 Disclosure - Passive/alpha)
10095	WARN	(Backup File Disclosure - Active/beta)
10096	WARN	(Timestamp Disclosure - Passive/release)
10097	WARN	(Hash Disclosure - Passive/beta)
10098	WARN	(Cross-Domain Misconfiguration - Passive/release)
10099	WARN	(Source Code Disclosure - Passive/alpha)
10105	WARN	(Weak Authentication Method - Passive/release)
10108	WARN	(Reverse Tabnabbing - Passive/beta)
10109	WARN	(Modern Web Application - Passive/beta)
10110	WARN	(Dangerous JS Functions - Passive/alpha)
10202	WARN	(Absence of Anti-CSRF Tokens - Passive/release)
2	WARN	(Private IP Disclosure - Passive/release)
20012	WARN	(Anti CSRF Tokens Scanner - Active/beta)
20014	WARN	(HTTP Parameter Pollution scanner - Active/beta)
20015	WARN	(Heartbleed OpenSSL Vulnerability - Active/beta)
20016	WARN	(Cross-Domain Misconfiguration - Active/beta)
20017	WARN	(Source Code Disclosure - CVE-2012-1823 - Active/beta)
20018	WARN	(Remote Code Execution - CVE-2012-1823 - Active/beta)
20019	WARN	(External Redirect - Active/release)
3	WARN	(Session ID in URL Rewrite - Passive/release)
30001	WARN	(Buffer Overflow - Active/release)
30002	WARN	(Format String Error - Active/release)
30003	WARN	(Integer Overflow Error - Active/beta)
40003	WARN	(CRLF Injection - Active/release)
40008	WARN	(Parameter Tampering - Active/release)
40009	WARN	(Server Side Include - Active/release)
40012	WARN	(Cross Site Scripting (Reflected) - Active/release)
40013	WARN	(Session Fixation - Active/beta)
40014	WARN	(Cross Site Scripting (Persistent) - Active/release)
40016	WARN	(Cross Site Scripting (Persistent) - Prime - Active/release)
40017	WARN	(Cross Site Scripting (Persistent) - Spider - Active/release)
40018	WARN	(SQL Injection - Active/release)
40019	WARN	(SQL Injection - MySQL - Active/beta)
40020	WARN	(SQL Injection - Hypersonic SQL - Active/beta)
40021	WARN	(SQL Injection - Oracle - Active/beta)
40022	WARN	(SQL Injection - PostgreSQL - Active/beta)
40023	WARN	(Possible Username Enumeration - Active/beta)
40026	WARN	(Cross Site Scripting (DOM Based) - Active/beta)
40028	WARN	(ELMAH Information Leak - Active/release)
40032	WARN	(.htaccess Information Leak - Active/release)
42	WARN	(Source Code Disclosure - SVN - Active/beta)
50000	WARN	(Script Active Scan Rules - Active/release)
50001	WARN	(Script Passive Scan Rules - Passive/release)
6	WARN	(Path Traversal - Active/release)
7	WARN	(Remote File Inclusion - Active/release)
90001	WARN	(Insecure JSF ViewState - Passive/release)
90002	WARN	(Java Serialization Object - Passive/alpha)
90003	WARN	(Sub Resource Integrity Attribute Missing - Passive/alpha)
90004	WARN	(Insufficient Site Isolation Against Spectre Vulnerability - Passive/alpha)
90011	WARN	(Charset Mismatch - Passive/release)
90019	WARN	(Server Side Code Injection - Active/release)
90020	WARN	(Remote OS Command Injection - Active/release)
90022	WARN	(Application Error Disclosure - Passive/release)
90026	WARN	(SOAP Action Spoofing - Active/alpha)
90028	WARN	(Insecure HTTP Method - Active/beta)
90029	WARN	(SOAP XML Injection - Active/alpha)
90030	WARN	(WSDL File Detection - Passive/alpha)
90033	WARN	(Loosely Scoped Cookie - Passive/release)

The context file provides an additional configuration that indicates what URLs should be scanned and additional scan configuration. Consult the desktop documentation for instructions on how to save the context from a configuration done with the application.

We shall be using the following context file

🖱️ Click to expand

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<configuration>
    <context>
        <name>Test API</name>
        <desc/>
        <inscope>true</inscope>
        <incregexes>https://c1sjmvgw-7023.uks1.devtunnels.ms/.*</incregexes>
        <tech>
            <include>Db</include>
            <include>Db.CouchDB</include>
            <include>Db.Firebird</include>
            <include>Db.HypersonicSQL</include>
            <include>Db.IBM DB2</include>
            <include>Db.Microsoft Access</include>
            <include>Db.Microsoft SQL Server</include>
            <include>Db.MongoDB</include>
            <include>Db.MySQL</include>
            <include>Db.Oracle</include>
            <include>Db.PostgreSQL</include>
            <include>Db.SAP MaxDB</include>
            <include>Db.SQLite</include>
            <include>Db.Sybase</include>
            <include>Language</include>
            <include>Language.ASP</include>
            <include>Language.C</include>
            <include>Language.JSP/Servlet</include>
            <include>Language.Java</include>
            <include>Language.Java.Spring</include>
            <include>Language.JavaScript</include>
            <include>Language.PHP</include>
            <include>Language.Python</include>
            <include>Language.Ruby</include>
            <include>Language.XML</include>
            <include>OS</include>
            <include>OS.Linux</include>
            <include>OS.MacOS</include>
            <include>OS.Windows</include>
            <include>SCM</include>
            <include>SCM.Git</include>
            <include>SCM.SVN</include>
            <include>WS</include>
            <include>WS.Apache</include>
            <include>WS.IIS</include>
            <include>WS.Tomcat</include>
        </tech>
        <urlparser>
            <class>org.zaproxy.zap.model.StandardParameterParser</class>
            <config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
        </urlparser>
        <postparser>
            <class>org.zaproxy.zap.model.StandardParameterParser</class>
            <config>{"kvps":"&amp;","kvs":"=","struct":[]}</config>
        </postparser>
        <authentication>
            <type>0</type>
            <strategy>EACH_RESP</strategy>
            <pollfreq>60</pollfreq>
            <pollunits>REQUESTS</pollunits>
        </authentication>
        <forceduser>-1</forceduser>
        <session>
            <type>0</type>
        </session>
        <authorization>
            <type>0</type>
            <basic>
                <header/>
                <body/>
                <logic>AND</logic>
                <code>-1</code>
            </basic>
        </authorization>
    </context>
</configuration>

For the actual scan we will be running we shall simply create a new ASP.NET Core Web API using the default Visual Studio template and tunnel this over the public internet from a development machine so our demonstration Azure Pipeline can access it without the need to deploy somewhere. The scan can be done against an OpenAPI, SOAP or GraphQL spec, in our case the Web API emits a Swagger OpenAPI spec that we will point to.

The most important being the incregexes which indicates the URL https://c1sjmvgw-7023.uks1.devtunnels.ms/.* This URL will be what we will tunnel our sample API via for scanning over the public internet.

⚠️ This is for demonstration purposes and to eventually allow Azure Pipelines to access an API that is on a local development machine.

On running this we will be able to see any scan issues in the console. For example, we may see something like this

WARN-NEW: Cookie No HttpOnly Flag [10010] x 2
	https://c1sjmvgw-7023.uks1.devtunnels.ms/WeatherForecast (200 OK)
	***/v1/swagger.json (200 OK)

You can find full usage instructions here).

Azure Pipelines

We can then proceed to use the same approach we have seen above, in an Azure Pipeline.

🖱️ Click to expand

name: $(Year:yyyy).$(Month)$(rev:.r)-owasp-zap

trigger:
  - none

parameters:
  - name: defaultLevelForAdditionalRules
    displayName: Default rules not in the config file to
    type: string
    default: WARN
    values:
      - PASS
      - IGNORE
      - INFO
      - WARN
      - FAIL
  - name: minimumLevel
    displayName: Minimum level to show in report
    type: string
    default: INFO
    values:
      - PASS
      - IGNORE
      - INFO
      - WARN
      - FAIL
variables:
  - group: [Group Name]
  - name: project
    value: "[Project Name]"

# decide on a schedule or trigger for the scan
# schedules:
#     - cron: "0 0 1 * *"
#       displayName: First day of each month midnight
#       branches:
#           include:
#               - master

stages:
  - stage: apiScan
    displayName: Run OWASP ZAP - API Scan
    jobs:
      - job: apiScanJob
        strategy:
          matrix:
            v1:
              apiSuffix: "v1"
            # v2:
            #   apiSuffix: 'v2'
        displayName: API Scan
        steps:
          - task: CopyFiles@2
            inputs:
              SourceFolder: "zap-config"
              Contents: "**"
              TargetFolder: "./"
              OverWrite: true


          - bash: |
              chmod -R 777 ./
              docker run \
              -v $(pwd):/zap/wrk/:rw \
              -t ghcr.io/zaproxy/zaproxy:stable zap-api-scan.py \
              -t '$(ZapApiTarget)/$(apiSuffix)/swagger.json' \
              -r report.html \
              -j \
              -I \
              -l ${{parameters.minimumLevel}} \
              -a \
              -c api-scan.conf \
              -n api.context \
              -f openapi \
              -i ${{parameters.defaultLevelForAdditionalRules}}
            displayName: Trigger docker scan
              # when working with API keys
              # -e ZAP_AUTH_HEADER_VALUE='$(ZAP_AUTH_HEADER_VALUE)' \
              # -e ZAP_AUTH_HEADER='$(ZAP_AUTH_HEADER)' \

          - task: PublishPipelineArtifact@1
            displayName: Publish Report
            condition: succeeded()
            inputs:
              targetPath: 'report.html'
              artifactName: 'artifact-zap-api-scan-report'

          - task: Bash@3
            inputs:
              targetType: "inline"
              script: |
                cat report.html | tr -d '\n' | jq --arg to "$(ZapRunRecipients)" \
                                                    --arg from "$(ZapSenderEmail)" \
                                                    --slurp --raw-input \
                                                    '{
                  personalizations: [
                    {
                      to: [
                        {
                          email: $to
                        }
                      ]
                    }
                  ],
                  from: {
                    email: $from
                  },
                  subject: "OWASP ZAP - API Scan Report",
                  content: [
                    {
                      type: "text/html",
                      value: .
                    }
                  ]
                }' > payload.json

                curl --request POST \
                  --url https://api.sendgrid.com/v3/mail/send \
                  --header 'Authorization: Bearer $(SendGridApiKey)' \
                  --header 'Content-Type: application/json' \
                  --data-binary "@payload.json"

This pipeline makes use of Send Grid via curl to send the report as an email. You can edit this step to your requirements. The report file is also available as a published artefact after the pipeline run.

Conclusion

This article barely scratches the surface on what you can do with OWASP ZAP. A good starting point would be to play with the desktop application and go through the documentation to see what ZAP can do and automate that. It will mostly be about exporting your configuration so you can consume it in the docker containers.

This article specifically covered API scanning, but you can use it to scan any web app, including running aggressive attacks to see how resilient the penetration measures are.

🛑Think twice before considering running OWASP ZAP on an application in a production environment as you can cause real damage.

🛑 NEVER run OWASP ZAP against a site that you do not own without authorization as that can constitute hacking and is unethical.

✅ DO keep the ethos of Playwright in mind. The tool is simple and does two things and two things well, perform actions and assert state against expectations. Unlike a lot of other traditional tools, having to wait for things prior to actions or asserts is not a thing and when done is usually a smell of something else being wrong.

These design choices allow Playwright users to forget about flaky timeouts and racy checks in their tests altogether.

✅ DO familiarize yourself with useful commands

# run all tests headless
npx playwright test

# run all tests in debug mode
npx playwright test --debug

# run all tests with parallelization disabled
npx playwright test --workers=1

# run tests with fail-fast
npx playwright test --max-failures=10

# run all tests with tracing on
# this should always be used on CI runs
npx playwright test --trace on

# run a specific test by file name
npx playwright test [test-file-name].ts

# run a specific test by test title
npx playwright test -g [target title]

# run tests that contain certain string in the file name
npx playwright test [target-string] # target strings can be chained

# run specific tests by  directory
npx playwright test test/[directory-name] # directories can be chanined

# run all tests in headed mode
npx playwright test --headed

# run all tests in a specific browser
npx playwright test --project webkit # --project can be chained

# view last run test report
npx playwright show-report

# run all test with the UI
npx playwright test --ui

# run the code gen to generate tests off a browser interaction
npx playwright codegen [optional-url]

# get help 🆘
npx playwright test --help

✅ DO familiarize yourself with the Playwright tracing and trace viewer.

✅ DO familiarize yourself with the various test annotations

✅ DO consider targeting the latest LTS node version at the point of creating a new Playwright project. This ensures that you start of with the latest guaranteed support with the widest compatibility and stability.

Consider using NVM or an alternative, to manage your node versions especially if you are likely to work on multiple things that may end up having different node version requirements.

✅ DO make use of the official Playwright VS Code extension for an improved experience.

The VS Code test runner runs your tests on the default browser of Chrome. To run on other/multiple browsers click the play button’s dropdown and choose another profile or modify the default profile by clicking Select Default Profile and select the browsers you wish to run your tests on.

✅ DO keep your playwright and related dependencies up to date. By updating regularly you keep up with the new features, continued improvements in browser support, bug fixes, compatibility with new features etc. Incrementally updating tends to be much easier than a big bang when you finally need a specific version, especially when dealing with breaking changes.

npm install -D @playwright/test@latest

# Also download new browser binaries and their dependencies:
npx playwright install --with-deps

# check the version
npx playwright --version

Consider npm-outdated as a tool to manage your dependencies.

Consider using the Version Lens extension in VS Code to check for outdated packages and update them on the click of a button.

✅ DO make use of the page object model approach to separate the management of a single page’s logic into one source of truth for a clean code structure that encourages the single responsibility making the maintenance of a large test suite over time easier.

This generally makes your tests easier to read at the top level, example:

test('given valid details, creating a new credit application from scratch should succeed', async ({
  page,
}) => {
  const businessDetailsPage = new BusinessDetailsPage(page);
  const businessVerificationPage = new BusinessVerificationPage(page);
  const documentsPage = new DocumentsPage(page);

  const businessDetails = new BusinessDetailsBuilder().thatIsValid().build();

  const businessVerification = new BusinessVerificationBuilder().thatIsValid().build();

  const documents = new DocumentsBuilder().thatIsValid().build();

  // submit business details
  await businessDetailsPage.goto();
  await businessDetailsPage.assertInitialState();
  await businessDetailsPage.fill(businessDetails);
  await businessDetailsPage.submit();

  // submit business verification details
  await businessVerificationPage.assertInitialState();
  await businessVerificationPage.fill(businessVerification);
  await businessVerificationPage.submitManualUpload();

  // upload documents
  await documentsPage.assertInitialState();
  await documentsPage.fill(documents);
  await documentsPage.submit();
});

A page object model class is generally just a class that accepts a page (or more) and encapsulates any logic about that page like locators, assertions, actions etc. Here is a sample skeleton

export class SamplePage {
  public readonly page: Page;
  public readonly doneButton: Locator;

  public constructor(page: Page) {
    this.page = page;
    this.doneButton = this.page.getByRole('button', { name: 'Done' });
  }

  public async assertInitialState() {}

  public async goto() {
    await this.page.goto('/');
  }

  public async submit() {
    await this.doneButton.click();
  }
}

✅ DO adopt fakers for creating random yet realistic test data. For TypeScript tests, the library Faker JS is an excellent choice for this.

✅ DO adopt a builder model to separate the creation of test data from everything as with one source of truth that allows for a test suite to remain maintainable. Sample:

export class LandLordBuilder extends Builder<LandLord> {
  constructor() {
    super(LandLord);
  }

  public override thatIsValid(): this {
    this.with((x) => x.fullName, faker.person.fullName());
    this.with((x) => x.cellphoneNumber, `0${faker.string.numeric(9)}`);
    this.with((x) => x.emailAddress, faker.internet.email());
    this.with((x) => x.leaseStartDate, faker.date.past());
    this.with((x) => x.leaseEndDate, faker.date.future());
    return this;
  }
}

# Example usage
const landlord = new LandLordBuilder()
  .thatIsValid()
  .build();

This example relies on the following base implementation.

export interface BuilderBase<T> {
  with<TProp>(propertyExpr: (obj: T) => TProp, value: TProp): this;

  withBuilder<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp,
    value: TBuilder,
  ): this;

  withMany<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp[],
    values: TBuilder[],
  ): this;

  ignoreProperty<TProp>(propertyExpr: (obj: T) => TProp): this;

  thatIsValid(): this;

  build(): T;
}

export abstract class Builder<T> implements BuilderBase<T> {
  private _properties: { [key: string]: unknown } = {};

  public constructor(private readonly typeConstructor: new () => T) {}

  public with<TProp>(propertyExpr: (obj: T) => TProp, value: TProp): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = value;
    return this;
  }

  public withBuilder<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp,
    value: TBuilder,
  ): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = value.build();
    return this;
  }

  public withMany<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp[],
    values: TBuilder[],
  ): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = values.map(v => v.build());
    return this;
  }

  public abstract thatIsValid(): this;

  public ignoreProperty<TProp>(propertyExpr: (obj: T) => TProp): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    delete this._properties[propertyName];
    return this;
  }

  public build(): T {
    const instance = this.createInstance();

    for (const key in this._properties) {
      instance[key] = this._properties[key];
    }

    return instance;
  }

  protected createInstance(): T {
    return new this.typeConstructor();
  }

  private extractPropertyName<TProp>(expr: (obj: T) => TProp): string {
    const parts = expr.toString().match(/\.([^.]+)/);
    if (!parts) {
      throw new Error("Couldn't extract property name.");
    }
    return parts[1];
  }
}

export class DynamicBuilder<T> extends Builder<T> {
  public constructor(private readonly typeCtor: new () => T) {
    super(typeCtor);
  }

  public thatIsValid(): this {
    throw new Error('Method not implemented.');
  }
}

✅ DO consider configuring a base URL for your tests so this can be defined in one place and reused across your tests.

export default defineConfig({
  use: {
    baseURL: 'http://127.0.0.1:3000',
  },
});

✅ DO consider reading any application configuration, regardless of source (environment variables, key vault secrets etc), in a central run once fixture.

Example. This will pull values from a .env file using dotenv and pull user secrets from an Azure Key Vault using a getSecret utility method.

import { test as base } from '@playwright/test';
import dotenv from 'dotenv';
import { env } from 'process';

import { getSecret } from '../helpers/secrets';

type ConfigurationFixture = {
  configuration: Configuration;
};

export const test = base.extend<ConfigurationFixture>({
  configuration: async ({}, use) => {
    dotenv.config();

    const secrets: Configuration = {
      userPassword: await getSecret('user-password'),
      imitateEmailAuth: await getSecret('websocket-auth'),
      userEmail: env.UserEmail as string,
    };
    await use(secrets);
  },
});

getSecret utility

import { DefaultAzureCredential } from '@azure/identity';
import { SecretClient } from '@azure/keyvault-secrets';

const credential = new DefaultAzureCredential();
const vaultName = '[vault name]';
const url = `https://${vaultName}.vault.azure.net`;
const client = new SecretClient(url, credential);

/**
 * Retrieves a secret from Azure Key Vault based on the given secret name.
 * @param name The name of the secret to retrieve.
 * @returns The value of the secret as a string.
 * @throws Will throw an error if unable to retrieve the secret.
 */
export const getSecret = async (name: string) => {
  try {
    const secret = await client.getSecret(name);
    return secret.value as string;
  } catch (error) {
    console.error(error);
    throw error;
  }
};

✅ DO write tests that assert the visible behaviour of your application and not on the inner workings. This will make your tests more resilient to changes in the underlying implementation. Examples of good things to check

Elements on the page and their state i.e. what is visible, what is enabled etc.
The URL of the page
The title of the page
A screenshot of the page

Examples of things that may be less resilient to changes

The exact structure of the DOM
The CSS classes used on elements

✅ DO critically think through and configure the desired browsers and devices to run your tests on. This will ensure that your tests are running on the platforms that are most important to you and will help you catch issues early.

export default defineConfig({
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    } /* Test against mobile viewports. */,
    {
      name: 'Mobile Chrome',
      use: { ...devices['Pixel 5'] },
    },
    {
      name: 'Mobile Safari',
      use: { ...devices['iPhone 12'] },
    } /* Test against branded browsers. */,
    {
      name: 'Microsoft Edge',
      use: { ...devices['Desktop Edge'], channel: 'msedge' },
    },
    {
      name: 'Google Chrome',
      use: { ...devices['Desktop Chrome'], channel: 'chrome' },
    },
  ],
});

✅ DO fast track your process by leveraging the Playwright test generator to create tests for you. This will give you a good starting point and you can then modify the tests to suit your needs.

npx playwright codegen [optional-url]

You can also generate tests using emulation to generate a test for a specific viewport, device, color scheme, as well as emulate the geolocation, language or timezone.

✅ DO enable retries on CI to make your tests more resilient to flakiness.

✅ DO enable tracing to help diagnose failing tests in CI.

Traces and debugging can also be leveraged locally to diagnose issues with your tests.

✅ DO make use of test hooks (test.describe, test.beforeEach, test.afterEach, test.beforeAll, test.afterAll) for grouping and granular tests to make sure each test is specific and focused. This will make it easier to diagnose failures and will make your tests more resilient to changes in the application.

✅ DO keep up with the [Playwright release notes](Release notes | Playwright) as a nice way to see what new APIs are being released.

✅ DO setup linting for your playwright tests with the [ESLint plugin for Playwright.](eslint-plugin-playwright - npm (npmjs.com)). Enable caching with your linting for faster subsequent runs.

npm init @eslint/config@latest

Sample .eslintrc.json config

{
  "parser": "@typescript-eslint/parser",
  "parserOptions": {
    "ecmaVersion": 12,
    "sourceType": "module"
  },
  "plugins": ["@typescript-eslint"],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
    "plugin:playwright/recommended"
  ],
  "rules": {},
  "env": {
    "browser": true,
    "es2021": true
  }
}

# in package.json
  "scripts": {
    "lint": "eslint \"**/*.ts\" --cache"
  },
  "lint-staged": {
    "*.ts": "eslint --cache"
  }

✅ DO consider lint staged as a way to be able to run linting only for the files that you have just changed. npm install -D lint-staged

# in package.json
  "lint-staged": {
    "*.ts": "eslint --cache"
  }

⚠️ DO give close attention to the need for any waits or sleeps in your tests. This is a common source of flakiness and should in most cases not be necessary given how Playwright is designed to work.

Playwright assertions are designed in a way that they describe the expectations that need to be eventually met.

Playwright automatically waits for the wide range of actionability checks to pass before performing each action.

✅ DO make use of an .env file to have a single source of truth for environment values. You can conveniently load env files globally by adding this into your playwright.config.ts

import { defineConfig, devices } from '@playwright/test';
import dotenv from 'dotenv';

// Read from default ".env" file.
dotenv.config();

✅ DO write custom assertions to simplify assertions you use in combination or common things you assert that require custom logic.

import type { Locator } from '@playwright/test';
import { expect as baseExpect } from '@playwright/test';

export const expect = baseExpect.extend({
    async toBeVisibleAndEmpty(locator: Locator) {
        const assertionName = 'toBeVisibleAndEmpty';
        const isVisible = await locator.isVisible();
        if (!isVisible) {
            return {
                message: () => `expected locator to be visible`,
                pass: false,
            };
        }

        const content = await locator.textContent();
        const isEmpty = content?.trim() === '';
        if (!isEmpty) {
            return {
                message: () => `expected locator to be empty, but got "${content}"`,
                pass: false,
            };
        }

        return {
            message: () => `locator is visible and empty`,
            pass: true,
            name: assertionName
        };
    },
...
});

🛑 DO not hard code secrets in tests or any artifact that will be committed to your repository. Consider making use of an Azure Key Vault instead.

Sample helper

You will need to install the following package

npm install -D @azure/keyvault-secrets
npm install -D @azure/identity

You will need to have at minimum the Azure IAM Key Vaults Secret User role assigned.

Automation Plan Checklist

Here is a list of questions you can ask yourself about what you want to automate so you can have a clear plan of action before you start. If you roll with the punches, you will be a slave to so many things that you will have much more control by planning and deciding.

Report on team activity for DevOps Pull Requests

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Towfiqu barbhuiya on Unsplash

Report on team activity for DevOps Pull Requests

The following PowerShell script, GetPullRequestDetails, is designed to retrieve and analyze pull request (PR) details from an Azure DevOps repository. By providing insights into the number of PRs created and reviewed by specified users, it aids in understanding team contributions over time and encouraging balance.

The function GetPullRequestDetailsXX takes four parameters:

$org: The Azure DevOps organization name.
$project: The project name within the organization.
$repo: The repository name.
$token: The Personal Access Token (PAT) for authentication.

Creating a Personal Access Token (PAT)

To use this script, a PAT with appropriate permissions is required. Follow these steps to create a PAT:

Navigate to Azure DevOps:
- Go to your Azure DevOps organization URL (e.g., https://dev.azure.com/{your_organization}).
Access User Settings:
- Click on your profile picture in the upper-right corner and select “Personal access tokens”.
Create a New Token:
- Click on “New Token”.
Set Token Details:
- Name your token appropriately (e.g., “Pull Request Analysis Token”).
- Set the expiration period as required.
Select Scopes:
- Under “Scopes”, select:
  - Code (Read) for reading repository data.
  - Code (Status) for accessing PR status information.
Create and Copy Token:
- Click “Create” and copy the generated token. Store it securely as it won’t be shown again.

function GetPullRequestDetails {
    param(
        [string]$org,
        [string]$project,
        [string]$repo,
        [string]$token
    )

    $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "test",$token)))
    # comma separated list of valid users, i.e @('one', 'two', 'three')
    $validUsers = @([USERS])
    $allPRs = @()
    $i = 0
    do {
        $uri = "https://dev.azure.com/$org/$project/_apis/git/repositories/$repo/pullRequests?api-version=5.0&`$top=100&`$skip=$i&searchCriteria.status=all"
        $i += 100
        Write-Output $uri
        $result = Invoke-RestMethod -Method Get -Uri $Uri -ContentType "application/json" -Headers @{Authorization=("Basic {0}" -f $base64AuthInfo)}
        $allPRs += $result.value
        if ($result.count -lt 100) {
            break
        }
    } while ($true)
    $endDate = (Get-Date).Date
    $startDate = $endDate.AddDays(-30)

    # Initialize hash tables for counts
    $createdPRsAllTime = @{}
    $createdPRsLast30Days = @{}
    $reviewedPRsAllTime = @{}
    $reviewedPRsLast30Days = @{}

    foreach ($user in $validUsers) {
        $createdPRsAllTime[$user] = 0
        $createdPRsLast30Days[$user] = 0
        $reviewedPRsAllTime[$user] = 0
        $reviewedPRsLast30Days[$user] = 0
    }

    foreach ($pr in $allPRs) {
        $creator = $pr.createdBy.displayName
        $reviewers = $pr.reviewers | Where-Object { $_.displayName -in $validUsers } | ForEach-Object { $_.displayName }
        $prCreationDate = [datetime]::Parse($pr.creationDate)

        # Handle creation
        if ($validUsers -contains $creator) {
            $createdPRsAllTime[$creator] += 1
            if ($prCreationDate -ge $startDate -and $prCreationDate -le $endDate) {
                $createdPRsLast30Days[$creator] += 1
            }
        }

        # Handle review
        foreach ($reviewer in $reviewers) {
            $reviewedPRsAllTime[$reviewer] += 1
            if ($prCreationDate -ge $startDate -and $prCreationDate -le $endDate) {
                $reviewedPRsLast30Days[$reviewer] += 1
            }
        }
    }

    # Output tables sorted by value in descending order
    Write-Output "All-time Created PRs by user:"
    $createdPRsAllTime.GetEnumerator() | Sort-Object Value -Descending | Format-Table -AutoSize

    Write-Output "Last 30 days Created PRs by user:"
    $createdPRsLast30Days.GetEnumerator() | Sort-Object Value -Descending | Format-Table -AutoSize

    Write-Output "All-time Reviewed PRs by user:"
    $reviewedPRsAllTime.GetEnumerator() | Sort-Object Value -Descending | Format-Table -AutoSize

    Write-Output "Last 30 days Reviewed PRs by user:"
    $reviewedPRsLast30Days.GetEnumerator() | Sort-Object Value -Descending | Format-Table -AutoSize

    Write-Output "Finish. Total Pull Request count: $($allPRs.Count)"
}


GetPullRequestDetails -org "[org name]" -project "[PROJECT NAME]" -repo "[REPO NAME]" -token "[PAT]"

Key Operations

Authentication:
- The script encodes the PAT into a Base64 string for secure authorization in API requests.
Fetching Pull Requests:
- The script constructs a URI to request PR data from Azure DevOps, iterating through pages of results (up to 100 PRs per page) until all PRs are retrieved.
Filtering and Counting PRs:
- The script initializes hash tables to count PRs created and reviewed by specified users, both all-time and within the last 30 days.
- It iterates through all PRs, updating counts based on PR creation and review activities.
Output:
- The script outputs sorted tables of PR counts for each user, distinguishing between all-time and the last 30 days.

Playwright best practices

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Igor Omilaev on Unsplash

Playwright best practices

These design choices allow Playwright users to forget about flaky timeouts and racy checks in their tests altogether.

✅ DO familiarize yourself with useful commands

# run all tests headless
npx playwright test

# run all tests in debug mode
npx playwright test --debug

# run all tests with parallelization disabled
npx playwright test --workers=1

# run tests with fail-fast
npx playwright test --max-failures=10

# run all tests with tracing on
# this should always be used on CI runs
npx playwright test --trace on

# run a specific test by file name
npx playwright test [test-file-name].ts

# run a specific test by test title
npx playwright test -g [target title]

# run tests that contain certain string in the file name
npx playwright test [target-string] # target strings can be chained

# run specific tests by  directory
npx playwright test test/[directory-name] # directories can be chanined

# run all tests in headed mode
npx playwright test --headed

# run all tests in a specific browser
npx playwright test --project webkit # --project can be chained

# view last run test report
npx playwright show-report

# run all test with the UI
npx playwright test --ui

# run the code gen to generate tests off a browser interaction
npx playwright codegen [optional-url]

# get help 🆘
npx playwright test --help

✅ DO familiarize yourself with the Playwright tracing and trace viewer.

✅ DO familiarize yourself with the various test annotations

Consider using NVM or an alternative, to manage your node versions especially if you are likely to work on multiple things that may end up having different node version requirements.

✅ DO make use of the official Playwright VS Code extension for an improved experience.

The VS Code test runner runs your tests on the default browser of Chrome. To run on other/multiple browsers click the play button’s dropdown and choose another profile or modify the default profile by clicking Select Default Profile and select the browsers you wish to run your tests on.

npm install -D @playwright/test@latest

# Also download new browser binaries and their dependencies:
npx playwright install --with-deps

# check the version
npx playwright --version

Consider npm-outdated as a tool to manage your dependencies.

Consider using the Version Lens extension in VS Code to check for outdated packages and update them on the click of a button.

This generally makes your tests easier to read at the top level, example:

test('given valid details, creating a new credit application from scratch should succeed', async ({
  page,
}) => {
  const businessDetailsPage = new BusinessDetailsPage(page);
  const businessVerificationPage = new BusinessVerificationPage(page);
  const documentsPage = new DocumentsPage(page);

  const businessDetails = new BusinessDetailsBuilder().thatIsValid().build();

  const businessVerification = new BusinessVerificationBuilder().thatIsValid().build();

  const documents = new DocumentsBuilder().thatIsValid().build();

  // submit business details
  await businessDetailsPage.goto();
  await businessDetailsPage.assertInitialState();
  await businessDetailsPage.fill(businessDetails);
  await businessDetailsPage.submit();

  // submit business verification details
  await businessVerificationPage.assertInitialState();
  await businessVerificationPage.fill(businessVerification);
  await businessVerificationPage.submitManualUpload();

  // upload documents
  await documentsPage.assertInitialState();
  await documentsPage.fill(documents);
  await documentsPage.submit();
});

A page object model class is generally just a class that accepts a page (or more) and encapsulates any logic about that page like locators, assertions, actions etc. Here is a sample skeleton

export class SamplePage {
  public readonly page: Page;
  public readonly doneButton: Locator;

  public constructor(page: Page) {
    this.page = page;
    this.doneButton = this.page.getByRole('button', { name: 'Done' });
  }

  public async assertInitialState() {}

  public async goto() {
    await this.page.goto('/');
  }

  public async submit() {
    await this.doneButton.click();
  }
}

✅ DO adopt fakers for creating random yet realistic test data. For TypeScript tests, the library Faker JS is an excellent choice for this.

✅ DO adopt a builder model to separate the creation of test data from everything as with one source of truth that allows for a test suite to remain maintainable. Sample:

export class LandLordBuilder extends Builder<LandLord> {
  constructor() {
    super(LandLord);
  }

  public override thatIsValid(): this {
    this.with((x) => x.fullName, faker.person.fullName());
    this.with((x) => x.cellphoneNumber, `0${faker.string.numeric(9)}`);
    this.with((x) => x.emailAddress, faker.internet.email());
    this.with((x) => x.leaseStartDate, faker.date.past());
    this.with((x) => x.leaseEndDate, faker.date.future());
    return this;
  }
}

# Example usage
const landlord = new LandLordBuilder()
  .thatIsValid()
  .build();

This example relies on the following base implementation.

export interface BuilderBase<T> {
  with<TProp>(propertyExpr: (obj: T) => TProp, value: TProp): this;

  withBuilder<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp,
    value: TBuilder,
  ): this;

  withMany<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp[],
    values: TBuilder[],
  ): this;

  ignoreProperty<TProp>(propertyExpr: (obj: T) => TProp): this;

  thatIsValid(): this;

  build(): T;
}

export abstract class Builder<T> implements BuilderBase<T> {
  private _properties: { [key: string]: unknown } = {};

  public constructor(private readonly typeConstructor: new () => T) {}

  public with<TProp>(propertyExpr: (obj: T) => TProp, value: TProp): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = value;
    return this;
  }

  public withBuilder<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp,
    value: TBuilder,
  ): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = value.build();
    return this;
  }

  public withMany<TProp, TBuilder extends BuilderBase<unknown>>(
    propertyExpr: (obj: T) => TProp[],
    values: TBuilder[],
  ): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    this._properties[propertyName] = values.map(v => v.build());
    return this;
  }

  public abstract thatIsValid(): this;

  public ignoreProperty<TProp>(propertyExpr: (obj: T) => TProp): this {
    const propertyName = this.extractPropertyName(propertyExpr);
    delete this._properties[propertyName];
    return this;
  }

  public build(): T {
    const instance = this.createInstance();

    for (const key in this._properties) {
      instance[key] = this._properties[key];
    }

    return instance;
  }

  protected createInstance(): T {
    return new this.typeConstructor();
  }

  private extractPropertyName<TProp>(expr: (obj: T) => TProp): string {
    const parts = expr.toString().match(/\.([^.]+)/);
    if (!parts) {
      throw new Error("Couldn't extract property name.");
    }
    return parts[1];
  }
}

export class DynamicBuilder<T> extends Builder<T> {
  public constructor(private readonly typeCtor: new () => T) {
    super(typeCtor);
  }

  public thatIsValid(): this {
    throw new Error('Method not implemented.');
  }
}

✅ DO consider configuring a base URL for your tests so this can be defined in one place and reused across your tests.

export default defineConfig({
  use: {
    baseURL: 'http://127.0.0.1:3000',
  },
});

✅ DO consider reading any application configuration, regardless of source (environment variables, key vault secrets etc), in a central run once fixture.

Example. This will pull values from a .env file using dotenv and pull user secrets from an Azure Key Vault using a getSecret utility method.

import { test as base } from '@playwright/test';
import dotenv from 'dotenv';
import { env } from 'process';

import { getSecret } from '../helpers/secrets';

type ConfigurationFixture = {
  configuration: Configuration;
};

export const test = base.extend<ConfigurationFixture>({
  configuration: async ({}, use) => {
    dotenv.config();

    const secrets: Configuration = {
      userPassword: await getSecret('user-password'),
      imitateEmailAuth: await getSecret('websocket-auth'),
      userEmail: env.UserEmail as string,
    };
    await use(secrets);
  },
});

getSecret utility

import { DefaultAzureCredential } from '@azure/identity';
import { SecretClient } from '@azure/keyvault-secrets';

const credential = new DefaultAzureCredential();
const vaultName = '[vault name]';
const url = `https://${vaultName}.vault.azure.net`;
const client = new SecretClient(url, credential);

/**
 * Retrieves a secret from Azure Key Vault based on the given secret name.
 * @param name The name of the secret to retrieve.
 * @returns The value of the secret as a string.
 * @throws Will throw an error if unable to retrieve the secret.
 */
export const getSecret = async (name: string) => {
  try {
    const secret = await client.getSecret(name);
    return secret.value as string;
  } catch (error) {
    console.error(error);
    throw error;
  }
};

Elements on the page and their state i.e. what is visible, what is enabled etc.
The URL of the page
The title of the page
A screenshot of the page

Examples of things that may be less resilient to changes

The exact structure of the DOM
The CSS classes used on elements

export default defineConfig({
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] },
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] },
    } /* Test against mobile viewports. */,
    {
      name: 'Mobile Chrome',
      use: { ...devices['Pixel 5'] },
    },
    {
      name: 'Mobile Safari',
      use: { ...devices['iPhone 12'] },
    } /* Test against branded browsers. */,
    {
      name: 'Microsoft Edge',
      use: { ...devices['Desktop Edge'], channel: 'msedge' },
    },
    {
      name: 'Google Chrome',
      use: { ...devices['Desktop Chrome'], channel: 'chrome' },
    },
  ],
});

✅ DO fast track your process by leveraging the Playwright test generator to create tests for you. This will give you a good starting point and you can then modify the tests to suit your needs.

npx playwright codegen [optional-url]

You can also generate tests using emulation to generate a test for a specific viewport, device, color scheme, as well as emulate the geolocation, language or timezone.

✅ DO enable retries on CI to make your tests more resilient to flakiness.

✅ DO enable tracing to help diagnose failing tests in CI.

Traces and debugging can also be leveraged locally to diagnose issues with your tests.

✅ DO keep up with the [Playwright release notes](Release notes | Playwright) as a nice way to see what new APIs are being released.

✅ DO setup linting for your playwright tests with the [ESLint plugin for Playwright.](eslint-plugin-playwright - npm (npmjs.com)). Enable caching with your linting for faster subsequent runs.

npm init @eslint/config@latest

Sample .eslintrc.json config

{
  "parser": "@typescript-eslint/parser",
  "parserOptions": {
    "ecmaVersion": 12,
    "sourceType": "module"
  },
  "plugins": ["@typescript-eslint"],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/recommended",
    "plugin:playwright/recommended"
  ],
  "rules": {},
  "env": {
    "browser": true,
    "es2021": true
  }
}

# in package.json
  "scripts": {
    "lint": "eslint \"**/*.ts\" --cache"
  },
  "lint-staged": {
    "*.ts": "eslint --cache"
  }

✅ DO consider lint staged as a way to be able to run linting only for the files that you have just changed. npm install -D lint-staged

# in package.json
  "lint-staged": {
    "*.ts": "eslint --cache"
  }

Playwright assertions are designed in a way that they describe the expectations that need to be eventually met.

Playwright automatically waits for the wide range of actionability checks to pass before performing each action.

✅ DO make use of an .env file to have a single source of truth for environment values. You can conveniently load env files globally by adding this into your playwright.config.ts

import { defineConfig, devices } from '@playwright/test';
import dotenv from 'dotenv';

// Read from default ".env" file.
dotenv.config();

✅ DO write custom assertions to simplify assertions you use in combination or common things you assert that require custom logic.

import type { Locator } from '@playwright/test';
import { expect as baseExpect } from '@playwright/test';

export const expect = baseExpect.extend({
    async toBeVisibleAndEmpty(locator: Locator) {
        const assertionName = 'toBeVisibleAndEmpty';
        const isVisible = await locator.isVisible();
        if (!isVisible) {
            return {
                message: () => `expected locator to be visible`,
                pass: false,
            };
        }

        const content = await locator.textContent();
        const isEmpty = content?.trim() === '';
        if (!isEmpty) {
            return {
                message: () => `expected locator to be empty, but got "${content}"`,
                pass: false,
            };
        }

        return {
            message: () => `locator is visible and empty`,
            pass: true,
            name: assertionName
        };
    },
...
});

🛑 DO not hard code secrets in tests or any artifact that will be committed to your repository. Consider making use of an Azure Key Vault instead.

Sample helper

You will need to install the following package

npm install -D @azure/keyvault-secrets
npm install -D @azure/identity

You will need to have at minimum the Azure IAM Key Vaults Secret User role assigned.

Automation Plan Checklist

Writing effective Postman tests

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Audrey Mari on Unsplash

Writing Effective Postman Tests

✅ DO maintain a hierarchy of your tests following the same structure as the URLs of the API being tested and write clear, descriptive names for each test case. This helps ensure that anyone reviewing the tests can understand their purpose at a glance.

POST /api/user/ GET /api/users/1, PUT /api/users/1, DELETE /api/users/1

POST /api/products/, GET /api/products/1, PUT /api/products/1, DELETE /api/products/1

May be structured as:

users

create users

get users

update users

delete users

products

create products

get products

update products

delete products

This structure helps in organizing tests and makes it easier to locate specific test cases.

✅ DO group related tests into collections for easier management and execution. This also helps in organizing tests by functional area or API endpoint.

✅ DO use environment variables to store and manage test data, such as API keys and server URLs. This promotes reusability and keeps sensitive data secure.

Ensure that only the current values ever have the secret values set and not the initial value. This prevents accidental exposure of sensitive information as the collection is shared or exported.

✅ DO incorporate both positive and negative test cases to ensure your API behaves correctly under various conditions. When asserting response bodies, consider both expected and unexpected values to have good coverage. For example, if you expect a response body to include validation messages for an expected failed input, also check that the response body does not include validation messages for an expected successful input or anything additional.

pm.test("Response body has the expected nested errors", function () {
    const responseBody = pm.response.json();

    const expectedErrors = {
        "Company.Contacts": ["'Contacts' must not be empty."],
        "Company.Addresses": ["'Addresses' must not be empty."],
        "Company.CompanyName": ["'Company Name' must not be empty."],
    };

    for (const [key, value] of Object.entries(expectedErrors)) {
        pm.expect(responseBody.errors).to.have.property(key);
        pm.expect(responseBody.errors[key]).to.eql(value);
    }

    const actualErrorKeys = Object.keys(responseBody.errors);
    const expectedErrorKeys = Object.keys(expectedErrors);

    expectedErrorKeys.sort();
    actualErrorKeys.sort();

    pm.expect(actualErrorKeys).to.eql(expectedErrorKeys);
});

✅ DO use Postman’s pre-request scripts to set up any prerequisites your tests need. This can include generating test data or configuring the request in a specific way.

Pre-request scripts can be used to set up test theories such as running the same test multiple times with different data to ensure consistency and maintain a single test.

Here is a simple example that sets multiple statement types for use in a request

let value = pm.environment.get("statements-count") ?? 0;
value = value >= 4 ? 0: value;
pm.environment.set('statements-count', value)

const statementTypes = [
    { key: 'Month' },
    { key: 'Life' },
    { key: 'Casual' },
    { key: 'Epic' },
];

const statementType = statementTypes[value];
pm.environment.set("statement-type", statementType.key);

A test can then use this information and trigger new requests

pm.test("Status code is 204 No Content", function () {
    pm.response.to.have.status(204);
});

let counter = pm.environment.get("statements-count") || 0;

if (counter < 1) {
    counter++;
    pm.environment.set("statements-count", counter);
    postman.setNextRequest(pm.info.requestName);
} else {
    pm.environment.unset("statements-count");
}

✅ DO make assertions specific and meaningful. Avoid generic messages like “Test passed” or “Test failed”. Instead, clearly state what was expected versus what was observed. It is usually best to have granular tests, for example, if you want to assert the status code, the response body and the response header, this would be best done as three separate tests.

✅ DO organize tests logically within each request. Start with simpler tests (like status code checks) and move towards more complex assertions (like content validation) and ensure consistency across all tests.

✅ DO include infrastructure and security validations as part of your tests. Check for things like rate limiting, authentication, and authorization mechanisms to ensure the API behaves as expected.

✅ DO regularly review and update your tests to reflect changes in the API. Obsolete tests can lead to false positives or unnoticed issues.

✅ DO use chaining requests to simulate real-world use cases where multiple API calls depend on each other. Extract data from one request and use it in another to maintain test flow. For example, you can have a test that creates a new record and then another test that retrieves the record.

✅ DO share your collections with team members for collaborative testing and feedback. Use Postman Workspaces to facilitate teamwork.

Ensure that only the current values ever have the secret values set and not the initial value. This prevents accidental exposure of sensitive information as the collection is shared or exported.

Ensure that the workspace is set to private if the collection contains sensitive information and is only shared with team members who need access to the collection.

✅ DO integrate your Postman collections with CI/CD pipelines using Newman, Postman’s command-line collection runner, to automate your testing process.

✅ DO monitor API performance and response times in addition to functional correctness. Use Postman’s visualizer feature to represent response times graphically for easier analysis.

✅ DO consider edge cases and limit conditions in your tests. This helps in uncovering potential issues that might not be evident in typical use cases.

✅ DO document your testing strategy and share it with your team. Include guidelines on how to write, organize, and execute tests, ensuring consistency across the board.

✅ DO encourage peer reviews of test collections to foster knowledge sharing and improve test quality. Constructive feedback can lead to more robust and effective tests.

✅ DO utilize Postman’s reporting tools to generate insights on test runs. Newman has inbuilt reporters that can be configured with Azure Dev Ops Pipelines, among other CI/CD integrations. Share these reports with your team or stakeholders to inform them about the API’s health and behaviour.

✅ DO make use of JavaScript by following best practices and ensuring consistency in your scripts. Use comments to explain complex logic and make your code more readable. Avoid commenting on every line, however.

✅ DO thoroughly document each request within your collection. Provide a clear description of what the request does, including its purpose, any specific parameters it uses, and what kind of response it expects. Use Postman’s description field for this purpose.

Employ Markdown syntax in your request and folder descriptions for better readability. Postman supports Markdown, allowing you to format text, insert links, and even embed images to make your documentation more comprehensive and user-friendly.

✅ DO document any dependencies or external variables that your collection requires. If your collection depends on specific environment or global variables, mention these in the collection’s overview section along with instructions on how to set them up.

✅ DO use the collection’s overview section to provide a high-level description of the collection’s scope, usage instructions, and any other important information that users should know before they start using it.

✅ DO organize your collection logically, using folders to group related requests. This not only helps with navigation but also allows you to document sections of your API in a structured manner.

✅ DO use Postman’s built-in dynamic variables for common values such as timestamps, UUIDs, and random integers. This reduces the need for external data sources and simplifies your test setup.

This allows for unique, yet consistent and real-world data to be used in tests.

✅ DO consider collections that make use of test input data to automate the testing of APIs with various input combinations

✅ DO leverage Postbot to automatically generate tests for your API endpoints. This is especially useful when starting tests for a new API or when you want to quickly expand your test coverage.

Postbot will generate tests based on the responses it gets.

While this is very nifty and effective. Dumping your response data into Chat GPT is more effective as it will generate more meaningful tests and you can request for additional things to be generated.

Example interaction with Chat GPT

Given a response with a `207` multi status and the body below, generate multiple Postman tests to assert the status code, the expected response (given the complete response, match that with response json), that the response has a header `X-Correlation-ID` that is a valid GUID and that the `Strict-Transport-Security` header is not empty.

{
    "responses": [
        {
            "statusCode": 200,
            "message": "OK: Resource uploaded successfully.",
            "resourceId": "Bank Statement - valid pdf.pdf"
        },
        {
            "statusCode": 400,
            "message": "Bad Request: Invalid document provided. Should be one of (JSON, PDF, JPEG, PNG, TIFF, BMP, WEBP)",
            "resourceId": "Transaction Data - invalid extension.php"
        }
    ]
}

Response

pm.test("Status code is 207", function () {
    pm.response.to.have.status(207);
});

const expectedResponse = {
    "responses": [
        {
            "statusCode": 200,
            "message": "OK: Resource uploaded successfully.",
            "resourceId": "Bank Statement - valid pdf.pdf"
        },
        {
            "statusCode": 400,
            "message": "Bad Request: Invalid document provided. Should be one of (JSON, PDF, JPEG, PNG, TIFF, BMP, WEBP)",
            "resourceId": "Transaction Data - invalid extension.php"
        }
    ]
};

pm.test("Response body matches expected response", function () {
    pm.expect(pm.response.json()).to.eql(expectedResponse);
});

pm.test("X-Correlation-ID is present and is a valid UUID", function () {
    const uuidRegex = /^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i;
    const correlationId = pm.response.headers.get("X-Correlation-ID");
    pm.expect(correlationId).to.match(uuidRegex);
});

pm.test("Strict-Transport-Security header is not empty", function () {
    const hstsHeader = pm.response.headers.get("Strict-Transport-Security");
    pm.expect(hstsHeader).to.not.be.empty;
});

✅ DO use the Collection Runner’s iteration feature to run flaky tests multiple times. This approach helps in identifying non-deterministic behaviour by running the tests under the same conditions repeatedly.

✅ DO use pre-request scripts for scenarios where you would like to apply a given change for a test or collection of tests such as dynamically setting header values, overriding environment variables, or setting up data for the test.

⭐ DO use Postman’s pull request features to manage changes to collections. This allows team members to propose, review, and discuss changes directly within Postman, streamlining the collaboration process. Use the built-in version control system in Postman as your primary method for tracking changes to collections. avoiding the complexity and overhead of exporting collections to Git for version control whenever possible. While exporting to Git provides a universal way to track changes, it can disconnect the collection from the interactive testing and documentation features of Postman and the output is not user-friendly as the Postman interface for reviewing changes.

✅ ENSURE that all team members are familiar with how to create, review, and merge pull requests within Postman. Providing training or resources can help teams transition to using these features effectively.

⭐ DO leverage built-in libraries like moment, lodash and querystring to simplify your scripts and make them more readable. These libraries provide a wide range of utility functions that can help you write more efficient and maintainable code. You can also add external libraries if required.

⭐ DO consider Postman tests for more than just testing APIs. You can use Postman to do sanity checks on tools that have APIs. For example, you may have an application that integrates with a CRM where you expect certain data structures to be present in the CRM and in a specific manner for a successful integration. You can write tests to check for this and run this as part of your release sign-off process. Another example is a tedious manual process that can be automated using Postman such as logging into a CRM on each exploratory test to change specific data. You can write a test to do this and run it as part of your exploratory testing with test assertions to ensure the data is changed as expected if an API is available.

⚠️ AVOID hard-coding values that can change, such as tokens, user IDs, or resource URLs. This can lead to tests that fail when the underlying data changes. Instead, use variables and pre-request scripts to dynamically set these values and have a single source of truth.

⚠️ AVOID overusing global variables. While global variables are convenient, they can lead to unexpected behaviour if tests are run in a different order or in parallel sessions. Prefer environment variables scoped to specific collections or tests.

⚠️ AVOID* writing tests that depend on the execution order of other tests. Each test should be able to run independently. If you need to set up data or state, use pre-request scripts or the collection runner’s iteration setup.

⚠️ DO NOT overlook the importance of cleaning up after your tests. If your tests create or modify data, consider adding steps or scripts to revert these changes. This helps maintain a clean testing environment and prevents interference with subsequent tests.

Postman can also be used beyond just REST APIs. You can use it to test GraphQL APIs, SOAP APIs, and more.

Promotion deployment strategy Azure pipelines

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Anna Vi on Unsplash

Promotion deployment strategy pipelines

Are you considering implementing deployment pipelines to multiple environments and unsure about how to structure this? You could have a pipeline per environment and take advantage of templates) for reuse across the pipelines. You will however need to apply the right level of care to ensure that you deploy the right things across the environments. When QA signs off on a release for instance, are you sure you are deploying the right artifacts/branch as was in QA?

If you have to rebuild artifacts on promotion to another environment, then you may be doing it wrong as unless you go the extra mile, you cannot guarantee that building the same branch commit twice results in the same artifacts. What if the underlying agent has updated dependencies, what if you do not have version pinning in your application and new dependencies are pulled, what if…?

Leveraging static and idempotent artefacts like Docker may be a great option as you would have the same artifacts move across environments. Even with that though, if you have to run multiple pipelines, what if you accidentally choose the wrong version on promotion?

Would a single pipeline that progressively promotes to multiple environments be a potential answer? Much like how Octopus Deploy) works.

Convinced? Then you are in the right place. Let’s explore how Azure Pipelines can be used to implement deployment strategies based on environment promotion, using YAML configurations to manage workflows that traditionally relied on UI-based tools like Azure Deployments.

Environment promotion in CI/CD is the practice of moving software between multiple stages or environments, such as development, QA, UAT, and production. Each environment serves a different purpose and typically requires that certain criteria be met before software is promoted to the next stage. This strategy helps ensure that only well-tested and verified code makes its way to production.

Let’s take a look at a sample pipeline that has all of this

trigger:
  batch: true
  branches:
    include:
      - develop
      - master
      - release/*
      - hotfix/*

# This will deploy the latest of the following branches if any changes are available
# If you have merges happening to multiple at the same time, this may not work best for you
schedules:
  - cron: "0 0 * * *"
    displayName: Daily Midnight Deployment to QA
    branches:
      include:
        - develop
        - release/*
        - hotfix/*

parameters:
  - name: ignoreSchedule
    # Allows for deploying when run on demand to QA
    # Default is to only deploy to QA nightly (when scheduled), this allows an override
    displayName: Ignore scheduling rules
    default: false
    type: boolean

stages:
  - stage: publishArtifactsA
    displayName: Publish Artifacts A
    dependsOn: []
    jobs:
      - job: publish
        displayName: Publish Artifacts A
        # Ideally pull build tasks from a template to reuse with verification pipelines
        # Custom tasks that do actual packaging can then be added
        # - template: [location]/[template].yml
        steps:
          - script: echo "Publish Artifacts A"

  - stage: publishArtifactsB
    displayName: Publish Artifacts B
    dependsOn: []
    jobs:
      - job: publish
        displayName: Publish Artifacts B
        # Ideally pull build tasks from a template to reuse with verification pipelines
        # Custom tasks that do actual packaging can then be added
        # - template: [location]/[template].yml
        steps:
          - script: echo "Publish Artifacts B"

  - stage: deployToDev
    displayName: Deploy to Dev 🟢
    dependsOn: [publishArtifactsA, publishArtifactsB]
    # Check if skipping deployment to Dev environment is false (i.e., deployment to Dev is required)
    jobs:
      # Ideally pull build tasks from a template to reuse with verification pipelines
      # Custom tasks that do actual packaging can then be added
      # - template: [location]/[template].yml
      - deployment: deploy_application
        displayName: Deploy Application
        environment: dev-env
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploy to Dev"

  - stage: deployToQa
    displayName: Deploy to QA 🟠
    dependsOn: [deployToDev]
    # Deploy if Dev succeeded or was skipped (fast deploy to QA on demand)
    # Deployment to QA only on a schedule or on demand run with parameter set to true
    condition: |
      or(
        eq(variables['Build.Reason'], 'Schedule'),
        eq('${{ parameters.ignoreSchedule }}', true)
      )
    jobs:
      # Ideally pull build tasks from a template to reuse with verification pipelines
      # Custom tasks that do actual packaging can then be added
      # - template: [location]/[template].yml
      - deployment: deploy_application
        displayName: Deploy Application
        environment: qa-env
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploy to QA"

  - stage: deployToUat
    displayName: Deploy to UAT 🟠
    dependsOn: [deployToQa]
    # UAT has an approval gate
    condition: |
      or(
        eq(variables['Build.Reason'], 'Schedule'),
        eq('${{ parameters.ignoreSchedule }}', true)
      )
    jobs:
      # Ideally pull build tasks from a template to reuse with verification pipelines
      # Custom tasks that do actual packaging can then be added
      # - template: [location]/[template].yml
      - deployment: deploy_application
        displayName: Deploy Application
        environment: uat-env
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploy to UAT"

  - stage: deployToProd
    displayName: Deploy to Prod 🔴
    dependsOn: [deployToUat]
    condition: |
      or(
        eq(variables['Build.Reason'], 'Schedule'),
        eq('${{ parameters.ignoreSchedule }}', true)
      )
    jobs:
      # Ideally pull build tasks from a template to reuse with verification pipelines
      # Custom tasks that do actual packaging can then be added
      # - template: [location]/[template].yml
      - deployment: deploy_application
        displayName: Deploy Application
        environment: prod-env
        strategy:
          runOnce:
            deploy:
              steps:
                - script: echo "Deploy to Prod"

Multiple stages are defined in the pipeline, each corresponding to a target function/environment. Each stage can specify dependencies on previous stages, ensuring that deployments follow the correct order. Here’s a brief outline:

This can additionally be used with branch protection on the environment. Recommendations:

Loose approach for dev, no protection allowing any branch to be deployed, including ones that have not been integrated via Pull Requests
Allow only release/*, hotfix/*, master and develop into QA
Allow only release/* and hotfix/*, into UAT and Prod

This simple conditional logic in the sample supports complex deployment strategies and mimics the functionality of dedicated deployment tools like Octopus Deploy, which focuses heavily on promoting releases between environments. This is also used hand in hand with branch control on environments, ensuring that certain branches are not promoted past QA, for instance, a developer’s feature branch.

In an ideal world, the promotion would strictly follow the steps from left to right. However in the real world, you will come across some scenarios where you may want to break the rules, for instance

You may want to fast deploy to QA without waiting to develop
You may be working on a hotfix with changes on DEV/QA dependencies like your database that are no longer compatible with Prod, in which case you may want to deploy straight to UAT
You may have one of those tough days where you just need to get some quick and safe hotfix out straight to production

In such circumstances, you would want the ability to skip stages, which you can do with this structure when you trigger a new run

The approach motivated here ensures that changes from PR merges only make it as far as the development environment, with a nightly run being the primary way to get changes into QA, coupled with empowering the team to be able to run on demand to get things into QA. In this case however one has to be explicit in this intention by ticking the Ignore scheduling rules parameter.

Having things change under the feet of a busy QA team is usually not a good idea and a motivation to give QA teams control of when to get things, however coupling this with nightly runs ensures that this does not become an entirely manual process.

Environment approval gates) should also be considered. Recommendations

Explicit approval requirements, where one more team members have to give the green light for promotion to UAT and Prod. Pay close attention to the duration you choose for the approval expiration.
Consider business hours gates to control when promotion happens
Consider exclusive locks to avoid contention of deployments. This is particularly useful for the development environment if you have all Pull Request merges resulting in deployments, where successive merges can trigger multiple deployments. An exclusive lock will allow only the first through and cancel all the rest except the last one. This is best used with a pipeline that has only one stage per environment, if you have multiple the exclusive lock kicks in on that too.

⚠️ Azure Pipelines currently lack a feature to cancel all pending approval requests but the latest or at least cancel all prior ones when a later one is approved. This would come in very handy. Manage this well to avoid a bunch of pending approvals. I also recommend making use of a variable and conditions that ensure that unless the team is in a release cycle, where a build is being stabilized for release, UAT and Prod are never considered at all, hence not triggering approval requests that will just be ignored.

Before you panic when you see your first approval blocking a pipeline run, note that Azure Pipelines smartly handles this by yielding back the agent used to run the pipeline. Once approval is given, the pipeline stages/jobs that follow will then be scheduled on another agent. Approvals can be pending for multiple days and that is not an issue at all.

A nice thing with approvals is that instead of business hours approval checks, you can also have control of when an approved deployment kicks off by scheduling it on approval. This feature is not available when using the Azure Pipelines extension in Microsoft Teams.

Consider the Azure Pipelines integration with Microsoft Teams to receive notifications of approval requests and provide approvals directly from there.

Screenshots of sample runs

Manual run with default parameter values

Run with Ignore scheduling rules

Skipping the development environment

Branch control kicking in

What about Azure Releases?

You can mimic this by making use of the classic Azure Releases. However YAML vs ClickOps! Using YAML for defining your pipelines has several advantages:

Version Control: Track changes to the pipeline configurations just as you do with source code.
Reusability: Templates and parameters allow you to reuse and customize components across different projects or environments.
Transparency and Auditability: Changes are visible and auditable through commit histories, enhancing security and compliance.

Technical analysis checklist

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Glenn Carstens-Peters on Unsplash

Technical analysis checklist

This technical checklist should serve as a reference for things to consider before starting any technical work. Many checks need to be in place that range from understanding the ticket and establishing that it is ready for development, to determining blockers/dependencies, options, potential challenges and more to ensure that the technical implementation will be as smooth as possible. This can also ensure that other necessary aspects, such as code quality, security, deployment, and maintenance, are considered.

ℹ️ This checklist is not meant as a ‘gate’ or ‘red tape`. Such a checklist acts as a roadmap, guiding teams through complex processes in a structured yet flexible manner. They ensure thoroughness and adherence to best practices, while also fostering efficiency and clarity in project execution. This is best used as a once-off read as a team to establish how the team gets ready for technical work and as a reference only thereafter as necessary. It is certainly not a checklist in the sense that for each ticket you need to open this and tick things off.

Checklist

Is the ticket well understood?

Read and comprehend the user story or task description thoroughly. Clarify any ambiguities or uncertainties with the relevant stakeholders

Equally important is understanding the ‘why’ behind the specific change and how it will benefit the client. This will require acquiring a comprehensive business context for both the tickets and the overarching goals

Technical considerations

The following technical considerations should be made as clear as possible as part of the technical analysis

Required test coverage

Specify the types of tests that should be implemented, such as unit tests, integration tests, or end-to-end tests. Identify any special testing requirements, such as performance testing or security testing.
Error handling and logging
Monitoring and observability

Determine the key metrics, logs, or events that need to be monitored to ensure the system’s health and performance.
Frameworks and tools that can/should be used
Coding practices/standards that should be adhered to
Any required configuration
Any additional steps for the feature to be available
Artifacts that are expected as part of the implementation (documentation, scripts, Shared Lib contributions etc)
Special performance considerations

Identify performance requirements, such as response time, throughput, or scalability. Determine any performance testing tools or frameworks that should be used, such as JMeter. Specify any performance optimization techniques or best practices to be followed.

Structurizr local setup

Sat, 28 Dec 2024 00:00:00 GMT

Photo by Amsterdam City Archives on Unsplash

Structurizr local setup

This requires an installation of Docker on your local machine.

Structurizr builds upon diagrams as code, allowing you to create multiple software architecture diagrams from a single model.

This setup makes use of the Structurizr DSL, the recommended option when creating C4 diagrams with Structurizr. Structurizr is specifically designed to support the C4 model for visualising software architecture, by its creator Simon Brown. Diagrams are interactive (e.g. zoom in/out), animatable, embeddable, and include an automatically generated diagram key/legend.

Local setup

To quickly get up and running run the following docker command

docker pull structurizr/lite

This will pull down the docker image and if you start up a container from this it will start with a default diagram. To create your diagrams, however, you would want to point the docker container to a directory locally with the DSL. For example, if the location is /Users/you/structurizr, the command would be

docker run -it --rm -p 8080:8080 -v /Users/you/structurizr:/usr/local/structurizr structurizr/lite

Your instance of Structurizr Lite is now running and will be available locally on port 8080

A recommended approach however will be to create your own docker compose file and in that same directory have a sub directory with your DSL.

Here is a sample docker compose file.

version: '3.8'

services:
  structurizr:
    container_name: structurizr
    image: structurizr/lite:latest
    ports:
      - '8080:8080'
    volumes:
      - './structurizr:/usr/local/structurizr'

In this case, the Structurizr DSL and any other artefacts like images would be in a sub folder named structurizr. The container can be spun up with the command docker compose up

Here is a sample for a workspace.dsl file you can use as a starter with your local setup.

workspace {

    model {
        user = person "User"
        softwareSystem = softwareSystem "Software System" {
            webapp = container "Web Application" {
                user -> this "Uses"
            }
            container "Database" {
                webapp -> this "Reads from and writes to"
            }
        }
    }

    views {
        systemContext softwareSystem {
            include *
            autolayout lr
        }

        container softwareSystem {
            include *
            autolayout lr
        }

        theme default
    }

}

On running this locally, if you see this in your browser, the setup is successful

Visual Studio + Visual Studio Look Parity

Sat, 24 Aug 2024 00:00:00 GMT

Photo by Riku Lu on Unsplash

If you cannot escape the reality of having to dance between using Visual Studio and Visual Studio Code, but really appreciate consistency you may find Theme Converter for Visual Studio )interesting.

This extension allows you to identify a theme you like in Visual Studio Code (great options available and export it for usage in Visual Studio.

There are other investments that can be made to make them more consistent. Small things go a long way like having the editor bar on the same side in both, having the font style, sizing and spacing be the same, choice of mini map etc.

Upgrading Angular

Sat, 13 Jul 2024 09:00:00 GMT

Photo by Tudor Baciu on Unsplash

Upgrading Angular - Lessons from a 13 to 18 move

This guide highlights some steps for consideration when upgrading the version of Angular you are using. This is based mainly on my experience of upgrading a project from Angular 13 to Angular 18. All the steps and options listed below have been applied for that upgrade. However, proceed with caution, all those changes required extensive research to understand the implications of the change fully and this was backed by cumbersome developer testing and a plan for a full test regression cycle before a production release.

The time will come to upgrade your Angular version. It is best to not lag too far behind as it becomes a bigger chore to eventually catch up, especially if you are now forced to do it due to the need for new features or packages that are only available for a later version of Angular.

⚠️ There are multiple steps to go through for these upgrades. Make sure to do a step at a time, build and test, and address any issues, before moving on to the next step. Doing a big bang is a shortcut way to failing the whole upgrade and eventually ditching it.

Do not do the upgrades by hand, but instead make use of the official update tools, using the tool available here. The tool allows you to choose the from and to version and provides a guide. If you are upgrading across multiple versions, I would recommend doing it one version at a time fully testing things and addressing any issues before progressing to the next version. Make sure to go through and tick other dependencies like Angular Material and whether or not you are on Windows.

It is best to select the Advanced application complexity in the tool so it gives you detailed guides on what to look out for and you can just ignore what is not relevant in your case. If not sure, do a code search for the mentioned components or APIs that should be updated.

The process boils down to running the command ng update @angular/core@xx @angular/cli@xx for the target version and going through the rest of the guide for any manual changes that are required. If you are working with material you will also be asked to run the command ng update @angular/material@xx. The tool will make any required changes to your package.json file and install the packages. This includes core angular dependencies, angular tooling and dependencies like TypeScript.

Make sure you have no pending changes in GIT before running the commands and commit after each step.

Check the node compatibility for the version of Angular you are targeting as you may also need to upgrade that. I highly recommend you make use of nvm, so you can work with multiple versions of node on your machine, especially if you work across multiple projects.

Breaking changes

You may encounter breaking changes, depending on your project. Here are some examples of things I ran into while upgrading from Angular 13 to 18.

Angular 15 requires form models to now have a generic type parameter. You can still opt out by using the untyped version of the form modal classes.
Some dependencies, rxjs being one you will likely run into and particularly component libraries, will have versions that support specific Angular versions so as you update Angular you will have to bump some of these up too.
The package-lock.json file can get corrupted and track incorrect information. If your npm install starts to fail oddly or complain about packages missing that are in the package.json file, it may be best to manually delete the package-lock.json file and run npm install again. If this does not work, delete the node_modules folder, run npm cache clean --force then npm install.
Angular 16 drops the entryComponents option in modules.
Angular 18 does not accept certain special characters in string literals in HTML templates, like the @ character which I had to replace with @. The failing build was very clear about this issue and made a change recommendation.
When updating material there is a chance that some of the look and feel in your application may change and some weird UI bugs may also pop up, for example, controls no longer being positioned properly. Make sure to fully test things.

These are just examples of some breaking changes you can encounter. The good news is that any Angular-related changes I ran into were already mentioned in the upgrade tool with recommendations.

I am quite pleasantly surprised at how useful the Angular upgrade tool is and how much it automates, combined with how well the build gives detailed errors for anything that needs attention. While these upgrades remain quite tedious and very demanding in terms of testing, the tooling did make this significantly better.

New Angular, now what?

Now you have the new Angular and all the new features at your fingertips and any of the benefits like improved build and run times etc. One beast you may want to confront already is that over time you may start to have inconsistencies in your project. For instance if you now suddenly have access to the new standalone components, do you now only use them going forward and retain the modules you already have?

That is a tough question and the answer may largely depend on how complex your project is and how bold you are feeling. I would however recommend migrating as much as you can to use new features if you are already going to invest in tons of regression testing.

Migrate to use standalone components

You can migrate to use the standalone components using the official tooling ng generate @angular/core:standalone, see more documentation here. This will migrate in 3 steps

Convert all components, directives and pipes to standalone
Remove redundant NgModule classes
Bootstrap the application using standalone APIs

I highly recommend biting the bullet and doing this one as this highly reduces boilerplate code.

When I did this I had to address a few things manually. Suddenly some components that were using app-sidebar that does not exist with Angular 18 only started to fail the build after this move. A manual step we have to pull app configuration was also not migrated well and only done for the app root, but not for child components that required it.

Migrate the material theme

If you are working with material and have updated across Angular material versions that also have an update to the material specification, for example from the material specification 2 to 3, you may need to update your base theme if you are using any, as those target a specific specification, documentation here. In our scenario we had the deeppurple-amber.css as the base and we chose to move over to azure-blue.css.

Migrate control flow syntax

You can migrate to the new and shiny control flow syntax available since Angular 17 that simplifies the use of directives like *ngFor, *ngIf and *ngSwitch. This can be run using the official tooling ng generate @angular/core:control-flow

Other migrations

There are other migrations that you may be interested in that have tooling support or are fully documented. See the available migrations here.

angular.json

Depending on the version update you may also want to make changes to your angular.json config file.

browserTarget is dropped for buildTarget
defaultProject is now obsolete
defaultCollection is now obsolete

You can find out what needs to be changed by opening the file in your IDE to see any warnings. Comparing what you have with the default that is generated for a new project on the same Angular version is also a good idea.

tsconfig.json

Given that the version of TypeScript would have been updated, you should also look to see what configuration can be updated in the tsconfig.json file

Consider newer settings like
- noPropertyAccessFromIndexSignature
- noImplicitOverride
- esModuleInterop
- skipLibCheck
- moduleResolution move from node to bundler
- bumping up the ECMAScript version under target, module and lib

Other dependencies

This is a good time to look into upgrading all other dependencies, and in particular, some dependencies will be tied to a particular version of Angular, Node or TypeScript, which would also likely have changed.

Make sure to read the official documentation of each package to see what has changed and check the compatibility information. If you are far behind you are very likely to meet many packages with API changes.

⚠️ Update packages one at a time making sure to build and test your project afterwards. A big bang update and then having issues is a nice way to get bogged fail here.

Consider making use of the command npm outdated to track packages that need updates through a convenient report.

Also, consider making use of the VS Code extension, Version Lens for convenient information directly in your package.json file with upgrade actions directly from there.

Specific dependencies you may want to immediately update if you depend on are

rxjs
component libraries, i.e. ngx-bootstrap
eslint

Hopefully you can update all packages and get to see this nice report on npm install.

Cleanup

We took the Angular upgrade as an opportunity to identify and clean up any unused packages. That is a little less to worry about upgrading and this can also help keep bundle sizes reasonable.

Unused packages are not as simple as checking the package.json and finding something that is not used at all. Some may be used in modules/components that are not used in your project which would mean also deleting those artefacts.

Package.json

Make sure the common CLI tasks that developers run are available as scripts in the package.json file. Examples are serving, building, testing and linting.

Add an engines section so that the team knows and has a reference point for the supported runtimes like node. Example

 "engines": {
    "node": "20.15.1",
    "npm": "10.7.0"
  }

⚠️ Make sure no dependencies that should be for development only are in the main dependencies list.

Package-lock.json

When making multiple changes and in particular when the version of node has been updated, the package-lock.json can at times get corrupted and have incorrect information. It is a good idea to delete the file once done with the upgrade and run npm install again to have the file re-populated.

If you manage to update all dependencies, great. If not, it is still worth running an audit fix with npm run audit --fix to attempt to address packages with known vulnerabilities.

Testing

I would not recommend investing time in component testing with Angular. Writing unit tests for algorithm type code for example if you have a function that applies some formatting on numbers, that is something you may want to write unit tests for. For this, I recommend bootstrapping Jest onto your project.

When using jest for tests, keep in mind that jest runs on node and not in a browser, so any code that makes use of browser APIs that you would like to test would require mocks. Local storage is an example of an API that is implemented by browsers and not by node.

If you are not going to be writing component tests, it is best to remove all the dependencies for that (one less thing to worry about and keep up to date). Some dependencies to drop are jasmine, and karma including configuration files (karma.conf.js, tsconfig.spec.json etc) and any specs you may have already generated (*.spec.ts). Make sure to also remove the test configuration in your angular.json file.

If you are making use of the ng generate command make sure to use the --skip-tests flag if you are not making use of component tests.

Node version

As part of the upgrade do not forget to update your version of node. Make sure that you are working with a supported version of Node both locally and for your deployments. Check the official compatibility matrix.

⚠️ Do not run on an unsupported version of node. It may work now until you run into very weird bugs in the future. In those situations, you may not figure out that it is due to the node version and even if you do, now what? Bump down and regression test everything? Best to just avoid this. Also, make sure that the version of node used in production is the same one used by developers on their machines.

A good reason to upgrade Angular regularly is to keep up with an LTS version of Node. Node does however have very long maintenance windows. Check the official documentation here.

Scenic route

If you are making such a big change, this will no doubt be backed up by a regression test cycle. This is an opportunity to address any other low hanging-fruit.

Imports

Does your project have a lot of imports that walk up and down the folder tree like ../../../... These paths are not great as they tend to be long and do not do very well with moving files around, although most IDEs can handle this.

This is a nice time to address this by identifying the most commonly imported items in your project and creating aliases for those in your tsconfig.json allowing us to simply something like ../../../components/.. to @components/..

Here is a sample configuration, this will differ depending on your folder structure.

{
  "compilerOptions": {
    "paths": {
      "@services/*": [
        "src/services/*"
      ],
      "@models/*": [
        "src/models/*"
      ],
      "@environments/*": [
        "src/environments/*"
      ],
      "@store/*": [
        "src/app/store/*"
      ],
      "@modules/*": [
        "src/app/modules/*"
      ],
      "@shared/*": [
        "src/app/modules/shared/*"
      ]
    }
}

IDEs like VS Code will be aware of this configuration and will take this into account for auto imports.

Jest does not pull this information, so if you using Jest you will need to create module name mappings in the jest.config.js, example

module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  moduleNameMapper: {
    '^@services/(.*)$': '<rootDir>/src/services/$1',
    '^@models/(.*)$': '<rootDir>/src/models/$1',
    '^@environments/(.*)$': '<rootDir>/src/environments/$1',
    '^@store/(.*)$': '<rootDir>/src/app/store/$1',
    '^@modules/(.*)$': '<rootDir>/src/app/modules/$1',
    '^@shared/(.*)$': '<rootDir>/src/app/modules/shared/$1',
  },
};

Warnings

Look out for warnings that would be easy to address

Build warnings
Lint warnings
npm install warnings

Some examples of things that may be shown as warnings after an upgrade

Ambiguous CSS when working with flex layouts, for example, start and end as opposed to flex-start and flex-end
Unsupported engines on doing npm install will warn you about the version of node being a mismatch to some packages
While not specific to the upgrade, it could be good to also address any vulnerabilities flagged by npm install. Consider updating all your packages and where that is not possible consider running npm audit --fix to patch what would likely not result in breaking changes.
Angular will warn of the use of any CommonJS or AMD dependencies that can cause optimization bailouts and larger than necessary bundles. Angular can better do optimization during builds when working with ECMAScript modules. This warning will likely be due to dependencies and one common library that causes this is Lodash. In this case, it comes with lodas-es, that you can replace with to address this.
Due to updating TypeScript, you may start to see new warnings depending on what you have in your project.

In the example above of lodash warnings, make sure to update the core and types packages to use lodash-es and update all imports for example, import { isEmpty } from 'lodash'; becomes import { isEmpty } from 'lodash-es';. If you are making use of jest, you will need to provide a module name mapper in your jest.config.js

/** @type {import('tsjest').JestConfigWithTsJest} */
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
  moduleNameMapper: {
    '^lodash-es$': 'lodash'
  },
};

Linting

If you are modernizing your Angular SPA project, if you are not already working with eslint, I would highly recommend it and if you already are, this is also a good time to review your linting rules. If you are able to, upgrade to the highest eslint version that is supported for the version of Angular you have targeted.

Here is a simple configuration as an example for the latest eslint that now makes use of eslint.config.js.

import path from "node:path";
import { fileURLToPath } from "node:url";
import eslint from "@eslint/js";
import { FlatCompat } from "@eslint/eslintrc";

const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const compat = new FlatCompat({
    baseDirectory: __dirname,
    recommendedConfig: eslint.configs.recommended,
    allConfig: eslint.configs.all
});

export default [{
    ignores: ["projects/**/*", "coverage/**/*", "**/generated.ts"],
}, ...compat.extends(
    "plugin:@angular-eslint/recommended",
    "plugin:@angular-eslint/template/process-inline-templates",
).map(config => ({
    ...config,
    files: ["**/*.ts"],
})), {
    files: ["**/*.ts"],
    linterOptions: {
        reportUnusedDisableDirectives: "off"
    },

    languageOptions: {
        ecmaVersion: 5,
        sourceType: "script",

        parserOptions: {
            project: ["tsconfig.json"],
            createDefaultProgram: true,
        },
    },

    rules: {
        "@angular-eslint/use-lifecycle-interface": ["error"],
        "@angular-eslint/no-empty-lifecycle-method": "off",
        "@angular-eslint/directive-selector": "off",
        "@angular-eslint/component-selector": "off",
        "@angular-eslint/no-output-on-prefix": "off",

        "no-unused-vars": ["error", {
            vars: "all",
            args: "none",
            ignoreRestSiblings: true,
        }],

        "no-console": "error",
        "no-alert": "error",
        "no-debugger": "error",
    },
}, ...compat.extends("plugin:@angular-eslint/template/recommended").map(config => ({
    ...config,
    files: ["**/*.html"],
})), {
    files: ["**/*.html"],
    rules: {},
}];

If you have a linting setup, make sure that your commands (ideally scripts in the package.json file pass the --cache flag for faster runs). This will ensure that the linter only runs on files that have changed since the last run. Make sure to ignore the caching file from GIT.

If you have a linting setup, I highly recommend making use of lint-staged, to allow developers to run linting only for files that they have changed and are staged in GIT.

Keep your linting clean. Having tons of warnings that no one ever looks into is a nice way to slowly get the team to not care about linting. Either address all warnings, ignore existing warnings that are too difficult/risky to address or a combination of both.

Other cleanups

Also, consider other cleanups

Remove any unused static assets to keep your bundles small.
Consider your node_modules as well as your npm cache a mess right now and delete the folder, run npm cache clean --force and then npm install

Linting in .NET

Mon, 03 Jun 2024 00:00:00 GMT

Photo by Tim Gouw on Unsplash

Linting in .NET

Utilizing Roslyn analyzers for linting in .NET projects offers a myriad of advantages that can significantly enhance code quality and developer productivity. Unlike external tools like SonarQube (which are great btw, and can also be used in combo with Roslyn), Roslyn analyzers provide a seamless integration directly into IDEs and the dotnet CLI, empowering developers to identify and rectify issues in real-time as they write code. This immediate feedback loop fosters a proactive approach to addressing potential bugs, adherence to coding standards, and best practices, resulting in cleaner, more maintainable codebases.

Moreover, Roslyn analyzers offer deep insights into code patterns specific to .NET, enabling developers to leverage language-specific optimizations and refactoring’s effectively.

Linting of projects is generally best done early as you will not be overwhelmed with many pre-existing warnings on setting it up and risk getting numb to these. For .NET solutions it is a great idea to enable the TreatWarningsAsErrors in the beginning (ideally via a Directory.Build.props file) and you are never in too deep to do it later as you can use the WarningsNotAsErrors to ignore some and possibly NoWarn to suppress some from being in your build warnings, with care however to not suppress something you wish you had seen later. You can apply these overrides globally with a Directory.Build.props file or per project. You can also suppress things inline, although this is only useful when you have only a hand full of things to suppress. Example

#pragma warning disable [Rule Code] // short comment explaining what is ignored
                    cfg.UseRetry(x => x.None());
#pragma warning restore [Rule Code] //

If TreatWarningsAsErrors is too harsh for your scenario you can instead replace the combo TreatWarningsAsError + WarningsNotAsErrors + NoWarn with WarningsAsErrors + NoWarn to target specific things.

If done well your pipeline should not be overwhelming with warnings and you IDE should reflect the same.

One really great thing about Roslyn is that if the analysis rule also has a fix, this shows up in the IDE and can be done conveniently. Even when working with 3rd party Analyzers, the IDE support is impressive. ![[Pasted image 20240419171537.png]]

Once you have configured the rules and build behavior, everything just works and concisely for you IDEs, CLI, and Pipeline runs.

Analysis Rules

The default rule set is very relaxed as there are mostly not rights to some of the rules, like the ones around formatting and style, which however are important to have a consistent approach decided upfront and enforced. Out of the box this means you do not get too much from Roslyn and have to invest the upfront effort to actually configure these rules.

If you do not already have analysis setup, a good starting point to is to add a new .editorconfig file using the Visual Studio Solution context menu. This will add one that reflects the way your solution code already looks. You can author the rules by hand, or double click the file which will open a visualizer in Visual Studio.

Lean general rule recommendation as a starting point

root = true

[*]
guidelines = 120

# All files
[*]
indent_style = space
csharp_indent_labels = one_less_than_current
csharp_using_directive_placement = outside_namespace:silent
csharp_prefer_simple_using_statement = true:suggestion
csharp_prefer_braces = true:silent
csharp_style_namespace_declarations = file_scoped:warning
csharp_style_prefer_method_group_conversion = true:silent
csharp_style_prefer_top_level_statements = true:silent
csharp_style_expression_bodied_methods = true:suggestion
csharp_style_expression_bodied_constructors = false:silent
csharp_style_expression_bodied_operators = false:silent
csharp_style_expression_bodied_properties = true:silent
csharp_style_expression_bodied_indexers = true:silent
csharp_style_expression_bodied_accessors = true:silent
csharp_style_expression_bodied_lambdas = true:suggestion
csharp_style_expression_bodied_local_functions = false:suggestion
csharp_style_throw_expression = true:suggestion
csharp_style_prefer_null_check_over_type_check = true:suggestion
csharp_prefer_simple_default_expression = true:suggestion
csharp_style_prefer_local_over_anonymous_function = true:suggestion
csharp_style_prefer_index_operator = true:suggestion
csharp_style_prefer_range_operator = true:suggestion
csharp_style_implicit_object_creation_when_type_is_apparent = true:suggestion
csharp_style_prefer_tuple_swap = true:suggestion
csharp_style_prefer_utf8_string_literals = true:suggestion
csharp_style_inlined_variable_declaration = true:suggestion
csharp_style_deconstructed_variable_declaration = true:suggestion
csharp_style_unused_value_assignment_preference = discard_variable:warning
csharp_style_unused_value_expression_statement_preference = discard_variable:suggestion
csharp_space_around_binary_operators = before_and_after
dotnet_diagnostic.API1000.severity = error
dotnet_diagnostic.API1001.severity = error
dotnet_diagnostic.API1002.severity = suggestion

# disallow async void
dotnet_diagnostic.S3168.severity = error

# Xml files
[*.xml]
indent_size = 2

[*.{cs,vb}]
#### Naming styles ####

# Naming rules

dotnet_naming_rule.interface_should_be_begins_with_i.severity = suggestion
dotnet_naming_rule.interface_should_be_begins_with_i.symbols = interface
dotnet_naming_rule.interface_should_be_begins_with_i.style = begins_with_i

dotnet_naming_rule.types_should_be_pascal_case.severity = suggestion
dotnet_naming_rule.types_should_be_pascal_case.symbols = types
dotnet_naming_rule.types_should_be_pascal_case.style = pascal_case

dotnet_naming_rule.non_field_members_should_be_pascal_case.severity = suggestion
dotnet_naming_rule.non_field_members_should_be_pascal_case.symbols = non_field_members
dotnet_naming_rule.non_field_members_should_be_pascal_case.style = pascal_case

# Symbol specifications

dotnet_naming_symbols.interface.applicable_kinds = interface
dotnet_naming_symbols.interface.applicable_accessibilities = public, internal, private, protected, protected_internal, private_protected
dotnet_naming_symbols.interface.required_modifiers =

dotnet_naming_symbols.types.applicable_kinds = class, struct, interface, enum
dotnet_naming_symbols.types.applicable_accessibilities = public, internal, private, protected, protected_internal, private_protected
dotnet_naming_symbols.types.required_modifiers =

dotnet_naming_symbols.non_field_members.applicable_kinds = property, event, method
dotnet_naming_symbols.non_field_members.applicable_accessibilities = public, internal, private, protected, protected_internal, private_protected
dotnet_naming_symbols.non_field_members.required_modifiers =

# Naming styles

dotnet_naming_style.begins_with_i.required_prefix = I
dotnet_naming_style.begins_with_i.required_suffix =
dotnet_naming_style.begins_with_i.word_separator =
dotnet_naming_style.begins_with_i.capitalization = pascal_case

dotnet_naming_style.pascal_case.required_prefix =
dotnet_naming_style.pascal_case.required_suffix =
dotnet_naming_style.pascal_case.word_separator =
dotnet_naming_style.pascal_case.capitalization = pascal_case

dotnet_naming_style.pascal_case.required_prefix =
dotnet_naming_style.pascal_case.required_suffix =
dotnet_naming_style.pascal_case.word_separator =
dotnet_naming_style.pascal_case.capitalization = pascal_case
dotnet_style_operator_placement_when_wrapping = beginning_of_line
tab_width = 4
indent_size = 4
end_of_line = crlf
dotnet_style_coalesce_expression = true:suggestion
dotnet_style_null_propagation = true:suggestion
dotnet_style_prefer_is_null_check_over_reference_equality_method = true:suggestion
dotnet_style_prefer_auto_properties = true:silent
dotnet_style_object_initializer = true:suggestion
dotnet_style_collection_initializer = true:suggestion
dotnet_style_prefer_simplified_boolean_expressions = true:suggestion
dotnet_style_prefer_conditional_expression_over_assignment = true:silent
dotnet_style_prefer_conditional_expression_over_return = true:silent
dotnet_style_explicit_tuple_names = true:suggestion
dotnet_style_prefer_inferred_tuple_names = true:suggestion
dotnet_style_prefer_inferred_anonymous_type_member_names = true:suggestion
dotnet_style_prefer_compound_assignment = true:suggestion
dotnet_style_prefer_simplified_interpolation = true:suggestion
dotnet_style_namespace_match_folder = true:suggestion

# cancellation tokens must be forwarded to methods that accept them
dotnet_diagnostic.CA2016.severity = error
# cancellation tokens must come last in method signatures
dotnet_diagnostic.CA1068.severity = error

My recommended rules may be rather aggressive for an existing code base, but I cannot insist more for new code bases as a good start and as a team as errors pop up you can pivot and decide on your preferences, however the rules that pick up on security, vulnerabilities and bugs are best left aggressive as those are the ones that really add value. For legacy code bases, I recommend an incremental approach where you make a wish list of most important rules and slowly eat away at this, fixing those that are low risk/easy to test right away and potentially suppressing existing scenarios that are risking to apply with haste.

3rd Party Analyzers

Here are my GOTO recommendations

xUnit Analyzers

This one comes out of the box with xUnit, however it requires some investment to configure to server you.

Go through the analysis rules and see what makes sense with your team.

I prefer to make use of xUnit as a test framework, but not as an assertion framework and use Fluent Assertions for that.

Recommended rules

# Tests
# XFA001: Use FluentAssertions equivalent
dotnet_diagnostic.XFA001.severity = error

# xUnit1004: Test methods should not be skipped
dotnet_diagnostic.xUnit1004.severity = error

# xUnit1006: Theory methods should have parameters
dotnet_diagnostic.xUnit1006.severity = error

# xUnit1008: Test data attribute should only be used on a Theory
dotnet_diagnostic.xUnit1008.severity = error

# xUnit1013: Public method should be marked as test
dotnet_diagnostic.xUnit1013.severity = error

# xUnit1014: MemberData should use nameof operator for member name
dotnet_diagnostic.xUnit1014.severity = error

# xUnit1025: InlineData should be unique within the Theory it belongs to
dotnet_diagnostic.xUnit1025.severity = error

# xUnit1026: Theory methods should use all of their parameters
dotnet_diagnostic.xUnit1026.severity = error

# xUnit2000: Constants and literals should be the expected argument
dotnet_diagnostic.xUnit2000.severity = error

# xUnit2003: Do not use equality check to test for null value
dotnet_diagnostic.xUnit2003.severity = error

# xUnit2004: Do not use equality check to test for boolean conditions
dotnet_diagnostic.xUnit2004.severity = error

# xUnit2007: Do not use typeof expression to check the type
dotnet_diagnostic.xUnit2007.severity = error

# xUnit2008: Do not use boolean check to match on regular expressions
dotnet_diagnostic.xUnit2008.severity = error

# xUnit2009: Do not use boolean check to check for substrings
dotnet_diagnostic.xUnit2009.severity = error

# xUnit2010: Do not use boolean check to check for string equality
dotnet_diagnostic.xUnit2010.severity = error

# xUnit2012: Do not use Enumerable.Any() to check if a value exists in a collection
dotnet_diagnostic.xUnit2012.severity = error

# xUnit2011: Do not use empty collection check
dotnet_diagnostic.xUnit2011.severity = error

# xUnit2013: Do not use equality check to check for collection size.
dotnet_diagnostic.xUnit2013.severity = error

# xUnit2015: Do not use typeof expression to check the exception type
dotnet_diagnostic.xUnit2015.severity = error

# xUnit2017: Do not use Contains() to check if a value exists in a collection
dotnet_diagnostic.xUnit2017.severity = error

# xUnit2018: Do not compare an object's exact type to an abstract class or interface
dotnet_diagnostic.xUnit2018.severity = error

# xUnit2019: Do not use obsolete throws check to check for asynchronously thrown exception
dotnet_diagnostic.xUnit2019.severity = error

Fluent Assertions Analyzer

The first thing I recommend with this is ensuring that only Fluent Assertions can be used as an assertions framework.

As a bonus, if you had xUnit assertions, you can just click on any one of the new errors and use Roslyn to convert all your assertions to Fluent Assertions.

# Replace Xunit assertion with Fluent Assertions equivalent
dotnet_diagnostic.MFA001.severity = error

# Replace MSTests assertion with Fluent Assertions equivalent
dotnet_diagnostic.MFA002.severity = error

# Replace NUnit assertion with Fluent Assertions equivalent
dotnet_diagnostic.MFA003.severity = error

Roslynator

Roslynator is a set of code analysis tools for C#, powered by Roslyn.

Sonar Analyzer

Saving the best for last. I think this is hands down my preferred Analyzer as it comes with really great rules and fixes, with a whooping 450+ rules. This one works really great when used with the Sonar Lint extension available in multiple IDEs

The documentation of the rules from Sonar is 1st class and goes a long way in helping a team get up to speed with anything that may be new to them.

dotnet format

This one is not really an Analyzer, but is a dotnet global tool, which comes shipped with any .NET SDK, 5 and beyond can be used to enforce formatting as well as Roslyn rules solution wide. This started off as a formatter and has grown to be more and it’s name no longer serves it justice. On running this tool preferences will be read from an .editorconfig file, if present, otherwise a default set of preferences will be used.

Formatting is one of those things that can be tricky to have rigid gates on. Do you really want to block a PR due to a redundant line or additional space at the end of an expression? So some of these issues go ignored or unnoticed, but maybe once or twice a year it is good to just run this solution wide and get formatting to a clean slate before proceeding. On teams with good discipline, I have noticed that on doing this surprisingly there is quite a low number of formatting issues, biased though due to the high use of code generators which make things consistent.

Documenting individual roles and responsibilities

Thu, 26 Oct 2023 00:00:00 GMT

Photo by Pascal Swier on Unsplash

Documenting individual roles and responsibilities

The need for a shared understanding of individual roles and responsibilities within a team is fundamental to ensuring smooth project execution and achieving collective goals efficiently. Most teams/ogarnizations will no doubt have some mature process of clearly defining individual roles such as the RACI model. When a team gets together however this may be too detailed and complex to use as a reference point. A simple and easy to understand document that can be used as a reference point is therefore needed.

I advocate for a simple summary of each role, a paragraph or so in your team’s Wiki that can be used as a reference point. This should be a living document that is updated as the team evolves and grows. The document should be easy to understand and should be written in a language that is easy to understand by all team members. The document should be easily accessible and should be linked to from other documents such as the team’s onboarding document. This will help the team with the following

Clarity and Alignment

A shared view of roles and responsibilities ensures that everyone is on the same page regarding who is responsible for what. This clarity minimizes confusion, aligns expectations, and fosters a collaborative environment
Efficiency and Productivity

When each team member understands their role and the roles of others, it streamlines work processes, reduces redundancy, and enhances productivity.
Accountability

Documenting roles and responsibilities provides a reference point for accountability. It’s clear who is responsible for each task or deliverable, promoting a culture of ownership and accountability.
Onboarding and Training

Having a succinct documentation of roles and responsibilities is invaluable for onboarding new team members. It accelerates the learning curve, enabling newcomers to quickly understand the team structure and their role within it.
Communication

Effective communication is facilitated when everyone knows who to go to for what. It enhances the decision-making process and ensures that information flows to the right places.
Conflict Resolution

In case of conflicts or misunderstandings, having a documented reference helps in resolving issues objectively and swiftly.

Example

ACME Project Team

Name	Role
Yaretzi O’Sullivan	Junior Developer
Einar Bjornsson	Intermediate Developer
Chizoba Nwankwo	Senior Developer
Anika Patel	Technical Lead
Ulysses St. Clair	Intermediate Tester
Quinton Abernathy	Senior Tester
Lysandra Papadopoulos	Analyst
Ignacio Villanueva	Engagement Lead

Developers

The role of software developers is to design, develop, and maintain computer software systems. They are expected to translate software requirements into functional code, ensuring that the software meets the desired specifications and functions efficiently. Software developers typically collaborate with a team of other developers, designers, QA, analysts and engagement lead to understand user needs, brainstorm solutions, and create software solutions that address those needs. They are expected to write code using programming languages and frameworks, conduct thorough testing to identify and fix bugs or errors and optimize software performance.

Testers

The role of software testers is to ensure the quality and functionality of software applications or systems before they are released to end users. Software testers work closely with the development team to understand software requirements and design test cases and scenarios to validate the software’s performance, functionality, and usability. They execute various testing techniques, including manual and automated testing, to identify bugs, errors, and inconsistencies in the software. Software testers meticulously document and report any issues found during testing, providing clear steps to reproduce the problem and collaborating with developers to resolve them. They also perform regression testing to ensure that software updates or changes do not adversely affect previously working features. Additionally, software testers contribute to improving the overall software development process by providing feedback and suggestions to enhance software quality and user experience. They play a critical role in ensuring that the software meets quality standards and delivers a seamless user experience.

Analyst

The role of a business analyst is to bridge the gap between business objectives and technology solutions within an organization. Business analysts work closely with stakeholders from different departments to understand their needs, gather requirements, and analyze existing business processes. They conduct thorough research and data analysis to identify areas for improvement and recommend solutions that align with organizational goals. Business analysts collaborate with cross-functional teams, including the engagement lead, developers, and designers, to translate business requirements into technical specifications and ensure the successful implementation of projects. They also play a crucial role in defining project scope, creating use cases, and conducting feasibility studies to assess the viability of proposed solutions. Additionally, business analysts provide ongoing support by monitoring progress, facilitating communication between stakeholders, and conducting user acceptance testing to ensure that the delivered solution meets business requirements. Together with the engagement lead, they act as the bridge between business stakeholders and technical teams, driving effective collaboration and enabling the successful delivery of projects that meet business needs.

Technical Lead

The role of a technical lead is to provide technical guidance, mentorship, and oversight to a team of developers working on a software development project. Technical leads are expected to have in-depth knowledge of programming languages, development frameworks, and best practices. They work closely with the development team, collaborating on architectural design, code reviews, and troubleshooting technical issues. Technical leads provide guidance and direction to the team, ensuring adherence to coding standards, scalability, and maintainability of the codebase. They are responsible for ensuring priorities are completed and ensuring efficient utilization of resources within the team. Technical leads also play a crucial role in identifying and implementing technical solutions, evaluating and selecting appropriate technologies, and mitigating risks associated with the development process. They provide guidance and support to developers, while also facilitating the alignment of technical decisions with overall project goals.

Client Lead

The role of a client lead is to serve as a liaison and primary point of contact between a company or organization and its clients or stakeholders. The client lead is responsible for managing and fostering strong relationships with clients, understanding their needs and objectives, and ensuring their satisfaction with the products or services being provided. They work closely with cross-functional teams within the company to coordinate and deliver on client requirements, and they also play a crucial role in identifying opportunities for growth and expansion within existing client accounts. The client lead acts as a strategic advisor, utilizing their knowledge of the client’s business and industry to provide recommendations and solutions that drive mutual success and long-term partnerships.

Cultivating a Comprehensive Project Wiki

Wed, 25 Oct 2023 00:00:00 GMT

Photo by Wesley Tingey on Unsplash

Cultivating a Comprehensive Project Wiki

In software development, knowledge is both a catalyst and a safeguard. The more organized and accessible our project information is, the smoother our journey towards deployment. A Project Wiki serves as a central knowledge repository, ensuring that every team member, from developers to managers, operates with a shared understanding and easily accessible project resources. So what does a robust Project Wiki encompass? We will dive into this, serving as a guide to establishing a well-documented and insightful Project Wiki.

Resource Linkage

Document links to all pertinent project resources such as external documentation, SharePoint, project artefacts like source code and design documents, Team Communication Channels, and more. This forms the backbone of the Project Wiki, ensuring every asset is a click away.

Environment Documentation

Detail each environment (Development, Staging, Production, etc.), its purpose, links to access them, and the process to deploy or troubleshoot them (consider run-books for this). This section should demystify the environment setup

Getting Started Guide

A step-by-step guide to checking out the code, setting up the development environment, and running the project locally. Link to the README files of relevant repositories to keep the instructions up-to-date and centralized.

Repository Links

Provide links to all the repositories relevant to the project along with a brief description of the codebase each repository holds. This facilitates quick navigation and understanding of the project structure.

Team Dynamics Documentation

Outline the expectations, processes, and workflows the team adheres to. This section serves as a blueprint for how the team operates and collaborates.

Specific examples

Pull Requests: Detail the process for submitting, reviewing, and merging Pull Requests, underlining the importance of peer review and code quality.
Coding Standards and Guidelines: Document the coding conventions, best practices, and tools the team uses to maintain code quality and consistency.
Team Ceremonies: Describe the routine team ceremonies like standups, sprint planning, and retrospectives, explaining their purpose and schedule.
Work Item Management: Explain how work items are created, prioritized, and tracked throughout the sprint.

Team Composition and Contacts

List the team members, their roles, and contact information (particularly useful for external). Also, document the project requirements and expectations from each role, fostering clarity in responsibilities and channels of communication.

Project Requirements

Detail the project’s objectives, requirements, and any other high-level information that gives insight into what the project aims to achieve.

Run Books

Incorporate a section for run books which are essentially a compilation of procedures and operational tasks applicable to the system or software. They should cover routine operations, deployment procedures, and troubleshooting guidelines. This section ensures that there is a standard protocol to follow, reducing operational errors and enhancing efficiency.

Reason for outage tracking

Having a dedicated section for documenting outages and their root causes is crucial for future reference and for improving system reliability. Include detailed post-mortem reports, the steps taken to resolve the issues, and the measures put in place to prevent such occurrences in the future.

Releases Documentation

Maintain a thorough documentation of each release including the release date, version number, new features, bug fixes, and known issues. This section should also include rollback plans for each release to ensure system stability in case of unforeseen problems. By keeping a well-documented history of releases, team members and stakeholders can easily track the evolution of the project.

Architecture and Design

Dedicate a section to document the system’s architecture and design decisions. Include diagrams, architectural principles, and design patterns followed in the project. This section serves as a roadmap to understanding the high-level structure and the underlying rationale behind the architectural choices made within the project. Include links to more detailed design documents if available.

Conclusion

Creating a comprehensive Project Wiki is a proactive step towards fostering a culture of transparency, accessibility, and shared understanding within a team. It’s about laying down a foundation of knowledge that empowers every team member to operate with insight and efficiency.

A key part of the success however is in keeping the content up to date and relevant as well as encouraging contributions and collaborations from the entire team. Do give sufficient attention to that.

It’s also imperative to prioritize linking to existing documentation over duplicating information. Whether it’s design documents on Figma, technical specifications on SharePoint, or any other platform where project-related information resides, embedding links to these resources ensures that the Project Wiki remains a single source of truth without becoming bloated.

Enriching Code Review Culture with Pull Request Templates

Wed, 25 Oct 2023 00:00:00 GMT

Photo by Glenn Carstens-Peters on Unsplash

Enriching Code Review Culture with Pull Request Templates

The practice of submitting Pull Requests (PRs) is a tradition that not only actions the introduction of changes to the code but also encourages a culture of peer review and collaborative improvement. The elegance of a PR lies in its ability to encapsulate changes, encourage discussions, and ensure a seamless merge with the existing code.

This post aims to showcase not just the structural makeup of a well-curated Pull Request template to complement Pull Request processes and culture, but the underlying ethos that it carries. It’s about progressing from mere code contributors to being custodians of a culture that values clarity, collaboration, and code quality above all.

However, the effectiveness of this practice relies on the quality of the information provided within a PR. An ill-described PR can easily become a hole of iterative clarifications, delayed merges, and potentially missed deadlines. Here’s where a Pull Request Template comes into play. A well-structured PR template serves to give clarity, ensuring that every proposed change is accompanied by sufficient context, visual aids, and a checklist ensuring its readiness for review. In essence, a PR template is not just a tool, but a culture-enhancer.

A Pull Request (PR) Template is a predefined file that provides a structured format for developers to fill out when they create a new Pull Request on platforms such as GitHub, Azure DevOps or Bitbucket. The template is automatically populated in the pull request description field when a new pull request is created. The primary aim is to ensure consistency and quality in the information provided with each pull request, which in turn facilitates a smoother review process and better collaboration among team members.

Let’s see an example of a simplistic yet effective PR template and dissect its potential impact:

### Description
What are you changing, and what does merging this achieve?

### Visuals
- **Before and After**: If there's a UI change, provide screenshots/gifs showcasing the change.

### Checklist
- [ ] Self-review completed.
- [ ] Code compiles without errors.
- [ ] Relevant tests added/updated.
- [ ] All tests pass locally.
- [ ] Relevant documentation updated.

💗 Thank you for contributing!

The value given by PR template:

Explicitness: By urging the contributors to provide a clear rationale and expected impact, we’re fostering a culture of accountability and clarity.
Visual Aids: Encouraging the inclusion of visuals like screenshots can significantly accelerate the understanding of the proposed changes. This can also save the reviewer a lot of time in not having to pull down and run the changes, while still having a view and confidence of what the change looks like. PRs should emphasize code review and not be about testing changes this helps to achieve that while giving context.
Self-review Checklist: A pre-merge checklist instils a sense of responsibility towards ensuring code quality and readiness.

Implementing a Pull Request template isn’t merely about adhering to a set protocol, but about nurturing a culture of better communication, self-review, and collaborative growth. It’s about making each stride towards code enhancement a well-informed, well-discussed, and well-documented journey.

In fostering a culture of engineering excellence, tools and practices like a Pull Request template become indispensable allies.

Go ahead and review your team’s Pull Request process. These questions can help you establish where you may find value with Pull Request templates.

What are the things you do very well that you want to continue to encourage?
What are some things you can do better or that prove to be common pain points in PRs?
What usually causes delays in PRs?
What would make it easier or faster for reviewers to go through PRs?
What is commonly forgotten by the team when making PRs?

Focus on elevating the following areas with Pull Request Templates

Consistent structure and format
Information/context probing
Self-check list
Reducing time to approval
Standardizing compliance and best practices
Encouraging lean yet effective documentation that can be reviewed later
Encouraging effective collaboration
Increasing the quality of contributions

Pull Request templates are meant to complement the Pull Request process and culture and can work hand in hand with other tools. For instance, a line could have been added to remind and encourage the submitter to link tickets to the Pull Request, however with something like Azure DevOps as an example, branch protection can be used to enforce that tickets should be linked to Pull Requests instead.

Tailoring ChatGPT Responses with Custom Instructions

Fri, 20 Oct 2023 00:00:00 GMT

Photo by Jamie Templeton on Unsplash

Tailoring ChatGPT Responses with Custom Instructions

Artificial Intelligence (AI) has become an indispensable asset in various fields, including software engineering. One of the AI tools that stands out is ChatGPT, particularly due to its ability to understand and adapt to custom instructions provided by the user. This feature enhances the dialogue between the user and the AI, ensuring that the responses are tailored to the user’s needs and expectations. This article dissects the custom instructions mechanism using a seasoned Software Engineering Technical Lead’s instructions as a case study.

ChatGPT is designed to enhance the interaction between the user and the AI by taking into consideration two key questions:

What would you like ChatGPT to know about you to provide better responses?
How would you like ChatGPT to respond?

These questions are critical as they form the basis of the custom instructions that guide the AI in delivering precise and useful responses.

Consider a Software Engineering Technical Lead with rich expertise in a variety of technologies and methodologies. They provided the following detailed instructions to ChatGPT:

What would you like ChatGPT to know about you to provide better responses?

I'm a seasoned Software Engineering Technical Lead with expertise in .NET, TypeScript, React, and Kubernetes.

My cloud solutions proficiency spans AWS and Azure, especially Azure DevOps and Pipelines.

Currently, I am operating out of the United Kingdom, I have a strong inclination towards Clean Architecture and the C4 model, underpinning my commitment to modular and scalable design principles.

Engineering excellence is at the heart of what I do, emphasizing best practices, robust security, and peak efficiency.

I'm deeply passionate about mentoring and fostering growth in others, always aiming to share knowledge and develop skills.

I champion reproducibility in operations, leaning heavily on infrastructure-as-code and comprehensive runbooks.

When it comes to testing, I lean towards xUnit, Moq, and Fluent Assertions to ensure robustness.

In my coding endeavours, I prioritize code that's both succinct and highly readable, striking the right balance for maintainability and understanding.

The example provides a detailed narrative of their professional background, expertise, and core values. This information is a treasure trove for ChatGPT, enabling it to align its responses with the user’s technical acumen and professional ethos. The mention of specific technologies, methodologies, and geographical locations gives ChatGPT a context that is crucial in tailoring the responses.

How would you like ChatGPT to respond?

- Prioritize best practices and efficiency in all responses.
- Recognize my expertise and respond accordingly.
- Be complete and thorough.
- Recognize my expertise and respond accordingly.
- Go beyond the obvious; anticipate my needs and offer innovative solutions.
- Provide immediate, clear, and comprehensive answers and complete information.
- No need to mention that you are an AI, your constraints or the cut-off date.
- Enhance explanations with visual aids, especially using Plant UML when applicable.
- Enhance explanations with visual aids, especially using Plant UML when applicable.
- Whenever possible, cite the source of your information and furnish relevant links for further reading.
- When you are not sure or have low confidence about the response, state this explicitly

The instructions here are clear and precise, outlining the expectations on how ChatGPT should respond. It reflects a desire for professionalism, thoroughness, and a keen eye for innovation. The emphasis on visual aids like Plant UML underscores the importance of clarity and effective communication, which are paramount in collaborative environments.

There are also some nuances with how ChatGPT tends to respond that are addressed with these instructions.

Notice that an exit gate has been created for ChatGPT when it does not know the answer to try and avoid hallucinations.

The Impact and Significance

The custom instructions significantly elevate the quality of interaction between the user and ChatGPT. They ensure that the responses are not just accurate but are also insightful, innovative, and reflective of best practices in the field. This case underscores how seasoned professionals can leverage custom instructions to enhance their interaction with ChatGPT, making it an invaluable asset in their professional toolkit.

Conclusion

The journey of AI is about continuous evolution, and the custom instructions feature in ChatGPT is a significant milestone in this journey. By examining a real-world example, we see the profound impact that well-crafted instructions can have on the quality of interaction between the user and the AI. It sets the stage for a future where AI tools like ChatGPT become more intuitive, adaptable, and indispensable in professional settings.

More information on this feature is available here.

Pull Request Guidelines

Mon, 18 Sep 2023 19:00:00 GMT

Photo by Stijn Swinnen on Unsplash

Pull Request Guidelines

✅ DO consider PRs as a useful process to reviewing code and ensuring standards are followed.

✅ DO create a process to ensure review of PRs is done in a timely manner to avoid blocking other team members’ progress. Not getting this right can be a common reason for a team to not commit to the process.

✅ DO provide actionable and constructive feedback when reviewing Pull Requests and be prepared to provide more time to assist in further understanding or addressing that feedback.

✅ DO encourage reviewers to be ready to pair with contributors to assist with recommendations they have provided.

✅ DO encourage reviewers to provde specific examples or suggestions when requesting changes or improvements.

✅ DO respond to comments and reply with urgency, whether you have opened the Pull Request or are a reviewer.

✅ DO make use of various markdown features to make your comments easier to follow through. For example, use code fences when code is provided. Providing links is also critical.

✅ DO keep your PRs concise. Small PRs are easier to review, less error-prone, and quicker to integrate. Aim for a single concern per PR.

✅ DO ensure that the PR title is descriptive. It should give a summary of the changes, making it easier to understand the purpose of the PR at a glance.

✅ DO link relevant tasks, user stories, or bugs from Azure Boards to your PR. This provides context and allows tracking of work across the project both from the PR and from the linked work items.

Enforce this by making linked work items a requirement for a Pull Request to be merged via branch policies. You also get the added benefit of potentially autocompleting linked items on Pull Request merge.

✅ DO use draft Pull Requests in Azure DevOps if your changes are still in progress. This signals to reviewers that feedback is welcome, but the PR isn’t ready for final review.

✅ DO assign specific team members as reviewers based on their expertise or involvement in the code/module you’ve modified where necessary. This ensures the right eyes are on your PR.

While this ensures the right eyes are on a PR, it should be used sparingly and necessary. It will make for better team dynamics if any team member can review Pull Requests and in the long run, this can avoid bottlenecks and potential biases.

✅ DO consider automatically include reviewers on Pull Requests. You can create a group with all the team members on the team and assign this as required, meaning one team member needs to review and approve the PR before the merge

✅ DO address review feedback promptly. This keeps the review process efficient.

✅ DO favour inline or file level comments to provide context close to were the remarks apply on a Pull Request, either as author or reviewer.

✅ DO make use of the suggested changes feature to suggest replacement text for one or more lines in a file which offers convenience for the author to apply the changes by staging a commit if they agree.

✅ DO make use of comment statuses accordingly.

Active: the default status for new comments.
Pending: the issue in this comment is under review and awaits something else.
Resolved: the issue in this comment is addressed.
Won’t fix: the issue in this comment is noted but won’t be fixed.
Closed: the discussion in this comment is closed.

✅ DO use the vote feature as a reviewer to indicate your status of the review.

Approve: Approve the proposed changes in the PR. This option is just a vote and doesn’t approve the PR.
Approve with suggestions: Approve the proposed changes in the PR with optional suggestions for improvement. This option is just a vote and doesn’t approve the PR.
Wait for author: asks the author to review the reviewer’s comments. The PR author should let the reviewers know to re-review the code after the PR author has addressed the comments. If a required reviewer sets this vote option, the vote will block PR approval.
Reject: indicates that the changes aren’t acceptable. When you choose this option, add a comment explaining why. If a required reviewer sets this vote option, the vote will block PR approval. This vote option should normally not be required.
Reset feedback: clears your vote. The absence of a vote doesn’t prevent a PR from being

✅ DO make use of labels and tags in Azure DevOps to categorize your PRs. This can help in filtering PRs based on their nature (e.g., bug, enhancement) or priority.

✅ DO consider pushing Pull Request notifications to the Team’s preferred messaging platform, preferably something that allows further interaction like Microsoft Teams.

✅ DO set up Azure Pipelines to conduct a pre-merge build to ensure the changes don’t break the main branch. This is essential to maintain the health of your main branch. Create at minimum a CI pipeline for your PRS to ensure that code compiles, tests and linting succeeds among other things.

Do similar checks on merging a branch as verifying a branch does not guarantee that the code will necessarily be sane once merged. Consider checks like builds, test runs, deployments to a development environment etc. Your Pull Request checks may include things you may want to omit in these checks like linting.

✅ DO configure your PR settings to require at least one (or more) approval before merging. This ensures that every change has been vetted by another set of eyes.

✅ DO configure your PR settings to allow only the merge types that match your team branching strategies. Azure Repos has several merge strategies, and by default, all of them are allowed. You can maintain a consistent branch history by enforcing a merge strategy for PR completion.

✅ DO configure your PR settings to require comment resolution before merging.

✅ DO consider an exit hatch to bypass branch policies by granting bypass permission to a user or group. In some cases, you might need to bypass policy requirements so having this exit hatch would be handy, however, you must ensure that this is not abused.

✅ DO consider enabling autocomplete on creating a Pull Request. This automatically completes a Pull Request once it fulfils all policies and also allows for automatically completing associated work items and deleting the branch after merging.

It is highly encouraged to always enable this with the branch deletion ticked to keep the remote clean.

✅ DO encourage drive by reviews. Even if one person has taken ownership or reviewed a Pull Request, it should be fine and is quite beneficial for someone else with the capacity to join and ask questions or make additional comments.

✅ DO encourage the creation of new work items based on suggestions that are great but outside the scope of the current Pull Request. This avoids slowing down the review process while still considering great suggestions that go through the triage and prioritization processes of the team.

✅ DO encourage an environment where it is OK to disagree with Pull Request review remarks or how an implementation has been done, however politely and constructively. Encourage discussions and feedback in PRs, and always be respectful and adhere to your team’s standards and practices

Effort in ensuring the team has a shared view on coding practices and principles, architecture etc. makes it easier to be constructive in feedback as there is always a reference point on the agreed upon team approach, which however should not be the law but can also be challenged as the team learns more from the project and grows

✅ DO look out for common and recurring feedback and find ways to automate the checks. A lot of linting-esque tools can be used to automate such checks, for example Roslyn Analyzers and the Editor Config in .NET with dotnet tools like dotnet-format, ES Lint for JavaScript/TypeScript, Hadolint for Docker files, kubeconform for Kubernetes, [Spectral] for OpenAPI and more.

Use CI tools to automate linting, coding practices, test runs, and other checks to streamline code reviews and maintain code quality. Do this with PR comments that become regular.

See Github’s Super-Linter for inspiration on some linters that can be used in Github actions and independently elsewhere, for example from Azure Pipelines.

✅ DO use a Pull Request template to standardize the information provided when opening a PR. This ensures consistency and provides reviewers with the necessary context.

Example

### Description
What are you changing, and what does merging this achieve? Why is this change being made?

### Screenshots
Is this a UX change or can screenshots help the reviewer see the before and after or understand the solution better? Do you have any manual testing evidence?

### Checklist

- [ ] Self-review complete
- [ ] Code builds without errors
- [ ] Relevant tests have been authored/updated
- [ ] All tests are passing locally
- [ ] Relevant documentation updates have been made

💗 Thank you for your contribution!

How to modify data in a temporal/versioned table

Tue, 29 Aug 2023 00:00:00 GMT

Photo by Jan Antonin Kolar on Unsplash

How to modify data in a temporal/versioned table

Temporal tables, also known as system-versioned temporal tables, allow you to keep a full history of data changes in a separate table called a history table. This history table is linked to the main table and is automatically populated by the database system whenever a row in the main table is updated or deleted.

Given this behavior, direct modifications (like DELETE or UPDATE operations) to the history table are generally not allowed, to ensure data integrity and the accuracy of the history.

⚠️ If you’re trying to delete rows from the main table, and an error is being raised, make sure you’re not directly targeting the history table by mistake. If you delete rows from the main table, the corresponding rows in the history table will be preserved (that’s the whole point of temporal tables). If you’re trying to clean up or archive old history data, consider looking into best practices for managing data in temporal tables.

If you really need to modify the history table:

-- Set system-versioning to OFF
ALTER TABLE YourMainTableName SET (SYSTEM_VERSIONING = OFF);

-- Make your changes to the history table

-- Set system-versioning back to ON
ALTER TABLE YourMainTableName SET (SYSTEM_VERSIONING = ON (HISTORY_TABLE = YourHistoryTableName));

API Design Guidelines for Success

Fri, 21 Jul 2023 00:00:00 GMT

Photo by Douglas Lopes on Unsplash

API Design Guidelines for Success

Introduction

In today’s fast-paced world of software development, the quality of your APIs can make or break your project. Well-designed APIs are not only user-friendly but also set the stage for robust and scalable applications. In this comprehensive guide, we’ll explore API Design Guidelines that will pave the way for your API’s success.

Why API Design Matters

APIs (Application Programming Interfaces) serve as the bridge between different software components, allowing them to communicate and exchange data. A well-designed API can:

Improve developer experience by prioritizing simplicity and comprehensibility.
Enhance long-term maintainability and consistency.
Promote scalability and adaptability to evolving business requirements.
Ensure secure and efficient data exchange.
Facilitate collaboration among development teams.

Now, let’s dive into the API design principles that can help you achieve these goals.

✅ DO prioritize simplicity, comprehensibility, and usability to create an irresistible appeal for consuming engineers.

✅ DO actively improve and maintain API consistency over the long term.

✅ DO generalized business requirements to avoid use-case-specific APIs.

Do design with this evolution in mind and specifically to avoid breaking changes to the consumers.

✅ DO use a standard format for date and time values.

Use the string-typed formats date, date-time, time, duration, or period for the definition of date and time properties. The formats are based on the standard RFC 3339 internet profile — a subset of ISO 8601

✅ DO use standard formats for country, language and currency properties.

ISO 3166-1-alpha-2 for country codes
ISO 639-1 for language codes
ISO 4217 for currency codes

✅ DO use UUIDs for IDs to avoid scaling problems in high-frequency use cases.

✅ DO consistently use plurals for resource names.

✅ DO think of resources and avoid actions.

Good

PUT /users/{user-id}

Bad

PUT /getUser

✅ DO apply domain-specific resource names.

✅ DO Identify resources and sub-resources with path segments.

/resources/{resource-id}/sub-resources/{sub-resource-id}

✅ DO use nouns and NOT verbs for resource names.

paths:
  # bad
  /getCompanies:
    get:

  # good
  /companies:
    get:

Other examples

GET /users to retrieve a list of users
GET /users/{id} to retrieve a specific user by ID
POST /users to create a new user
PUT /users/{id} to update a specific user
DELETE /users/{id} to delete a specific user

✅ DO create types that convey the complete meaning for example favor types like stringified timestamps to convey relative time or types that carry the unit to explicitly communicate this.

{
  "weight": {
    "value": 68.0388555,
    "unit": "kg"
  }
}

# consider this as an alternative to the duration format
{
  "timeDuration": "01:30:00"
}

✅ DO pluralize array names.

🛑 DO NOT use null for empty arrays.

✅ DO explicitly name dates with date as a suffix or consider suffixes like at.

✅ DO use casing consistently across the API for operation ID, query parameters, path parameters, and property names in payloads.

camelCase for operation IDs, paths and schemas properties
snake_case for parameters
Choose a casing and stick to it for body content and stick to it, the preferable JSON with camelCase

✅ DO use the appropriate HTTP method for each operation. Standard methods include GET (retrieve), POST (create), PUT (update), PATCH (partial update), and DELETE (delete).

✅ DO avoid PUT requests that do not end with path parameters. For example, use /api/products/{product_id} instead of a more generic endpoint like /api/products/update.

✅ DO not return response bodies as a bare array, always nest in an object. As your API evolves and new fields or metadata need to be included in the response, it is easier to extend an object rather than a bare array. Additionally, when responses are nested in an object, it becomes straightforward to include relevant error information alongside the data.

Good

{
  "data": [
    {"id": 1, "name": "Product A"},
    {"id": 2, "name": "Product B"}
  ]
}

Bad

[
    {"id": 1, "name": "Product A"},
    {"id": 2, "name": "Product B"}
]

✅ DO apply care with required types that may surface with default in JSON or the language of the server implementation for these like numerics and enumerations. For validation and sanity purposes, it is great to make a distinction between not provided (null) vs a default especially when the values carry semantic meaning in the domain which can cause tough-to-track data corruption issues.

✅ DO return bodies for all 2xx responses except 204.

✅ DO avoid POST for operations that create resources.

✅ DO ensure that an operation that returns a list that is potentially large supports pagination.

GET /api/products?offset=1&limit=20

✅ DO ensure that an operation that returns 202 should not return other 2XX responses.

✅ DO ensure that all parameters for an operation are case-insensitive and unique.

🛑 DO NOT create GET long-running operations. The reason for this is that GET requests are meant to be used for retrieving data from the server, and they are expected to be relatively fast and non-blocking.

🛑 DO NOT return a body with a 204 response. The HTTP status code 204 No Content should not have a response body according to the HTTP/1.1 specification. When a server returns a 204 No Content status code, it indicates that the request was successful, but there is no additional content to send in the response payload.

🛑 DO NOT accept a body for a GET or DELETE request. According to HTTP/1.1 specifications, GET and DELETE requests should not have a request body. Both GET and DELETE methods are considered “safe” methods, meaning they should not have any significant side effects on the server’s resources or data. As a result, including a request body for GET or DELETE requests goes against the intended semantics of these HTTP methods.

🛑 DO NOT use special characters in paths. The use of special characters in paths can lead to issues with parsing, encoding, and security, and it may confuse both API consumers and developers.

# Bad
/api/user-profiles/{user-id}

🛑 DO NOT use GET for operations that change state. It should be safe and idempotent, meaning it can be called any number of times without changing the result.

✅ DO use the appropriate HTTP status codes to indicate the outcome of the operation. 2xx for success, 4xx for client errors, and 5xx for server errors. The following are common response types that you will likely need to use:

200 OK: The request has succeeded.
201 Created: The request has been fulfilled, and a new resource has been created.
202 Accepted: The request has been accepted for processing, but the processing is not yet complete.
204 No Content: The server has successfully processed the request, but there is no content to send back.
207 Multi-Status: The response body contains status information for multiple different parts of a batch/bulk request
400 Bad Request: The server cannot understand the request due to invalid syntax or other client-side errors.
401 Unauthorized: The request requires user authentication or the authentication has failed.
403 Forbidden: The server understood the request but refused to authorize it.
404 Not Found: The requested resource could not be found on the server.
405 Not Found: The method is not supported, and handled out of the box by most server frameworks.
409 Conflict: The request cannot be completed due to conflict with the current state of the target resource.
500 Internal Server Error: An unexpected error occurred on the server.
503 Service Unavailable: The server is currently unable to handle the request due to temporary overload or maintenance.

✅ DO return meaningful error messages with appropriate HTTP status codes.

Example Bad Request response

{
  "errors": {
    "Deal": [
      "'User ID' must not be empty."
    ]
  },
  "type": "https://tools.ietf.org/html/rfc7231#section-6.5.1",
  "title": "One or more validation errors occurred.",
  "status": 400,
  "traceId": "00-2a45f765492f8b44e8b42296ce8b37f8-2ad0369bf4ca922a-00"
}

Matching schema

components:
  schemas:
    error:
      type: object
      properties:
        type:
          type: string
          format: uri-reference
          description: The error type.
        title:
          type: string
          description: The error title.
        status:
          type: integer
          format: int32
          description: The `HTTP` error status code.
          example: 400
        traceId:
          type: string
          description: The internal error trace ID
    badRequestResponse:
      allOf:
        - $ref: "#/components/schemas/error"
        - type: object
          description: Bad Request response.
        - properties:
            errors:
              allOf:
                - $ref: "#/components/schemas/badRequestResponseError"
                - description: A specific error from the request input.
                - type: object
    badRequestResponseError:
      type: object
      description: A Bad Request error response.
      additionalProperties:
        type: array
        description: Properties with validation errors.
        items:
          type: string
          example: "User ID must not be empty"

  responses:
    BadRequest:
      description: Bad Request (validation failure - DO NOT RETRY).
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/badRequestResponse"

✅ DO version your API to allow changes without breaking existing clients. This can be done via the URL (/v1/users) or headers (Accept-version: v1).

Without versioning, making changes to your API can break existing clients.

⚠️ Parameter versioning is not recommended because versioning is not an optional feature or filter, and should not be treated as such

GET /users?version=1

✅ DO limit array, string and integer sizes to mitigate resource exhaustion attacks.

✅ DO use HTTPS for secure connections, except when a local host is used.

✅ DO ensure that path parameters use random IDs that cannot be guessed, such as UUIDs. Using UUIDs as path parameters is a straightforward way to enhance the security and privacy of your API, prevent unauthorized access, and ensure global uniqueness.

# Good
GET /api/users/82556e7a-0d8a-4c87-98e7-7395f0f8a3e6

✅ DO use token-based authentication: Tokens like JWT can be used to authenticate users. Tokens should be included in the Authorization header of the HTTP request.

Security scheme description must state that the implementation conforms with JSON Web Tokens RFC7519, the JSON Web Token standard.

GET /users
Authorization: Bearer <token>

🛑 DO NOT expose sensitive information in error messages, such as database details or confidential data.

🛑 DO NOT expose credentials like API keys, passwords, secrets or personally identifiable information (PII) in query parameters.

# Bad
GET /api/data?api_key=your_api_key_here

🛑 DO NOT use the Basic Auth security schema. Use a more secure authentication method like OAuth 2.0

# BAD
http://api.yourwebsite.com/users?password=secretPassword

🎇 DO implement rate limiting to protect the API against abuse and denial-of-service attacks.

When a client exceeds the rate limit, you should return a 429 Too Many Requests HTTP status code, along with a message indicating that the rate limit has been exceeded.

HTTP/1.1 429 Too Many Requests
Content-Type: text/html
Retry-After: 3600

{
    "message": "Rate limit exceeded. Try again in 3600 seconds."
}

The Retry-After header indicates how many seconds the client should wait before making another request.

✅ DO provide comprehensive API documentation, including available endpoints, request/response examples, error messages, etc. Tools like Swagger or Redoc can be used for interactive documentation.

Documentation should include:

The HTTP methods and endpoints available.
The request format including headers and body (with examples).
The response format includes status codes, headers, and body (with examples).
Any error messages that the API could return.

Endpoint: GET /users/{id}

Description: Fetches a user with the given ID.

Headers:
Authorization: Bearer {token}

Path parameters:
id - The ID of the user to fetch.

Example request:
GET /users/123
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9

Example response:
200 OK
{
    "id": 123,
    "name": "John Doe",
    "email": "john.doe@example.com"
}

✅ DO provide a consistent response format. Most RESTful APIs use JSON.

Using a consistent, well-structured response format makes your API easier to use and understand. It can also make it easier for clients to predict how your API will behave and to handle responses programmatically.

Linting and enforcing standards

For linting APIs, tools like Spectral can be used. It checks an OpenAPI document for issues like:

Compliance with naming conventions.
Missing required fields.
Incorrect data types.
Ensuring examples match the schema.
Ensuring paths are unique.
Enforcing best practices for versioning, parameters, responses, etc.
Custom rules defined as per the organization’s/team’s standards.

Open API Spec

The Open API Specification (formerly known as Swagger Specification) is an industry-standard specification for defining, documenting, and describing RESTful APIs. It is a language-agnostic and machine-readable format used to describe the structure and behaviour of APIs, making it easier for developers and computers to understand and interact with the API.

A good way to understand the specification is to navigate the Open API mind map

The specification is typically written in JSON or YAML format and defines various aspects of the API, including:

API Endpoints: It defines the available API endpoints (URLs) along with their supported HTTP methods (GET, POST, PUT, DELETE, etc.).
Request and Response Parameters: It specifies the parameters that can be sent in API requests and the data expected in API responses.
Data Models: These describe the data models and structures used in API requests and responses.
Authentication and Security: It defines how clients can authenticate themselves to access the API securely.
Error Handling: It describes how errors and error responses are handled by the API.
API Versioning: It provides a mechanism for versioning the API to support backward compatibility.
API Documentation: The Open API Specification can also include human-readable documentation to explain how to use the API effectively.

✅ DO call for early peer/ client consumer review of the API and set clear expectations for the review process.

✅ DO consider YAML when creating Open API specification for the following reasons:

Human-Readable Format: YAML has a more human-friendly and easy-to-read syntax compared to JSON. It uses indentation and whitespace, making it visually clearer and more natural to work with, especially when dealing with complex data structures.
Concise Syntax: YAML allows the omission of unnecessary punctuation and closing tags, leading to a more concise representation of data. This can result in shorter and more readable OpenAPI specifications.
Easier Comments: YAML supports inline comments, making it simpler to include explanatory notes within the specification. Comments help improve the clarity and understanding of the API design for developers and other stakeholders.
Multiple Documents in One File: YAML allows multiple documents to be defined in a single file, separated by three hyphens (---). This feature can be useful when managing related but distinct API specifications, allowing them to be kept together for convenience.
Built-in Support for Lists and Maps: YAML has native support for lists (arrays) and maps (objects), making it more intuitive to define complex data structures. This is especially useful when specifying request and response parameters for API endpoints.

Some tools allow for conversion between JSON and YAML

✅ DO consider Open API spec first approaches when in the design phase. This can open up many advantages

Clarity and Collaboration: Starting with an Open API specification allows you to define the API contract clearly and comprehensively. This specification acts as a single source of truth that fosters collaboration among various stakeholders, such as developers, designers, and testers. Everyone works based on the same specifications, reducing misunderstandings and miscommunications.
API Design Early in the Process: By focusing on the API specification at the beginning of the development process, you can design the API’s structure, endpoints, parameters, and responses thoroughly before implementing any actual code. This helps in making informed decisions and reduces the risk of rework or changes later in the development process.
Client-Server Independence: The OpenAPI specification allows developers to design APIs independently of the client or server implementation. This promotes the separation of concerns and allows teams to work concurrently on the API and the client or server implementations, making development more efficient.
Automated Documentation: The OpenAPI specification can be leveraged to generate interactive and comprehensive API documentation automatically.
Code Generation: There are many tools out there that support the generation of server and client code for various platforms and languages which can greatly accelerate development.

✅ DO ensure that the following information is always present in your specification

Tags, with descriptions
API host
API schemas
Info contact
Operation IDs
Security definition

✅ DO ensure that all operations are unique.

✅ DO version your API specifications and consider the YYYY-MM-DD format, optionally suffixed with ‘-preview’ over semantic versioning with numbers.

✅ DO always specify the format of integer values as either int32 or int64.

✅ DO ensure that all operations have at least one tag.

✅ DO ensure that every operation has at least one 2xx response.

✅ DO ensure that all schema properties have a defined type.

✅ DO ensure that all path parameters are of type string and specify the maxLength and pattern.

✅ DO provide descriptions for all schema properties.

✅ DO be careful of the source of the documentation for security reasons.

Apply linting to ensure that you do not have eval or script tags in the markdown.

🛑 DO NOT create paths that end in a trailing slash (/).

Tooling is inconsistent in how this is handled and this could lead to issues.

🛑 DO NOT explicitly define the Authorization, Content-type, and Accept headers.

🛑 DO NOT provide defaults for required parameters or properties.

You need to be able to make the distinction between a required property that is empty vs a default being provided.

🛑 DO NOT inline types, instead always explicitly define a schema.

Doing this consistently is great and also makes it easy to see all the schema definitions in one place and avoid duplications.

Sample showing some of what is highlighted in this document

info:
  title: [title]
  description: [description]
  version: [version]
  contact:
    name: [contact name]
    email: [contact email]
  x-audience: [audience]
servers:
  - url: [base url]
    description: [description]
    variables:
      version:
        default: v1
        description: [description]
tags:
  - name: [tag name]
    description: [description]
components:
  schemas:
    provinces:
      type: string
      description: An enumeration of all the supported provinces.
      enum:
        - "Gauteng"
        - "WesternCape"
        - "EasternCape"
        - "KwaZuluNatal"
        - "FreeState"
        - "Limpopo"
        - "Mpumalanga"
        - "NorthernCape"
        - "NorthWest"
    address:
      type: object
      description: An address.
      properties:
        line1:
          type: string
          description: The first line of the address
          example: "Apartment 4B"
        line2:
          type: string
          description: The second line of the address
          example: "123 Main Street"
        suburb:
          type: string
          description: The address suburb
          example: "Willow Creek"
        city:
          type: string
          description: The address city
          example: "Johannesburg"
        province:
          allOf:
            - $ref: "#/components/schemas/provinces"
            - description: The address province
            - example: Gauteng
        postalCode:
          allOf:
            - $ref: "#/components/schemas/postalCode"
            - description: The address postal code
      required:
        - line1
        - suburb
        - city
        - province
        - postalCode
  responses:
    BadRequest:
      description: [description]
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/badRequestResponse"
paths:
  /user/address:
    post:
      summary: [summary]
      operationId: updateUserAddress
      description: [description]
      tags:
        - [name]
      requestBody:
        content:
          application/json:
            schema:
              $ref: "#/components/schemas/address"
      responses:
        "202":
          description: [description]
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/addressUpdatedResponse"
        "400":
          $ref: "#/components/responses/BadRequest"
        "401":
          $ref: "#/components/responses/UnauthorizedError"
        "404":
          $ref: "#/components/responses/NotFound"
        "415":
          $ref: "#/components/responses/UnsupportedMediaType"
        "500":
          $ref: "#/components/responses/InternalServerError"

API consumer manual

✅ DO consider creating a user manual to complete the API to be shared with all stakeholders.

This does not have to be the classic manual as a document like a PDF, but would be much better as a living resource like a Postman collection that can be complete with extensive documentation and use case-specific examples that can easily be run on demand.

Spectral

Spectral is an open-source tool designed to enforce API design best practices and ensure API specifications (such as Open API or Swagger) adhere to specific rules and guidelines. It acts as a linter, which means it checks API definitions against a set of pre-defined rules or custom rules to identify potential issues or deviations from best practices.

The primary purpose of Spectral is to improve the quality and consistency of API specifications by providing automated checks on various aspects of the API design. This includes checking for naming conventions, validation of data types and formats, ensuring required fields are present, and detecting other common mistakes or discrepancies.

See here a recommended configuration that encompasses all the design guidelines discussed in this document.

Conclusion

Creating well-designed APIs is essential for building successful software applications. By adhering to these API design guidelines, you’ll improve developer experience, ensure long-term maintainability, and promote scalability and security. Remember, a well-designed API is not only a technical asset but also a strategic one.

Ready to implement these guidelines? Start by reviewing your existing APIs and making improvements where necessary. Your users and fellow developers will thank you for it!

TypeScript and Angular Standards

Wed, 19 Jul 2023 19:00:00 GMT

Photo by Tim Wildsmith on Unsplash

Node

✅ DO npm ci over npm i when running installs in continuous integration. Utilizing this in continuous integration workflows, such as Azure DevOps, offers significant benefits for reliable and reproducible builds. Unlike npm install, which can have varying outcomes due to the package-lock.json file, npm ci ensures deterministic package installations and fosters consistent build results

Deterministic Installations: With npm ci, you can be confident that the exact versions of packages listed in the package-lock.json file will be installed. It disregards any existing node_modules folder and starts from scratch, guaranteeing a reproducible environment across CI builds. This eliminates potential issues caused by conflicting or mismatched dependencies.
Faster and Reliable Builds: By leveraging the package-lock.json file, npm ci performs faster installations compared to npm install. It bypasses dependency resolution, fetching packages directly based on the lockfile. Consequently, it reduces network requests, resulting in faster and more reliable builds, which is particularly valuable in time-sensitive CI processes.
CI-Friendly Caching: In CI environments, caching is vital to optimize build times. With npm ci, the lockfile integrity check is more straightforward, making it easier to leverage caching mechanisms. By caching the node_modules directory and the package-lock.json file, subsequent CI builds can benefit from faster dependency restoration, providing efficient pipeline execution

TypeScript and Angular Standards

⛔ DO NOT use string interpolation when creating HTTP requests for the request parameters and instead make use of HttpParams

// BAD
preVetCreditApplication(creditApplicationId: string, dealId: string): Observable<PrevettingResponseDto> {
  const url = `${this.baseUrl}/v1/CreditApplications/prevet?creditApplicationId=${creditApplicationId}&dealId=${dealId}`;

  return this.http.get<PrevettingResponseDto>(url);
}

// GOOD
preVetCreditApplication(creditApplicationId: string, dealId: string): Observable<PrevettingResponseDto> {
  const params = new HttpParams()
    .set('creditApplicationId', creditApplicationId)
    .set('dealId', dealId);

  return this.http.get<PrevettingResponseDto>(`${this.baseUrl}/v1/CreditApplications/prevet`, { params });
}

By using HttpParams the code becomes more readable and self explanatory especially when there are a lot of parameters involved. More importantly this allows for dynamic handling of parameters with conditionals and parameter serialization ensuring that the values are properly encoded for URL transmission. This eliminates the need to manually handle encoding concerns, reducing the risk of errors and security vulnerabilities.

⛔ DO NOT make use of as unless absolutely required. Type assertions with as in TypeScript remove static type checking which can lead to issues.

interface User {
    name: string;
}

// BAD
// adding a new property on User for instance would not cause an error here
const getUser = () => {
    return {
        name: "John Doe",
    } as User;
};

// GOOD
const getUser = (): User => {
    return {
        name: "John Doe",
    };
};

✅ DO favor the use of immutable variables where possible. Immutable variables are easier to reason about as no change is anticipated.

let fileToLoad = 'documents';

if (connected) {
  fileToLoad = 'documents-other';
}

// can be collapsed into
const fileToLoad = connected ? 'documents' : 'documents-other';

// easier to read, easier to reason about

✅ DO favor the use of absolute imports over relative imports.

// given src/services/translate.ts

// an import from /src/../something.ts
import { translate } from "../../services/translate";

// changes to regardless of where imported from
import { translate } from "services/translate";

Can be achieved by adding the following to tsconfig.json

 "compilerOptions": {
  "baseUrl": "src"
}

If we want to tell local imports apart from rest we can use aliasing

 "compilerOptions": {
    "baseUrl": "./",
    "paths": {
    "~component/*": ["src/components/_"],
     }
}

// import {Login} from ‘~component/Login;

✅ DO prefer use of interfaces. Use type when you need specific features offered by types.

✅ DO use const and let over var, using let only where mutation is expected. The scoping semantics of var lead to bugs.

✅ DO use !x instead of x==null.

if (!x) {
    //'x' is falsey, so it will evaluate to false when undefined, null or zero (0)
}

✅ DO favor the use of the use of modern javascript features over the more classic counterparts. Examples of features to be familiar with:

Template literals (string interpolation)
Rest and spread operators
Destructuring assignments
Object literals
Arrow functions
Yield
Ternary operator
Nullish coalescing operator
Promises
async and await

✅ DO familiarize yourself with array methods.

find
some
every
includes
map
filter
reduce

ℹ Favor functional map, filter, reduce, forEach, some ect over collection mutation

Jest

✅ DO group related tests under a describe if multiple tests are in one file such as with tests for utils.

✅ DO name test files with suffix tests i.e utils.test.ts

✅ DO favor data driven tests with test.each over duplicated tests

test.each([
    [new Date(2000, 11, 12), false],
    [new Date(1970, 11, 12), false],
    [new Date(1969, 11, 12), true],
    [new Date(1950, 11, 12), true],
])("isDateBefore_1970 returns relevant boolean", (date: Date, expected: boolean) => {
    expect(isDateBefore_1970(date)).toBe(expected);
});

✅ DO favor naming tests in the format given then should or given when then should. Long test names are perfectly fine.

✅ DO make it easy to tell apart the Arrange, Act and Assert sections of your test and in particular to test what the system under test (sut) is.

✅ DO NOT have magic values in tests. Simple things like inlining variables to have name can go a long way in test readability.

// BAD - why is this value of significance?
expect(result.count).toBe(24);

✅ DO use the most specific assertions, these give the most specific and useful failure messages as well.

⛔ DO NOT suppress linting rules, these are in place for a reason. Do consult other developers before suppressing rules, whether for specific code lines, files or entire workspace

⛔ DO NOT write tests with a multiple responsibilities, instead a test should have a single focus. Multiple unrelated assertions are a red flag.

⛔ DO NOT have conditional logic in tests. Rather make data driven tests with theories or separate tests even.

GOTCHAS

Jest

✨ When setting up a mock with jest.mock, ideally you may want to setup one of the properties bases on a variable so that you can use the same value in you assertions. Jest will however fail the test complaining that mocks cannot access outside variables. This is in place to avoid dirty mocks, however you can by pass this by simply naming the variables with the mock suffix i.e mockUser

const mockUnknown = "#9E9E9E";

jest.mock("../../redux/store", () => ({
    getState: (): RecursivePartial<RootState> => ({
        app: {
            statusColors: {
                unknown: mockUnknown,
            },
        },
    }),
}));

3 Key life lessons Photography has taught me.

Mon, 10 Jul 2023 00:00:00 GMT

Photo by NordWood Themes on Unsplash

3 Key life lessons Photography has taught me.

Sometime in 2017, I found myself with plenty of time to spare, bored, tired of routine life and desperately in need of a new hobby. I had always been curious about amateur photography, so I decided to give it a try. I promptly acquired a Canon 1300D (Rebel T6) camera, a popular choice among beginner photographers looking to test the waters without spending a fortune.

Going in, I somewhat expected this to be much like one of the many other things I tried and eventually lost interest in. Fast forward to some 6 years later, my passion for photography is stronger than ever, and I am further invested in it in many other ways. This pursuit has led my family and me to travel a lot, unashamedly, often motivated by my desire to capture the next great shot.

My journey through photography has taken me to extraordinary places, some unusual, others scary. It’s been an adventure filled with highs and lows, including navigating a slump during the COVID lockdowns and adjusting to a shift from urban to rural life. But along the way, I’ve acquired lessons that extend far beyond the scope of photography, profoundly influencing my approach to life.

Lesson 1 – Balance being in the moment and living the moment.

In the early days of photography journey, I was focused on capturing every moment. This often led me to experience life through the lens, detached from the reality around me. While this wasn’t a problem on solo adventures, it became noticeable when traveling with loved ones, especially abroad. I realized that in my quest to capture the perfect moment, I was missing out on creating meaningful memories, and sometimes, even diminishing the experience for others. Decisions like the accumulation of bulky gear didn’t just slow me down; it also impacted those around me.

I learned the hard way that being present is as important as capturing the moment. This insight applies not just to photography but to life and career. It’s easy to get absorbed in a single task or project, missing out on life’s little joys. Productivity is important, but so is enjoying the journey. Our careers and lives should be more than just a collection of achievements; they should be filled with learning, growth, connections, and personal discoveries.

It’s easy to become trapped in a single moment. We might get absorbed in a demanding project and miss out on other things, like having lunch with colleagues or attending social events. But in the end, we’re not truly living in the present. While productivity and work are important, it’s equally important to enjoy the journey because there will come a time when what we have missed weighs heavier on us than we anticipated. Perhaps the project was a success, but we realize that the year wasn’t particularly enjoyable, and we did not grow much. The greatest lesson photography has given to me is the importance of being present. Let’s approach our lives and careers with the wisdom of a seasoned photographer. Let’s focus on our goals, but also be mindful of the journey. Let’s capture the achievements, but also live the experiences.

On the day of the shot above, I went too far, dragging a ton of gear up a mountain, including the giant lens which you see my wife being burdened with, and worst of all looking at it all through the lens. The experience could have been a whole lot better in so many ways.

Lesson 2 – Seek for and be open to constructive criticism.

Right from the first shot I took, I was excited to share my photographs with others and hear their feedback, especially once I became comfortable using my gear. However, I often found that others didn’t always share my excitement about certain shots. Initially, this was confusing and disheartening, but I gradually understood the critical difference in perspectives. As a photographer, I focused on technicalities, while viewers reacted to the emotions, visual appeal, and stories conveyed. This perspective is significant because award-winning photos are not judged by other photographers, but by the public, many of whom have no detailed knowledge of photography. This realization underlined the importance of diverse perspectives, not just in photography but in life and career.

Recognizing this important difference made me realize the relevance of other perspectives in my career and life. As someone whose work is overly technical, it is easy to focus on and get absorbed in specific metrics and Key Performance Indicators. However, what I deliver is not viewed in that lens. What truly matters is how my technical expertise translates into meaningful outcomes for others, the clients in particular. The feedback from these diverse groups, who may not understand the intricacies of my work but are the ultimate beneficiaries, is crucial. This experience has made me value the opinions of this group as a key pillar to attaining success.

I have also observed the same phenomenon in life. There are many situations we encounter, where our perception is influenced by our specific perspective, potentially leading to many biases. When we open ourselves up to the opinions of others, whether they are family, friends, or even strangers, we can learn something interesting.

I would recommend approaching life in the same way as a seasoned photographer. Be open to criticism, and if anything, seek it out. You will grow as an individual when people honestly review your work, evaluate the situations you are dealing with, or provide feedback on anything in life, for that matter. Any difference in perception is not a setback but a valuable insight.

Silhouettes as shown in the 1st two shots, are a technique I enjoy a lot and one I will go out of my way to get. I waited in the same position for over an hour to get that first shot. However, the images that often make an impression on others are the ones that evoke some emotions, much like the last one which usually happens in a split moment, are captured hastily, and do not require much technical finesse to get right. While I continue to shoot what I like and enjoy when I am out to get impressions, I know what my audience likes.

Lesson 3 – Go out of your way on preparation and embrace unpredictability.

The third and final key lesson was particularly tough for me to learn, as I had to learn it the hard way. Throughout my photography journey, there have been many off days that could have been avoided if I had been a little more prepared. Missing out on shots due to uncharged batteries or forgotten tripods and memory cards, or perhaps not checking the weather, has all been frustrating. Then the time came when I was more prepared for what I knew and expected, but it all changed on the day or in the moment of the shot.

It has been a tough lesson, but over time, what I needed to do better clicked. Now, when I have an important shot or when my family and I are travelling, I have a checklist. I know exactly what I need to bring. I don’t bring all my gear, but I always refer to the checklist. I will consider the area we’re going to, the type of photography I’ll be doing, the sunrise and sunset times, the weather, the distances we’ll need to travel and more.

This lesson applies to so many aspects of life and career. The preparation, however, is only half the story. With photography, there lies the ability to adapt to the unexpected: a sudden change in weather, or a spontaneous moment. These unpredictable elements often transform a good photographer into a great one. In our professional and life endeavors, we often equate success with planning – setting clear goals, strategizing, and methodically working towards our objectives. While this is the first part of the lesson and a key thing, the unpredictable nature of the world often throws us curveballs and must be acknowledged. Trends shift, new technologies emerge/deprecate, relationships evolve, and unforeseen challenges arise. The ability to adapt, pivot our strategies, and embrace change is what often separates those who are resilient from those who aren’t.

As we walk the path of life/careers, let us be like the photographer who sets up their camera with care but is always ready to capture an unplanned, perfect shot. Let’s prepare for the future, but also stay open and adaptable to the unforeseen twists and turns of life. Embracing this balance is the key to not just surviving but thriving in the unpredictable journey of life.

You need to have all the conditions just right as a photographer to take that one great shot. You need the gear, lighting, camera settings and many other variables to be just right that need preparation. When working with a subject like a dragonfly that only gives you a second or so to work with, preparation and adaptability are everything.

My photography journey has taught me more than just these three lessons. Knowing your tools intimately, stepping out of your comfort zone, valuing your unique journey, avoiding dogmatism about tools or methods, sharing knowledge generously, seeking community and support, and above all, having fun are equally important. These principles, derived from photography but are applicable to every facet of life.

Photography has been more than a hobby; it’s been a journey of discovery and learning. It’s taught me to view life through various lenses, both literally and metaphorically. As I continue to explore the world through my camera, I’m reminded that life, like photography, is about the journey and the experiences, not just the outcomes. These lessons encourage a fulfilling and adaptive approach to life’s challenges and opportunities. Let’s navigate our paths with the curiosity of a photographer, prepared yet adaptable, ready to capture life’s unexpected beauty.

Web API Conventions

Fri, 12 May 2023 19:00:00 GMT

Photo by Brian McGowan on Unsplash

ASP.NET Core Web API Conventions

ASP.NET Core Web API conventions are a set of guidelines that help standardize the most common return types, status codes, and parameters for specific types of actions. Conventions are useful for enhancing the understandability and usability of your API. Moreover, they help to create uniform API documentation, especially when using tools like Swagger.

Using Conventions

To use conventions, you need to create a class that inherits from the ApiConventions class. This class will contain the definitions for your conventions. For example, you could create a convention for the Get action that always returns a 200 OK status code and a JSON object.

Once you have created your convention class, you need to register it with the API. This can be done in the ConfigureServices method of your Startup class.

services.AddMvc(options =>
{
  options.Conventions.Add(new MyConventions());
});

[assembly: ApiConventionType(typeof(MyConventions))]

Alternatively, you can apply this at an assembly level:

[assembly: ApiConventionType(typeof(MyConventions))]

Controller level:

[ApiConventionType(typeof(DefaultApiConventions))]
public class ContactsConventionController : ControllerBase
{
}

And Controller action level:

[HttpPut("{id}")]
[ApiConventionMethod(typeof(DefaultApiConventions),
                     nameof(DefaultApiConventions.Put))]
public IActionResult Update(string id, Contact contact)
{
}

Once you have registered your conventions, they will be applied to all actions that match the criteria you defined.

Conventions in ASP.NET Core Web API can help you define:

The return type of an action
The status code of an action
The parameters of an action

To define the return type and status code of an action, you use the [ProducesResponseType] attribute. For example, the following attribute defines the Get action to return a JSON object with a 200 status code:

[ProducesResponseType(StatusCodes.Status200OK, Type = typeof(IEnumerable<MyObject>))]
public IEnumerable<MyObject> Get()
{
  // ...
}

To define the parameters of an action, you use the [ApiConventionNameMatch] and [ApiConventionTypeMatch] attributes. For example, the following attribute defines the GetById action to take an id parameter:

[ApiConventionNameMatch(ApiConventionNameMatchBehavior.Suffix)]
[ApiConventionTypeMatch(ApiConventionTypeMatchBehavior.Any)]
public IEnumerable<MyObject> GetById(string id)
{
  // ...
}

[ApiConventionNameMatch(ApiConventionNameMatchBehavior.Suffix)]
[ApiConventionTypeMatch(ApiConventionTypeMatchBehavior.Any)]
public IEnumerable<MyObject> GetById(string id)
{
  // ...
}

Benefits of Using Conventions

There are several benefits to using conventions in ASP.NET Core Web API:

Standardization: Conventions can help you to standardize your API, making it easier to understand and use.
Reusability: Conventions can be reused across multiple actions, making your code more concise and easier to maintain.
Documentation: Conventions can be used to generate documentation for your API, making it easier for users to understand how to use it.

Default Conventions

ASP.NET Core Web API includes a set of default conventions that can be used to define the return type, status code, and parameters of common actions. These conventions are defined in the DefaultApiConventions class.

The following table shows the default conventions for the most common actions:

Action	Return Type	Status Codes	Parameters
`Get`	`IEnumerable`	`200`, `404`	None
`Post`	`object`	`201`, `400`	`object`
`Put`	`object`	`200`, `404`, `400`	`object`
`Delete`	`void`	`200`, `404`, `400`	`int`
`Create`	`object`	`201`, `400`	`object`
`Edit`	`void`	`204`, `404`, `400`	`int`, `object`
`Find`	`IEnumerable`	`200`, `404`	`object`
`Update`	`void`	`204`, `404`, `400`	`int`, `object`

Conclusion

ASP.NET Core Web API conventions are a powerful tool that can significantly improve the quality and maintainability of your API. They ensure standardization, promote reusability and facilitate the automatic generation of API documentation. By using conventions, you can make your API more understandable and easier to use. If you’re not already using them, it’s certainly worth giving them a try.

Dependency injection with Scrutor

Mon, 01 May 2023 19:00:00 GMT

Photo by Mick Haupt on Unsplash

Dependency injection

Dependency injection is a key concept in modern software development, as it allows developers to write code that is modular, testable, and easy to maintain. Microsoft.Extensions.DependencyInjection is a popular DI container in the .NET ecosystem, which allows developers to register dependencies and resolve them at runtime. However, as projects grow in size and complexity, managing all the dependencies can become challenging.

That’s where Scrutor comes in. Scrutor is a library that complements Microsoft.Extensions.DependencyInjection, providing additional features for scanning and registering dependencies. It allows developers to scan assemblies and register dependencies based on naming conventions, interfaces, and attributes, reducing the amount of boilerplate code required for DI configuration. In this article, we’ll explore some of the benefits of using Scrutor and show code examples of how it can be used to improve the dependency injection experience.

Scanning for Dependencies with Scrutor

Let’s start by looking at an example of registering dependencies without Scrutor

services.AddScoped<IRegistrationService, RegistrationService>();
services.AddSingleton<RedisHealthCheck>();
services.AddSingleton<SqlDbHealthCheck>();
services.AddSingleton<AzureServiceBusHealthCheck>();

This is a straightforward approach to registering dependencies, but it can become cumbersome as the number of dependencies grows. Additionally, if a new dependency is added to the project, it needs to be manually registered with the container, for example, if we were to introduce more implementations of health checks we would need to remember to include them manually.

With Scrutor, we can simplify this process by scanning for dependencies based on naming conventions or attributes. For example, we can scan all assemblies in the current application domain and register all classes that implement an interface with a given name

services.Scan(scan => scan
    .FromAssemblyOf<IHealthCheck>()
    .AddClasses(classes => classes.AssignableTo<IHealthCheck>())
    .AsSelf()
    .WithSingletonLifetime());

The FromAssembliesOf method specifies which assemblies to scan, and the AddClasses method specifies which classes to include in the scan. The AsSelf method tells Scrutor to register each concrete class as itself, and the WithSingletonLifetime method sets the lifetime of each dependency to scoped.

You can also use the following registrations

As(params Type[] types) and As(IEnumerable<Type> types): Registers each matching concrete type as each of the specified types
As<T>(): Registers each matching concrete type as T
AsImplementedInterfaces(): Registers each matching concrete type as all of its implemented interfaces
AsImplementedInterfaces(Func<Type, bool> predicate): Registers each matching concrete type as all of its implemented interfaces, with a predicate to filter which interfaces to register
AsMatchingInterface(): Registers the type with the first found matching interface name
AsMatchingInterface(Action<Type, IImplementationTypeFilter> action): Registers the type with the first found matching interface name with a filter for the matching type
As(Func<Type, IEnumerable<Type>> selector): Registers each matching concrete type as each of the types returned from the func
UsingAttributes(): Registers each matching concrete type according to their definition of a ServiceDescriptorAttribute

All the scopes supported by Microsoft.Extensions.DependencyInjection are available:

WithSingletonLifetime()
WithScopedLifetime()
WithTransientLifetime()
WithLifetime()

There are multiple ways to add classes:

AddClasses(): Adds all public, non-abstract classes from the selected assemblies
AddClasses(bool publicOnly): Adds all public, non-abstract classes from the selected assemblies with a boolean on whether to add public types only
AddClasses(Action<IImplementationTypeFilter> action): Adds all public, non-abstract classes from the selected assemblies that match the requirements specified in the filtering action
AddClasses(Action<IImplementationTypeFilter> action, bool publicOnly): Adds all public, non-abstract classes from the selected assemblies that match the requirements specified in the filtering action with a boolean on whether to add public types only

There are also multiple ways to target and assembly:

FromCallingAssembly(): Scans for types from the calling assembly
FromExecutingAssembly(): Scans for types from the currently executing assembly
FromEntryAssembly(): Scans for types from the entry assembly
FromApplicationDependencies(): Will load and scan all runtime libraries referenced by the currently executing application. Calling this method is equivalent to calling FromDependencyContext(DependencyContext) and passing in DependencyContext.Default.
FromApplicationDependencies(Func<Assembly, bool> predicate): Will load and scan all runtime libraries referenced by the currently executing application with a predicate filter. Calling this method is equivalent to calling FromDependencyContext(DependencyContext) and passing in DependencyContext.Default.
FromAssemblyDependencies(Assembly assembly): Will load and scan all runtime libraries referenced by the specified assembly
FromDependencyContext(DependencyContext context): Will load and scan all runtime libraries in the given DependencyContext
FromDependencyContext(DependencyContext context, Func<Assembly, bool> predicate): Will load and scan all runtime libraries in the given DependencyContext with a filter predicate
FromAssemblyOf<T>(): Will scan for types from the assembly of T
FromAssemblyOf(params Type[] types) and FomAssembliesOf(IEnumerable<Type> types): Will scan for types from the assemblies of each passed-in type
FromAssemblies(params Assembly[] assemblies) and FromAssemblies(IEnumerable<Assembly> assemblies): Will scan for types from each passed in assembly

By using Scrutor to scan for dependencies, we can reduce the amount of boilerplate code required for DI configuration and ensure that all dependencies are automatically registered with the container.

Scrutor means to explore/scan/search carefully, search into or out, examine thoroughly, explore a thing, investigate to scrutinize.

Open Generics

Scrutor also supports registering open generic types. For example, let’s say we have an interface IFoo<T> and a class Foo<T> that implements it

public interface IFoo<T> { }
public class Foo<T> : IFoo<T> { }

We can register all implementations of IFoo<T> using the AddClasses method with the AssignableTo and AsImplementedInterfaces methods, like so:

services.Scan(scan => scan
    .FromAssemblyOf<Foo<int>>()
    .AddClasses(classes => classes
        .AssignableTo(typeof(IFoo<>))
        .AsImplementedInterfaces()));

Decorators with Scrutor

Another powerful feature of Scrutor is the ability to use decorators to modify or replace registered dependencies. For example, let’s say we have an existing service that we want to enhance with additional functionality. We can do this by registering a decorator that wraps the existing service

Let’s say we have an existing health check implementation:

public class MyHealthCheck : IHealthCheck
{
    public Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
    {
        // Perform health check logic here
    }
}

We want to add logging to this health check without modifying the original code. We can do this by registering a decorator that wraps the existing health check:

services.Decorate<IHealthCheck>((check, serviceProvider) =>
    new LoggingHealthCheckDecorator(check));

The Decorate method takes a delegate that creates the decorator instance, passing in the existing health check and the service provider. The decorator can then modify or replace the existing health check as needed.

Here’s an example of what the LoggingHealthCheckDecorator might look like:

public class LoggingHealthCheckDecorator : IHealthCheck
{
    private readonly IHealthCheck _innerCheck;
    private readonly ILogger<LoggingHealthCheckDecorator> _logger;

    public LoggingHealthCheckDecorator(
        IHealthCheck innerCheck,
        ILogger<LoggingHealthCheckDecorator> logger)
    {
        _innerCheck = innerCheck;
        _logger = logger;
    }

    public async Task<HealthCheckResult> CheckHealthAsync(
        HealthCheckContext context,
        CancellationToken cancellationToken = default(CancellationToken))
    {
        _logger.LogInformation("Checking health");
        var result = await _innerCheck.CheckHealthAsync(context, cancellationToken);
        _logger.LogInformation("Health check result: {Result}", result.Status);
        return result;
    }
}

By using decorators with Scrutor, we can easily enhance or replace existing dependencies without modifying the original code.

Conclusion

Scrutor is a powerful library that complements Microsoft.Extensions.DependencyInjection by providing additional features for scanning and registering dependencies. With Scrutor, we can simplify the DI configuration process and ensure that our code remains modular, testable, and easy to maintain.

By leveraging the power of Scrutor, we can write cleaner, more maintainable code and spend less time managing dependencies.

Effective Git Management Guide

Sat, 01 Apr 2023 19:00:00 GMT

Photo by Yancy Min on Unsplash

Effective Git Management Guide

Commits

✅ DO push your commits often, in case Murphy (Murphy’s Law) decides to visit.

✅ DO break your work into smaller, logical pieces as commits.

This helps with:

Making it possible to cherry-pick and revert specific changes
Making git blame easier when trying to track a change, especially around bug fixes, which will then tie nicely with a matching commit message
Making it possible to review code by commits which can be useful for large Pull Requests

Overall, this approach helps maintain a clean and understandable commit history, making it easier to track the origin of changes and simplifying the review process.

✅ DO ensure each commit makes a single, focused change.

✅ DO use clear and descriptive commit messages that explain the purpose and effect of the change.

✅ DO use the imperative verbs/mood in the subject line. Use a consistent style for your commits, such as starting with a verb and using the present tense.

The idea is that instead of a commit message saying what has been done, it should be read as this is what will happen when the commit is applied. Commit messages should read like chapters in a book, the book being the unit of work that will result in a Pull Request.

Branches Management

✅ DO use a well-defined branching strategy like GitFlow or GitHub Flow to manage the development process effectively and keep the main branch stable.

✅ DO protect critical branches (e.g., main or develop) by applying branch protection rules, such as requiring PR reviews and passing CI checks before merging.

✅ DO use branch prefixes, starting with the name of the committer, a change type like feature, and preferably a task code if you have a tracking system, i.e. john\feature\128952. Most git visual tools will create a hierarchy out of this which makes it easy to navigate and filter, especially at the contributor level.

Examples of useful branch prefixes:

feature: Introduces a new feature
bug or fix: Addresses a bug
boyscout or chore: Performs a change not related to features of fixes such as code formatting, updated dependencies etc. You can also get more granular, i.e. with format, updates, style etc.
refactor: Code specific chore or boy scout
doc: A contribution to documentation
test: Introduces new test coverage or updates existing tests
ci: Changes related to CI artefacts

When it comes to reverts, generally it is better to retain the default message provided by git as there is a lot of context that can be lost by omitting it.

✅ DO clean up remote branches after they are merged to keep the repository clean and organized.

Pull Requests

✅ DO consider PRs as a useful process for reviewing code and ensuring standards are followed.

✅ DO use CI tools to automate linting, coding practices, test runs, and other checks to streamline code reviews and maintain code quality. Do this with PR comments that become regular.

✅ DO include a summary of the changes, the motivation behind them, and any additional context that will help reviewers understand your work.

✅ DO use a pull PR template to enforce the desired structure and information.

Example:

# Description

What are you changing, and what does merging this achieve? Why is this change being made?

# Screenshots

Is this a UX change or can screenshots help the reviewer see the before and after or understand the solution better? Do you have any manual testing evidence?

# Checklist

- [ ] Self-review complete
- [ ] Code builds without errors
- [ ] Relevant tests have been authored/updated
- [ ] All tests are passing locally
- [ ] Relevant documentation updates have been made

✅ DO keep the scope of your PRs limited to make it easier for reviewers to understand and provide feedback.

✅ DO encourage discussions and feedback in PRs, and always be respectful and adhere to your team’s standards and practices.

✅ DO create at minimum a CI pipeline for your PRS to ensure that code compiles, tests and linting succeed among other things.

✅ DO configure the PR to autocomplete with branch deletion.

✅ DO enforce due diligence on your PRs, for example, you can enforce that a PR should have tasks linked to it and have default individuals/groups assigned as reviewers.

✅ DO use labels to categorize and organize pull requests.

✅ DO create a process to ensure the review of PRs is done promptly to avoid blocking other team members’ progress. Not getting this right can be a common reason for a team to not commit to the process.

✅ DO encourage reviewers to provide specific examples or suggestions when requesting changes or improvements.

✅ DO encourage reviewers to be ready to pair with contributors to assist with recommendations they have provided.

✅ DO wire up your messaging system with your DevOps tool so notifications of PRs can be automatically pushed to the team.

Other

✅ DO master your git tool and common git tasks.

🛑 DO NOT allow credentials to leak in git. Leverage tools like Azure Vault and purge the history if credentials are accidentally committed. Also, assume the worst for credentials that leak to your repo and change them.

🎃 DO resolve merge conflicts carefully, ensuring that the intended changes are preserved and the code remains functional. Communicate with your team members when resolving conflicts to avoid misunderstandings.

✅ DO use rebasing to maintain a cleaner, linear commit history that makes it easier to track changes and debug issues by perusing through the history with or without visualisation and when using tools like git bisect.

Rebasing helps to integrate changes from one branch into another by replaying the commits of the feature branch onto the base branch. This creates a linear commit history, which can be easier to understand and navigate. In contrast, merging combines the two branches by creating a new merge commit, which may result in a more complex commit history.

In the following example, we have a long-running branch, feature, that branches off main. After our feature branch, we then have two additional commits from another branch bug that makes their way into the main first.

One way to sync the feature branch is to merge main into feature

This creates noise in the history as the fact that bug got merged first is not an important detail and we do not need to see the commit in feature. If the feature had been started after bug was merged would it have mattered? If the answer is no then you want to lean towards rebasing

Notice how the history now looks cleaner.

🎃 DO BE AWARE that while rebasing is great and encouraged, the potential risks and complications when rebasing shared branches can be quite compromising.

Recommended workflow (simplified)

Handling Large PRs

In instances where you’re dealing with large pull requests, it’s beneficial to break them into smaller, more manageable parts. This approach makes reviewing easier and faster, as each part can focus on a specific feature or issue. When breaking down a PR, aim to ensure that each smaller PR is self-contained and does not introduce incomplete features or dependencies on other PRs. This strategy not only streamlines the review process but also minimizes the risk of introducing bugs due to complex changes being overlooked in a massive PR.

While it’s generally best to avoid large pull requests (PRs) as they can complicate the review process and increase the risk of bugs, there are scenarios where they might be acceptable. For example, changes that are widespread but superficial, such as updates to a package.json file or mass-renaming operations, can be handled in a single large PR. However, for most development work, large PRs should be discouraged due to their complexity and the challenges they present in thorough reviewing and understanding the impact of changes.

One effective strategy for managing large changes is through granular commits. Even if a PR is sizeable, having well-structured, logical commits within it can significantly ease the review process. Each commit should represent a single, coherent change, making it easier for reviewers to understand and track changes step by step. This approach not only facilitates easier code reviews but also aids in future troubleshooting, as each commit stands as a clear record of a specific change.

In cases where large PRs are unavoidable, ensure that they are well-documented, with a clear description of each change and its purpose. Encourage reviewers to focus on individual commits rather than the entire PR at once, to better manage the review process. This granular approach to both commits and PR reviews can help maintain the quality and integrity of the codebase, even when dealing with substantial changes.

Conclusion

In conclusion, effective Git management is pivotal for a disciplined and efficient development workflow. By adhering to best practices for commits, branch management, and pull requests, teams can maintain a clean and understandable Git history, facilitate easier code reviews, and ensure secure and streamlined collaboration. Large pull requests should be approached with caution, broken down into smaller parts when possible. Real-world experiences teach us the value of these practices in maintaining project health and team productivity. Embracing these guidelines will not only enhance your individual skill set but also contribute significantly to the success of your team’s collaborative efforts in software development.

Dotnet must know - Serializable Exceptions

Fri, 17 Mar 2023 10:00:00 GMT

Photo by UX Indonesia on Unsplash

Ensuring Custom Exceptions are Serializable and Implement the Four Core Constructors

This article is part of a series for Dotnet must know

Exceptions are an integral part of any software application, and custom exceptions provide a way to handle application-specific errors in a more specific and meaningful way. However, when creating custom exceptions, it’s important to ensure they are serializable and implement the four core constructors.

Why Is This Important?

When an exception occurs in an application, it may need to be propagated across different tiers or processes or even persisted to a log or database. In such cases, the exception object must be serializable so that it can be transmitted or stored. If the exception object is not serializable, it may not be possible to propagate or store it, resulting in the loss of valuable debugging information.

In modern .NET architectures like distributed systems and microservices, the ability to serialize exceptions becomes increasingly significant. In these environments, exceptions often need to travel across different service boundaries or machine boundaries. For instance, when an exception occurs in a microservice, it might need to be serialized to be sent to a logging service or to be propagated to another service for handling. Without proper serialization, these distributed components would lose the ability to communicate detailed error information effectively, leading to challenges in diagnosing and resolving issues in a distributed architecture.

The four core constructors provide a way to initialize an exception object with the necessary information. The parameterless constructor initializes the exception object with default values, while the constructor that takes a string message initializes it with a user-defined message. The constructor that takes a string message and an inner exception initializes it with a user-defined message and the inner exception that caused the current exception. Finally, the constructor that takes a SerializationInfo object and a StreamingContext object is used to initialize an exception object during deserialization.

Enforcing this rule ensures consistency as the expectations will always be the same for all custom Exceptions.

Enforcing This Using Unit Tests

To ensure that all custom exceptions in an application are serializable and implement the four core constructors, we can use unit tests. Here’s an example of how to do this in xUnit and Fluent Assertions:

/// <summary>
/// At a minimum, you should mark your custom exception as serializable and implement the four basic constructors
/// </summary>
public class ExceptionSerializationTests
{
    [Fact]
    public void GivenAnAssemblyContainingExceptionTypes_ShouldEnsureAllExceptionTypesAreSerializable()
    {
        var assembly = typeof(AssemblyMarker).Assembly;

        assembly.Should().ContainSerializableExceptions();
    }

    [Fact]
    public void GivenAnAssemblyContainingExceptionTypes_ShouldEnsureAllExceptionTypesHaveTheFourBasicConstructors()
    {
        var assembly = typeof(AssemblyMarker).Assembly;

        assembly.Should().ContainExceptionsImplementingAllBaseConstructors();
    }
}

This test checks all custom exceptions in the given assembly to ensure they are marked with the Serializable attribute. It’s crucial because it allows these exceptions to be serialized and deserialized, a necessary feature for error handling across different application domains or services.

This test ensures that each custom exception type has the four essential constructors. These constructors provide flexibility in initializing exceptions with different levels of detail - from a simple message to wrapping another exception as an inner exception.

/// <summary>
/// Extension methods for Fluent Assertions to provide custom assertions for the Assembly type.
/// </summary>
public static class AssemblyAssertionsExtensions
{
    /// <summary>
    /// Asserts that all custom exception types in the assembly are decorated with the Serializable attribute.
    /// </summary>
    /// <param name="assertions">The assembly assertions.</param>
    /// <returns>A chainable assertion object.</returns>
    public static AndConstraint<AssemblyAssertions> ContainSerializableExceptions(
        this AssemblyAssertions assertions)
    {
        var exceptionTypes = assertions.Subject.GetTypes()
            .Where(t => t.IsSubclassOf(typeof(Exception)) && !t.IsAbstract);

        foreach (var exceptionType in exceptionTypes)
        {
            Execute.Assertion.ForCondition(Attribute.IsDefined(exceptionType, typeof(SerializableAttribute)))
                .FailWith(
                    $"Expected exception type '{exceptionType.Name}' to be decorated with SerializableAttribute.");
        }

        return new AndConstraint<AssemblyAssertions>(assertions);
    }

    /// <summary>
    /// Asserts that all custom exception types in the assembly have the four basic constructors: parameterless,
    /// string message, string message and inner exception, and SerializationInfo and StreamingContext.
    /// </summary>
    /// <param name="assertions">The assembly assertions.</param>
    /// <returns>A chainable assertion object.</returns>

    public static AndConstraint<AssemblyAssertions> ContainExceptionsImplementingAllBaseConstructors(
        this AssemblyAssertions assertions)
    {
        var exceptionTypes = assertions.Subject.GetTypes().Where(t => typeof(Exception).IsAssignableFrom(t));

        foreach (var exceptionType in exceptionTypes)
        {
            var constructors =
                exceptionType.GetConstructors(BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.Public);

            Execute.Assertion.ForCondition(constructors.Any(c => c.GetParameters().Length == 0))
                .FailWith($"Expected exception type '{exceptionType.Name}' to have a parameterless constructor.");

            Execute.Assertion.ForCondition(constructors.Any(c =>
                    c.GetParameters().Length == 1 && c.GetParameters()[0].ParameterType == typeof(string)))
                .FailWith(
                    $"Expected exception type '{exceptionType.Name}' to have a constructor that takes a string message.");

            Execute.Assertion.ForCondition(constructors.Any(c =>
                    c.GetParameters().Length == 2 &&
                    c.GetParameters()[0].ParameterType == typeof(string) &&
                    c.GetParameters()[1].ParameterType == typeof(Exception)))
                .FailWith(
                    $"Expected exception type '{exceptionType.Name}' to have a constructor that takes a string message and an inner exception.");

            Execute.Assertion.ForCondition(constructors.Any(c =>
                    c.IsFamily && c.GetParameters().Length == 2 &&
                    c.GetParameters()[0].ParameterType == typeof(System.Runtime.Serialization.SerializationInfo) &&
                    c.GetParameters()[1].ParameterType == typeof(System.Runtime.Serialization.StreamingContext)))
                .FailWith(
                    $"Expected exception type '{exceptionType.Name}' to have a protected constructor that takes a SerializationInfo object and a StreamingContext object.");
        }

        return new AndConstraint<AssemblyAssertions>(assertions);
    }
}

By enforcing these requirements using unit tests, we can ensure that all custom exception types in our application are serializable and can be properly initialized with the necessary information. This helps us avoid potential bugs and loss of valuable debugging information in the event of an exception.

Best Practices in Designing Custom Exceptions

While ensuring exceptions are serializable and implementing the core constructors are foundational, there are additional best practices to consider for robust custom exception design:

Meaningful Properties: Include properties that carry relevant information about the error condition. For example, if you have a FileReadException, consider adding properties like FileName or FileLocation.
Exception Hierarchies: Establish a clear hierarchy of exceptions, especially in larger applications. This helps in categorizing and handling exceptions at different levels.
Avoid Excessive Details: While detailed exceptions are helpful, avoid including sensitive information like database credentials or encryption keys in the exception details.
Documentation: Clearly document each custom exception, specifying when and how it should be used. This clarity is vital for anyone else using or maintaining your code.

By adhering to these practices, you not only enhance the functionality of your custom exceptions but also improve the maintainability and clarity of your error handling logic.”

Incorporating these enhancements will provide a more comprehensive understanding for readers, making the blog not only a guide but also a best practices reference in the realm of .NET exception handling.

Must know Roslyn rules - Cancellation Tokens

Fri, 17 Mar 2023 10:00:00 GMT

Photo by Tingey Injury Law Firm on Unsplash

Must know Roslyn rules - Cancellation Tokens

This article is part of a series for Must know Roslyn rules

CA1068: CancellationToken parameters must come last

This rule suggests that CancellationToken parameters should be placed at the end of the parameter list. This is because CancellationToken is a special type of parameter that is used to signal cancellation of a long-running operation. By convention, it is placed at the end of the parameter list to make it clear that it is an optional parameter. We can configure the rule as follows in an .editorconfig file.

dotnet_diagnostic.CA1068.severity = error
dotnet_diagnostic.CA1068.exclude = *_generated.cs

In this example, we’ve set the rule to error, which fails the build on violation. We’ve also excluded generated files from this rule using the exclude property.

CA2016: Forward the CancellationToken parameter to methods that take one

CA2016 is a code analysis rule in Roslyn that checks whether a method that performs a long-running operation takes a CancellationToken parameter, and if it does, whether it forwards that parameter to any other methods that also take a CancellationToken parameter.

This rule suggests that methods that take a CancellationToken parameter should forward that parameter to any other methods that also take a CancellationToken parameter. This can help ensure that a long-running operation can be cancelled gracefully if needed.

Let’s look at an example. Consider the following method:

public async Task<int> DoLongRunningOperationAsync(CancellationToken cancellationToken)
{
    // Do something that takes a long time
    await Task.Delay(5000);

    // Return a value
    return 42;
}

This method takes a CancellationToken parameter but does not forward it to any other methods. If we run the code analysis tool in Visual Studio, we’ll get a warning that suggests we should forward the CancellationToken parameter to any other methods that also. We can fix the violation.

public async Task<int> DoLongRunningOperationAsync(CancellationToken cancellationToken)
{
    // Do something that takes a long time, and pass the CancellationToken parameter to Task.Delay
    await Task.Delay(5000, cancellationToken);

    // Return a value
    return 42;
}

In this updated implementation, the cancellationToken parameter is passed to the Task.Delay method, which also takes a CancellationToken parameter. This ensures that the long-running operation can be cancelled gracefully if needed.

To configure the CA2016 rule in an .editorconfig file, we can use the following settings:

dotnet_diagnostic.CA2016.severity = error
dotnet_diagnostic.CA2016.exclude = *_generated.cs

Conclusion

While the CA1068 and CA2016 rules are independent of each other and can be configured separately, there is value in configuring them together to ensure that all method signatures are consistent and readable.

The CA1068 rule has exceptions for methods with optional parameters or ref/out parameters following a CancellationToken parameter. While these exceptions make sense for those specific scenarios, it’s still a good idea to keep method signatures consistent and easy to read. For example, even if a method has optional parameters following a CancellationToken parameter, we might still want to keep the CancellationToken parameter at the end of the parameter list to make it clear that it is optional and to maintain consistency with other methods that have a CancellationToken parameter

Consuming SOAP Services with the HttpClient

Thu, 09 Mar 2023 10:00:00 GMT

Photo by Aurélia Dubois on Unsplash

Consuming Legacy SOAP Services with the HttpClient in .NET

In this article, we will discuss how to consume legacy SOAP services using the HttpClient in .NET. The HttpClient class is used to send HTTP requests to a web server and receive its responses. We will see how to use it to send SOAP requests to a SOAP service and process its responses.

Introduction to SOAP Services

SOAP (Simple Object Access Protocol) is a messaging protocol used to exchange structured data between applications. It relies on XML to provide a common message format that can be understood by different systems, regardless of their programming languages and platforms.

In recent years, SOAP has been superseded by more modern protocols such as REST and gRPC. However, there are still many legacy SOAP services that are widely used and need to be integrated into modern applications.

In this blog post, I will show you how to consume a legacy SOAP service using the HttpClient in .NET. We will use a SOAP service that converts a given number to words as an example.

Overview of our example

Our SOAP service, called NumberToWords, takes an integer as input and returns its English representation in words. For example, if the input is 500, the service will return “five hundred”.

NumberConversion has a service, NumberToWords, which returns the word corresponding to the positive number passed as parameter limited to quadrillions. Due to its simplicity we will use it in this example

To call this service, we need to send an XML request message in the SOAP format to the service endpoint. The service will then return an XML response message in the SOAP format containing the result.

Here is an example input:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <NumberToWords xmlns="http://www.dataaccess.com/webservicesserver/">
      <ubiNum>500</ubiNum>
    </NumberToWords>
  </soap:Body>
</soap:Envelope>

and matching output:

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <m:NumberToWordsResponse xmlns:m="http://www.dataaccess.com/webservicesserver/">
            <m:NumberToWordsResult>five hundred </m:NumberToWordsResult>
        </m:NumberToWordsResponse>
    </soap:Body>
</soap:Envelope>

Consuming SOAP Services with HttpClient

To consume a SOAP service with the HttpClient, we need to create an XML request message that follows the SOAP protocol and send it to the SOAP service using an HTTP POST request. The response message is also an XML message that follows the SOAP protocol.

To create the SOAP request message, we need to create an XML document that follows the SOAP protocol and contains the SOAP envelope, header, and body. The body of the SOAP message contains the actual request data that is sent to the service.

We can use the HttpClient class to send the SOAP request message to the service using an HTTP POST request. To process the response message, we need to deserialize the XML response message into a .NET object. We can use the XmlSerializer class to deserialize the XML message into a .NET object

Sample implementation

The code provided in this blog post is written in C#. Let us start by looking at the classes involved in consuming the NumberToWords service.

public class NumberToWordsRequestBody
{
    [XmlElement(Namespace = Constants.NumberConversionNamespace, ElementName = "NumberToWords")]
    public NumberToWordsRequest Request { get; set; }
}

public class NumberToWordsRequest
{
    [XmlElement(ElementName = "ubiNum")]
    public int Input { get; set; }
}


public class NumberToWordsResponseBody
{
    [XmlElement(ElementName = "NumberToWordsResponse", Namespace = Constants.NumberConversionNamespace)]
    public NumberToWordsResponse Response { get; set; }
}

public class NumberToWordsResponse
{
    [XmlElement(ElementName = "NumberToWordsResult")]
    public string Result { get; set; }
}

[XmlType(Namespace = Constants.EnvelopeNamespace, TypeName = "Envelope")]
public class Envelope<T>
{
    public T Body { get; set; }
}

The NumberToWordsRequestBody and NumberToWordsRequest classes represent the input message. The NumberToWordsResponseBody and NumberToWordsResponse classes represent the output message. The Envelope<T> class is a generic class used to wrap the input and output messages.

Next, let us look at a sample client, NumberConversionClient, which is responsible for making the HTTP request to the NumberToWords service.

public class NumberConversionClient : BaseClient
{
    private const string ApiPath = "NumberConversion.wso";

    public NumberConversionClient(IHttpClientFactory httpClientFactory, ILogger<NumberConversionClient> logger)
        : base(httpClientFactory, logger)
    {
    }

    public async Task<string> ConvertNumberToWords(int number, CancellationToken cancellationToken = default)
    {
        var message = new Envelope<NumberToWordsRequestBody>
        {
            Body = new NumberToWordsRequestBody
            {
                Request = new NumberToWordsRequest
                {
                    Input = number
                }
            }
        };

        var result =
            await Post<Envelope<NumberToWordsRequestBody>, Envelope<NumberToWordsResponseBody>>(ApiPath, message,
                cancellationToken);

        return result.Body.Response.Result;
    }
}

public class BaseClient
{
    private readonly ILogger _logger;

    protected readonly HttpClient HttpClient;

    public BaseClient(IHttpClientFactory clientFactory, ILogger logger)
    {
        HttpClient = clientFactory.CreateClient(nameof(BaseClient));
        _logger = logger ?? throw new ArgumentNullException(nameof(logger));
    }

    protected async Task<TResponse> Post<TRequest, TResponse>(string path, TRequest body,
        CancellationToken cancellationToken = default) =>
        await DeserializeResponse<TResponse>(
            await GetApiResponse(HttpMethod.Post, path, body: body, cancellationToken: cancellationToken),
            path,
            cancellationToken);

    protected virtual async Task<HttpResponseMessage> GetApiResponse<TRequest>(HttpMethod method, string path,
        TRequest body = default, CancellationToken cancellationToken = default)
    {
        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            ["path"] = path
        });

        var httpRequest = new HttpRequestMessage(method, path);

        if (body != null)
            httpRequest.Content =
                new StringContent(body.Serialize(), Encoding.UTF8, Constants.Format);

        return await HttpClient.SendAsync(httpRequest, cancellationToken);
    }

    protected virtual async Task<TResponse> DeserializeResponse<TResponse>(HttpResponseMessage httpResponse,
        string path, CancellationToken cancellationToken)
    {
        var statusCode = (int)httpResponse.StatusCode;

        using var scope = _logger.BeginScope(new Dictionary<string, object>
        {
            [nameof(statusCode)] = statusCode,
            ["responseType"] = typeof(TResponse).FullName,
        });

        if (httpResponse.IsSuccessStatusCode)
        {
            try
            {
                var xml = HttpUtility.HtmlDecode(await httpResponse.Content.ReadAsStringAsync(cancellationToken));

                return xml.Deserialize<TResponse>();
            }
            catch (Exception exception)
            {
                const string deserializeFailureMessage = "Could not deserialize the response input stream";
                _logger.LogError(deserializeFailureMessage);
                throw new DomainException($"{deserializeFailureMessage} {typeof(TResponse).FullName}", exception);
            }
        }

        var content = await httpResponse.Content.ReadAsStringAsync(cancellationToken);
        _logger.LogError(
            "SOAP responses should always return OK status code with details. Received {statusCode} {content}",
            statusCode, content);

        throw new DomainException("SOAP responses should always return OK status code with details");
    }
}

The code relies on some extensions. SoapExtensions is a static class that contains extension methods to help with the serialization and deserialization of SOAP requests and responses. It consists of the following methods:

ToXDocument: This method takes an object of type T and returns an instance of XDocument. The method first creates a new instance of XDocument with the necessary XML declaration and then uses an XmlSerializer to serialize the input object into an XML document. If any errors occur during serialization, a DomainException is thrown.
Serialize: This method takes an object of type T and returns a string representing the serialized XML document. It simply calls the ToXDocument method and returns the resulting XDocument as a string.
Deserialize: This method takes a string representing an XML document and deserializes it into an object of type T. The method first creates an XmlSerializer instance and then uses it to deserialize the input XML string into an object of type T. If any errors occur during deserialization, a DomainException is thrown.

By defining these extension methods, we can easily serialize and deserialize SOAP requests and responses using simple method calls.

For completeness here are the constants referenced in the other code snippets.

public static class Constants
{
    public const string TextXmlContentType = "text/xml; charset=utf-8";
    public const string Format = "text/xml";
    public const string EnvelopeNamespace = "http://schemas.xmlsoap.org/soap/envelope/";

    public const string Version = "1.0";
    public const string Standalone = "1.0";
    public const string Encoding = "1.0";

    public const string NumberConversionNamespace = "http://www.dataaccess.com/webservicesserver/";
}

Testing our sample

Let’s write an integration test for the client that we have just created. First, let’s introduce some configuration options.

public class NumberConversionOptions
{
    public const string Key = "NumberConversion";

    public Uri BaseAddress { get; set; }
    public TimeSpan RequestTimeOut { get; set; }
}

public class NumberConversionOptionsValidator : AbstractValidator<NumberConversionOptions>
{
    public NumberConversionOptionsValidator()
    {
        RuleFor(m => m.BaseAddress)
            .Must(x => x.IsAbsoluteUri && x.Scheme == Uri.UriSchemeHttps);
    }
}

public static class NumberConversionOptionsValidatorExtensions
{
    public static void ValidateAndThrow(this NumberConversionOptions x)
        => new NumberConversionOptionsValidator().ValidateAndThrow(x);
}

The NumberConversionOptions class defines the configuration options for the SOAP service. It has two properties, BaseAddress and RequestTimeOut. The BaseAddress property is the base URI for the service, while the RequestTimeOut property specifies the timeout for the HTTP request.

The NumberConversionOptionsValidator class is used to validate the configuration options specified in the NumberConversionOptions class. It uses the FluentValidation library to ensure that the BaseAddress property is an absolute HTTPS URI.

The NumberConversionOptionsValidatorExtensions class provides an extension method that allows us to easily validate an instance of the NumberConversionOptions class using the ValidateAndThrow method. If the options are invalid, an exception is thrown.

Our configuration can look as follows in JSON:

{
  "NumberConversion": {
    "BaseAddress": "https://www.dataaccess.com/webservicesserver/",
    "RequestTimeOut": "00:01:00"
  }
}

Next, let us add an extension for bootstrapping the client into a service collection.

public static class NumberConversionExtensions
{
    public static IServiceCollection AddHttpClient(this IServiceCollection services,
        IConfigurationSection config)
    {
        if (config == null)
            throw new ArgumentNullException(nameof(config));

        services.Configure<NumberConversionOptions>(config);

        var options = config.Get<NumberConversionOptions>();

        options.ValidateAndThrow();

        services.AddHttpClient(nameof(BaseClient), (_, client) =>
        {
            client.BaseAddress = options!.BaseAddress;
            client.Timeout = options.RequestTimeOut;
            client.DefaultRequestHeaders.Add("Accept", Constants.TextXmlContentType);
        });

        services.AddTransient<NumberConversionClient>();

        return services;
    }
}

The NumberConversionExtensions class provides an extension method to bootstrap the NumberConversionClient. The AddHttpClient method takes an IServiceCollection and an IConfigurationSection as parameters. It configures an instance of NumberConversionOptions and registers it with the service collection.

Next, the method validates and throws an exception on invalid options, using the validator we created earlier with FluentValidation. Then it creates and configures an instance of HttpClient with the base address and timeout values provided in NumberConversionOptions. Finally, it registers an instance of NumberConversionClient with the service collection, which allows it to be used in the application.

Before we can write integration tests, we need to create a fixture. A fixture is a reusable object that contains test data, configuration settings, and any other objects required by the tests. Fixtures can help reduce code duplication and make tests easier to read and maintain.

In our case, we’ll create a fixture that sets up the HttpClient and other required dependencies, such as the NumberConversionOptions. We’ll use the AutoMocker library to help us create and inject these dependencies.

[CollectionDefinition(nameof(NumberConversionCollectionFixture))]
public class NumberConversionCollectionFixture : ICollectionFixture<NumberConversionFixture>
{
}

public class NumberConversionFixture : IAsyncLifetime
{
    public Task InitializeAsync()
    {
        var configSection = GetConfigurationSection(out var configuration);

        var (httpClientFactory, options) = CreateHttpClient(configSection);

        AutoMocker = new AutoMocker();
        AutoMocker.Use(httpClientFactory);
        AutoMocker.Use(options);
        AutoMocker.Use(configuration);

        return Task.CompletedTask;
    }

    private static IConfigurationSection GetConfigurationSection(out NumberConversionOptions configuration)
    {
        var configSection =
            TestConfigurationExtensions.GetConfiguration<NumberConversionFixture>()
                .GetSection(NumberConversionOptions.Key);
        configuration = configSection.Get<NumberConversionOptions>();

        return configSection;
    }

    public Task DisposeAsync() => Task.CompletedTask;

    public AutoMocker AutoMocker { get; set; }

    private static (IHttpClientFactory, IOptions<NumberConversionOptions>) CreateHttpClient(
        IConfigurationSection configSection)
    {
        var serviceCollection = new ServiceCollection();
        serviceCollection.AddHttpClient(configSection);

        var provider = serviceCollection.BuildServiceProvider();

        return (provider.GetService<IHttpClientFactory>(),
            provider.GetService<IOptions<NumberConversionOptions>>());
    }
}

This code block defines two classes that are used to initialize and dispose of the dependencies required for testing NumberConversionClient.

NumberConversionCollectionFixture is a collection fixture that provides the context for the tests. This fixture is used to set up the HttpClient and the dependencies for NumberConversionClient. The NumberConversionFixture implements IAsyncLifetime, which provides the methods for setting up and tearing down the dependencies.

To ensure that our SOAP service client works as expected, we need to write integration tests that will verify the client’s behaviour. For this, we’ll use xUnit.net as our test framework.

[Collection(nameof(NumberConversionCollectionFixture))]
[Trait("TestCategory", TestCategories.Integration)]
public class NumberConversionClientTests
{
    private readonly NumberConversionFixture _fixture;

    public NumberConversionClientTests(NumberConversionFixture fixture) => _fixture = fixture;

    [Fact]
    public async Task GivenValidInput_ThenConvertNumberToWords_ShouldReturnInWords()
    {
        var sut = _fixture.AutoMocker.CreateInstance<NumberConversionClient>();

        var result = await sut.ConvertNumberToWords(500);

        const string expected = "five hundred ";

        result.Should().NotBeNullOrEmpty();
        result.Should().Be(expected);
    }
}

First, we created a test class called NumberConversionClientTests and marked it with the [Collection] attribute. This attribute is used to indicate that the tests in this class require a shared resource, in our case the NumberConversionFixture. This fixture will set up the required infrastructure to create an instance of the NumberConversionClient class.

Next, we added a test method called GivenValidInput_ThenConvertNumberToWords_ShouldReturnInWords that will test our SOAP client’s ability to convert a number to words. In this method, we’ll create an instance of the NumberConversionClient class using the AutoMocker instance provided by the fixture. We’ll then call the ConvertNumberToWords method with a valid input value of 500 and verify that the result is five hundred using the FluentAssertions library.

Running this integration test will ensure that our SOAP service client is working as expected and can be used in our application with confidence.

Conclusion

In this article, we discussed how to consume legacy SOAP services using the HttpClient in .NET. We saw how to create the SOAP request message and send it to the SOAP service using an HTTP POST request. We also saw how to deserialize the XML response message into a .NET object.

By using the HttpClient class, we could easily integrate with SOAP services in .NET and process their responses. While SOAP services are being phased out in favour of RESTful services, there are still many legacy systems that use SOAP services, and it is important to know how to integrate with them using modern tools and technologies.

Bonus Section - Working with CData

CDATA (short for Character Data) is a way to include unescaped character data, including markup characters such as <, >, and &, within an XML document. CDATA sections begin with the string <![CDATA[ and end with ]]>. Within a CDATA section, characters are not parsed as markup, but are instead treated as normal character data.

Using CDATA sections can be useful when including text that contains reserved characters or markup that should be preserved as-is, without being parsed as XML. In the context of SOAP web services, CDATA can be used to include XML fragments in messages without causing parsing errors or losing information.

    public class SampleObject
    {
        [XmlElement(ElementName = "CDataSection")]
        public XmlCDataSection Transaction
        {
            get
            {
                var xDocument = SampleNestedObjectInternal.ToXDocument();
                xDocument.RemoveNamespaceAttributes();

                return new XmlDocument().CreateCDataSection(xDocument.ToString());
            }
            set => _ = value; // Without an explicit setter the serializer ignores this
        }

        [XmlIgnore]
        public SampleNestedObject SampleNestedObjectInternal { get; set; }
    }

    public class SampleNestedObject
    {
        [XmlElement(ElementName = "Name")]
        public string Name { get; set; }
    }

This code shows an example of how to serialize an object to XML and include a property that contains a CDATA section. In this case, the object being serialized is SampleObject, which contains a nested object of type SampleNestedObject. This code can be useful when working with SOAP services that expect CDATA sections in their XML payloads. The Transaction field in the SampleObject is required to be CDATA and to achieve this we create a property for the SampleNestedObject with the XmlIgnore attribute, then a separate utility that does the bidding to CDATA

If we were to serialize this, we would get the following xml

<SampleObject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.xmlsoap.org/soap/envelope/">
    <CDataSection><![CDATA[<SampleNestedObject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://schemas.xmlsoap.org/soap/envelope/">
    <Name>Dummy</Name>
  </SampleNestedObject>]]></CDataSection>
</SampleObject>

Notice that we used the extension RemoveNamespaceAttributes, to avoid getting an empty namespace sections on the CData section, specifically getting back the following:

<SampleObject xmlns:xsi=" http:// www.w3.org/ 2001/ XMLSchema-instance"
    xmlns:xsd=" http:// www.w3.org/ 2001/ XMLSchema"
    xmlns=" http:// schemas.xmlsoap.org/ soap/ envelope/">
    <CDataSection><![CDATA[<SampleNestedObject>
    <Name>Dummy</Name>
  </SampleNestedObject>]]></CDataSection>
</SampleObject>

This uses the following extension method:

    public static void RemoveNamespaceAttributes(this XDocument xDocument)
    {
        xDocument.Descendants()
            .Attributes()
            .Where(x => x.IsNamespaceDeclaration)
            .Remove();

        foreach (var elem in xDocument.Descendants())
            elem.Name = elem.Name.LocalName;
    }

The method works by using LINQ to iterate over all elements in the document, removing the namespace attributes, and then setting the element name to only the local name (i.e., without the namespace). This essentially flattens the document, removing all references to namespaces.

Note that removing namespace attributes can have unintended consequences, such as removing the ability to distinguish between two elements with the same local name in different namespaces. Therefore, this method should be used with caution and only when necessary.

How to Create SOAP Requests Using Postman

Thu, 09 Mar 2023 10:00:00 GMT

Photo by Mika Baumeister on Unsplash

How to Create SOAP Requests Using Postman

Postman is a popular tool used to test APIs and make HTTP requests. It can also be used to make SOAP requests, which is a messaging protocol used to exchange structured information between web applications. In this blog post, I will guide you on how to create SOAP requests using Postman.

Step 1: Enter Your SOAP Endpoint

To make a SOAP request, you will need to enter your SOAP endpoint URL in the address field of a new request tab in Postman. Select the POST method from the request method dropdown list. For example, we will use the following endpoint URL: https://www.dataaccess.com/webservicesserver/NumberConversion.wso

NumberConversion has a service, NumberToWords, which returns the word corresponding to the positive number passed as parameter limited to quadrillions. Due to its simplicity we will use it in this example

Step 2: Add Body Data

In the Body tab, select raw and choose XML from the dropdown list. Enter your XML in the text entry area. Your request body must include the SOAP Envelope, Header, and Body tags as required by the endpoint, as well as any namespaces. The data needs to include the name of the operation, together with any values you need to post to the service.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <NumberToWords xmlns="http://www.dataaccess.com/webservicesserver/">
      <ubiNum>500</ubiNum>
    </NumberToWords>
  </soap:Body>
</soap:Envelope>

Step 3: Set Your Request Headers

When you select an XML body type, Postman automatically adds a Content-Type header of application/xml. However, depending on your service provider, you may need text/xml for some SOAP requests. Check with your SOAP service to decide which header is appropriate. If you need the text/xml header or otherwise, you will need to override the default setting added by Postman.

You can also include the character set in the Content-Type header, for example, text/xml; charset=utf-8

For the number conversion SOAP API example, you need to change the content type header to text/xml. To set request headers, do the following:

Open the request Headers. If the auto-generated headers are hidden, select the notice to display them.
Deselect the Content-Type header Postman added automatically.
Add a new row with Content-Type in the Key field and text/xml in the Value field.

Step 4: Send Your Request

Select Send to make your call to the SOAP service. If your call is successful, Postman displays the response in the lower tab. You can inspect the response to verify that your SOAP request was processed correctly.

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
    <soap:Body>
        <m:NumberToWordsResponse xmlns:m="http://www.dataaccess.com/webservicesserver/">
            <m:NumberToWordsResult>five hundred </m:NumberToWordsResult>
        </m:NumberToWordsResponse>
    </soap:Body>
</soap:Envelope>

In conclusion, Postman makes it easy to create SOAP requests and test SOAP APIs. By following the steps outlined in this blog post, you can make SOAP requests in Postman with ease.

Here is the complete curl you can import in Postman to skip the steps highlighted.

curl --location 'https://www.dataaccess.com/webservicesserver/NumberConversion.wso' \
--header 'Content-Type: text/xml; charset=utf-8' \
--data '<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <NumberToWords xmlns="http://www.dataaccess.com/webservicesserver/">
      <ubiNum>500</ubiNum>
    </NumberToWords>
  </soap:Body>
</soap:Envelope>
'

Clean Architecture

Tue, 07 Mar 2023 10:00:00 GMT

Photo by Kimon Maritz on Unsplash

Clean Architecture

This article summarizes the principles of Clean Architecture based on my own experiences using this architecture in different scenarios. Robert C. Martin’s book on this topic was an anchor to my understanding of this topic, especially having discovered the book after working on the architecture for a while. I recently read the book again as a refresher before creating this article and therefore this article serves as a summary of the good parts and additions from my own experiences.

Architecture

When considering architecture, it is crucial to keep in mind several important factors.

First and foremost, the system’s life cycle should be supported, which involves ensuring that it can be maintained, upgraded, and expanded as needed with minimal complications.
Additionally, the system’s lifetime cost should be minimized to keep it viable for as long as possible. A well-designed architecture should also prioritize maximizing programmer productivity by enabling them to work more efficiently and effectively.
Finally, the architecture should make the operation of the system apparent to developers, who need a clear understanding of how the system works and how it is intended to be used. This is achieved by elevating the use cases, features, and required behaviours of the system to first-class entities that serve as visible landmarks for developers.

It is important to recognize that among all aspects of a software system, maintenance tends to be the most expensive. This is mainly due to two factors: spelunking and risk.

©️ Photo by Bobby Stevenson on Unsplash

Spelunking involves the cost of thoroughly examining the existing software to identify the most suitable approach for implementing new features or resolving defects. However, such changes can also raise the risk of inadvertently introducing new defects, thereby contributing to the overall cost.

Good Architecture

A sound architecture seeks to maintain as many options open for as long as possible. In light of this, it is critical to note that all decisions are trade-offs, and the context plays a pivotal role. An architecture that enables deferring decisions is generally favourable. The elements that require extended options include insignificant details such as database implementation specifics, web services, frameworks, and communication protocols.

Effective architecture renders the system simple to comprehend, develop, maintain, and deploy.

A well-designed architecture should be expressive (screams 😱), drawing attention to critical boundaries and highlighting use cases.

Important terminology

Policy vs Details

To ensure clarity in our discussion, it’s important to establish some recurring terminology.

In all software systems, two fundamental components can be identified: policy and details.

The policy encompasses all the business rules and procedures that represent the real value of the system.

The details refer to the elements required to interact with the policy but do not have an impact on the policy’s behaviour. These may include database implementations, web systems, and communication protocols.

The objective is to prioritize the policy component as the most vital part of the system while ensuring that the details are irrelevant to the policy’s behaviour. This approach enables decisions about the details to be postponed.

Clean Architecture Brief

Clean Architecture aims to create systems that are:

Independent of frameworks
Independent of the user interface
Independent of the database
Independent of any external agencies for that matter like 3rd party services you may consume via SDKs or APIs
Testable, more specifically, easily testable
Easy to understand, build and maintain. It should be easy for developers to comprehend how the system works, and the design should facilitate changes and updates without introducing undue complexity

S.O.L.I.D principles

To have a complete understanding and appreciation of Clean Architecture, it’s crucial to have a solid foundation in certain building blocks. Therefore, we will take a comprehensive approach, beginning with the SOLID principles.

Clean code is fundamental to building effective software systems, and the SOLID principles are instrumental in achieving this goal.

The SOLID principles are a set of guidelines that aim to create modules in a software system that can be changed with minimal cost or friction, are easy to understand, and have a high potential for reuse.

SOLID is an acronym for five principles, which are:

Single Responsibility Principle (SRP) - A module should have only one reason to change, meaning it should have only one responsibility.
Open/Closed Principle (OCP) - A module should be open for extension but closed for modification. This means that the behaviour of a module can be extended without changing its source code.
Liskov Substitution Principle (LSP) - A sub-type should be substitutable for its base type. This means that any object of a base class should be replaceable with an object of its derived class without altering the correctness of the program.
Interface Segregation Principle (ISP) - A client should not be forced to implement interfaces they do not use. This means that the interface of a module should be specific to the needs of the client.
Dependency Inversion Principle (DIP) - High-level modules should not depend on low-level modules. Both should depend on abstractions. This means that a module should not depend on the details of another module but rather on its abstractions. It’s important to note that the emphasis is on inversion and not injection in DIP. We will cover this in more detail later.

Single Responsibility Principle

The Single Responsibility Principle (SRP) is often misunderstood and incorrectly defined as A module must do one and only one thing. The principle is derived from Conway’s law, which suggests that software systems are heavily influenced by the social structures of the organization that uses them.

The correct definition of SRP is that a module should have only one reason to change, meaning that it should only honour one actor. In other words, a module should have a single responsibility, but that responsibility may involve multiple functions.

©️ Photo by Erik Mclean on Unsplash

In the given example, there is a violation of the Single Responsibility Principle, as there is a coupling between concerns from different actors.

public class Customer
{
  // data fields

  void CalculateDeliveryCost() {}

  dynamic ReportNextDelivery () {}

  void Save() {}
}

The Calculate Delivery Cost function may be intended for the accounting department, which reports to the CFO. The Get Next Delivery Details function may be intended for the delivery department, which reports to the COO. The Save function may be specified by the database administrators who report to the CTO.

The proximity of the code may encourage premature reuse. For example, the calculation of delivery costs may depend on a method that filters deliveries. The delivery team may choose to modify the shared code to exclude deliveries from a collection centre in a postal code that already has other deliveries due on the same day. However, this change may have unintended consequences for accounting, resulting in incorrect reporting and potentially leading to a disgruntled CFO.

This example illustrates the importance of adhering to the Single Responsibility Principle to avoid such coupling of concerns from different actors and the resulting problems.

public class Customer
{
    // data fields
}

public interface DeliveryCostCalculator
{
   void CalculateDeliveryCost(Customer customer);
}

public interface IDeliveryReporter
{
   dynamic ReportNextDelivery(Customer customer);
}

public interface IEmployeeRepository
{
   void Save(Customer customer);
}

To address the violation of the Single Responsibility Principle in the previous example, one solution could be to adopt an anaemic model, which is favoured in functional programming. In this approach, the behaviour is offloaded from the data, and each component has a single responsibility.

For example, instead of having a Delivery object with methods like CalculateDeliveryCost and GetNextDeliveryDetails, we could have a DeliveryData object that only holds the delivery data and separate classes that handle the behaviours associated with the data.

public class CalculateDeliveryCostUseCase
{
    public Execute(IDeliveryCostCalculator deliveryCostCalculator) {}
}

public class ReportNextDeliveryUseCase
{
    public Execute(IDeliveryReporter deliveryReporter) {}
}

In the example of using an anaemic model to adhere to the Single Responsibility Principle, we can offload the orchestration of the new moving parts into a use case. The use case is a separate component that is responsible for handling the interactions between the different parts of the system, making the actions explicit and enforcing SRP through a single Execute method.

Although it’s possible to directly call through the newly added abstraction, it’s often beneficial to use a use case to orchestrate the interactions between the components. This is because use cases are great at handling application-specific concerns and orchestrating complex interactions.

For example, imagine that we want to send emails after the delivery cost is calculated. Ideally, this should not be added to the DeliveryCostCalculator as it can make the component less reusable. A use case is better suited for this scenario as it can handle this application-specific concern without dirtying the DeliveryCostCalculator.

Therefore, services should be kept generic, while use cases can be tailored to specific application needs. By using use cases, we can achieve a more modular, maintainable, and reusable system.

This would result in each component having a single reason to change, making the system more modular and easier to maintain. By offloading the behaviour from the data, we can achieve a more clear separation of concerns and better adhere to the Single Responsibility Principle.

Open-Closed Principle

The Open/Closed Principle (OCP) states that software systems should be designed to allow for changes in behaviour by adding new code, rather than modifying existing code. The idea is to make the system easy to extend without having a significant impact on the existing codebase.

To achieve this goal, the system should be partitioned into components, and these components should be organized into a dependency hierarchy that protects higher-level components from changes in lower-level components. For instance, the business rules should remain unaffected when switching to a different database implementation.

Would you knock down the Burj Khalifa to install a sewage system? The answer is obviously no! Yet as software engineers, we knock down the Burj Khalifa all the time. This analogy highlights the importance of preserving the existing codebase while still allowing for changes and extensions. A well-designed system should strike a balance between preserving the existing code and allowing for future enhancements. This is where the OCP comes into play, as it provides a guideline for designing systems that are easy to extend without having to modify the existing codebase.

©️ Photo by Mohit Kumar on Unsplash

In the given example, we have an Order class with a method called CalculateTotal, which calculates the total cost of an order. However, the method violates the Open/Closed Principle because it’s not closed for modification.

public class Order
{
    public decimal CalculateTotal()
    {
        decimal total = 0;
        if (TotalAmount > 1000)
        {
            total -= TotalAmount * 0.05m;
        }
        if (DiscountPercentage > 0)
        {
            total -= TotalAmount * DiscountPercentage / 100;
        }
        return total;
    }
}

If we want to add a new type of discount, such as a “buy one, get one free” offer, we would need to modify the CalculateTotal method to include this new type of discount. This violates the OCP because we are modifying existing code rather than extending it.

To adhere to the OCP, we could introduce a Discount class that implements a Discount interface, using the strategy pattern. The Discount interface could have a Calculate method, which calculates the discount based on the order. The CalculateTotal method in the Order class could then be modified to use the Discount interface, allowing for the addition of new discount types without modifying the Order class.

public interface IDiscountStrategy
{
    decimal CalculateDiscount(decimal totalAmount);
}

public class Order
{
    public decimal CalculateTotal()
    {
        decimal total = 0;
        if (DiscountStrategy != null)
        {
            total -= DiscountStrategy.CalculateDiscount(TotalAmount);
        }
        return total;
    }
}

public class PercentageDiscountStrategy : IDiscountStrategy
{
    public decimal Percentage { get; set; }
    public decimal CalculateDiscount(decimal totalAmount)
    {
        // calculate the discount for percentage offer
    }
}

public class BuyOneGetOneFreeDiscountStrategy : IDiscountStrategy
{
    public decimal CalculateDiscount(decimal totalAmount)
    {
        // calculate the discount for buy one, get one free offer
    }
}

By doing this, we ensure that the Order class is closed for modification and open for extension, as new discount types can be added by implementing the Discount interface, without changing the Order class itself. This makes the system more modular, flexible, and easier to maintain.

Let’s look at another violation. In the given example of a reporting service that pulls data out of a database, massages the data, and sends out an email report with an excel file attached, there is a violation of the Open/Closed Principle.

Suppose we want to display the reports on the web or print them out. In that case, we would need to modify the existing code to accommodate these changes, violating the OCP.

To adhere to the OCP, we can introduce a Report class that takes care of generating the report data, and separate ReportFormatter classes for each output format (e.g., EmailReportFormatter, WebReportFormatter, PrintReportFormatter). The ReportFormatter classes implement a ReportFormatter interface that has a format method. The format method takes in the Report data and generates the output in the desired format.

By doing this, we can add new output formats without modifying the existing Report class, making the system closed for modification and open for extension. This approach makes the system more modular, flexible, and easier to maintain.

Liskov Substitution Principle

The Liskov Substitution Principle (LSP) states that objects of a superclass should be replaceable with objects of its subclasses without affecting the correctness of the program. The principle emphasizes the importance of adhering to a contract that allows the parts to be substituted one for another to build software systems from interchangeable parts.

By following the LSP, we can avoid pollution in the codebase when multiple implementations follow a contract. In other words, we can ensure that different implementations of the same contract behave in the same way, and they can be used interchangeably.

The LSP is particularly important for achieving modularity and ensuring that the system is easy to extend and maintain. By adhering to the principle, we can make sure that the system remains correct and consistent, even as we add new components or modify existing ones.

©️ Photo by Derek Steen on Unsplash

Suppose we have a billing service that generates reports on active licenses, and the service deals with licenses without knowledge of different types of licenses. In this case, different types of licenses can be substituted without affecting the function of the billing service because they all follow the same contract.

For example, we could have different types of licenses, such as Corporate License, Personal License and Education License that implement the same interface or extend the same superclass. As long as they adhere to the contract, they can be used interchangeably, without affecting the correctness of the program.

This is the essence of the LSP: designing software components that follow the same contract, so they can be substituted one for another without affecting the system’s correctness or behaviour. By following the LSP, we can ensure that the system is modular, extensible, and easy to maintain.

Consider the case of a Rectangle class with a SetWidth and SetHeight method, and a Square class that extends Rectangle. Since a square has equal sides, we might be tempted to make the SetWidth and SetHeight methods override each other to ensure that the sides are always equal. However, doing so violates the LSP, because a square is not a rectangle in the traditional sense, and it doesn’t follow the contract of the Rectangle class.

// this unit test would fail
var rectangle = new Rectangle(width: 50, height: 20);
rectangle.SetWidth(rect, 100);
rectangle.Height.Should.Be(20);

To adhere to the LSP in this case, we could create separate Square and Rectangle classes, each with its own SetWidth and SetHeight methods. Alternatively, we could introduce an abstract Shape class or interface that defines the common properties and behaviours of both Rectangle and Square and then have separate implementations for each.

The key takeaway here is that the Liskov Substitution Principle is not just about substituting objects of a superclass with objects of its subclasses; it’s also about ensuring that the subclasses follow the same contract as the superclass, without introducing impure inheritance hierarchies. By following this principle, we can ensure that our software systems are modular, flexible, and maintainable.

Interface Segregation Principle

The Interface Segregation Principle (ISP) advises software designers to avoid depending on things that they don’t use. The principle emphasizes the importance of creating simple and focused interfaces that are easier to understand, use, test, and validate.

The ISP states that clients should not be forced to depend on interfaces they do not use. In other words, an interface should only include methods that are relevant to the client. This principle helps to avoid bloated interfaces that carry unnecessary baggage and can cause problems later on.

©️ Photo by John Barkiple on Unsplash

Suppose we have a Customer Operations class that has information about deliveries and payments. We also have two reporting services, a Delivery Reporting Service that consumes information about deliveries, and a Billing Reporting Service that consumes information about payments. The Customer Operations class has a method that returns all the information, and both reporting services depend on this method.

In this scenario, we have violated the ISP because the reporting services depend on an interface (i.e., the Customer Operations class) that carries baggage they don’t use. This means that if we make changes to the Customer Operations class, it can affect the reporting services, even if the changes are unrelated to deliveries or payments.

To fix this, we can introduce two interfaces, IDeliveries and IPayments, that contain only the relevant methods for each reporting service. The Customer Operations class can then implement both interfaces and provide the necessary information to the reporting services via these interfaces.

By doing this, we have eliminated the unnecessary dependencies between the reporting services and the Customer Operations class. The Delivery Reporting Service only depends on the IDeliveries interface, and the Billing Reporting Service only depends on the IPayments interface. This makes the interfaces simple, focused, and relevant to the client, and it also makes the reporting services easier to test and validate.

By creating focused interfaces, we can improve the modularity and flexibility of the system. We can also make the system easier to test and validate since we only need to test the relevant methods. Additionally, focused interfaces can simplify the implementation of the client code since they only need to implement the necessary methods.

Acknowledge this:

Interfaces that are simple and focused are easier to understand and consume
Interfaces that are focused on a specific behaviour can be more easily tested

Dependency Inversion Principle

The Dependency Inversion Principle (DIP) states that high-level modules should not depend on low-level modules, but both should depend on abstractions. This means that the policy (i.e., high-level module) should not depend on the details (i.e., low-level module), but rather the details should depend on the policy.

To adhere to the DIP, we need to program abstractions rather than concretions. This means that we should avoid depending on concrete classes and instead depend on abstractions or interfaces. By doing this, we can make our systems more flexible and maintainable since we can change the implementation of the low-level details without affecting the high-level policies.

The key takeaway from the DIP is that we should program to the interfaces or abstractions, not the implementations. This approach enables us to decouple the modules in our system and create a more flexible and maintainable architecture.

©️ Photo by Dmitriy Zub on Unsplash

The analogy of the excavator is a good illustration of how the DIP works in practice. By depending on the abstract concept of a fixture rather than a specific fixture, the excavator can be easily adapted to different tasks without having to replace the entire machine. This approach enables us to reuse the excavator and make it more flexible and adaptable to different situations. If this were violated, imagine having to get a new excavator each time a task requires a different fixture.

Suppose we have an application host, such as a console application or web application, that depends on an application core. The application core contains the business logic or policy of the system. However, the application core also depends on a CRM SDK, which is an implementation detail or infrastructure concern.

In this scenario, we have violated the DIP because the application core depends on the details of the CRM SDK. This makes the system less flexible and harder to maintain since any changes to the CRM SDK could affect the application core and the application host.

To fix this, we can introduce an interface, such as ICustomerLeadsRepo, that defines the methods for creating, updating, and adding notes to customer leads. The application host can own or define this interface since it needs to interact with the CRM SDK. The application core can then depend on this interface instead of the CRM SDK directly.

By doing this, we have inverted the dependency and made the system more flexible and maintainable. The application host can now provide the necessary details to the application core via the ICustomerLeadsRepo interface, and the CRM SDK can implement this interface without affecting the application core or the application host.

In summary, the DIP is about inverting the dependency between high-level policies and low-level details by programming to abstractions rather than concretions. By doing this, we can create a more flexible and maintainable system that can adapt to changes in the implementation details without affecting the high-level policies.

Component Principles

Now that we have covered the SOLID principles that operate at the module level, let us move a level higher, to components.

The SOLID principles operate at the module level, while the principles of component design operate at a higher level of abstraction. Three principles of component design are important:

Reuse/Release Equivalence Principle: This principle states that classes and modules that are grouped into a component should be releasable together. This means that if you want to reuse a particular component in another project, you should be able to release that component without having to release any other unrelated components.

Common Closure Principle: This principle states that we should group those classes that change for the same reasons and at the same time. In other words, we should create components that have high cohesion and low coupling. This principle is similar to the Single Responsibility Principle but operates at the component level.
Common Reuse Principle: This principle states that we should not force users of a component to depend on things they don’t need. This means that we should create components that are highly cohesive and loosely coupled so that users can easily reuse specific parts of the component without having to depend on other unrelated parts.
The Reuse/Release Equivalence Principle and the Common Closure Principle are inclusive principles, while the Common Reuse Principle is not. Together, these principles help to create components that are easy to understand, easy to modify, and easy to reuse.

The three principles of component design create a tension between the need for reuse, the need for maintainability, and the need to avoid forcing users to depend on things they don’t need. The optimal position in this tension triangle will depend on the current concerns of the development team and the project’s current stage of development.

Early in the development of a project, the focus is often on developability, so the Common Closure Principle (CCP) is more important than the Reuse/Release Equivalence Principle (REP) or the Common Reuse Principle (CRP). As the project matures, the focus may shift towards reuse, and the team may prioritize the REP and CRP over the CCP.

Ultimately, the value of these principles lies in their ability to create components that are easy to understand, easy to modify, and easy to reuse. By following these principles, developers can create software systems that are flexible and adaptable to changing requirements and can be easily maintained over time.

Stable Dependencies Principle

Stability in software components refers to how difficult it is to make changes to that component without affecting other parts of the system. A stable component is less likely to change, and thus less likely to cause cascading changes throughout the system.

It is generally better to depend on stable components than on unstable ones. If a component is unstable, it may change frequently and cause cascading changes in dependent components, leading to more maintenance work and potential issues. On the other hand, stable components are less likely to change frequently and can provide a more reliable foundation for the software system.

The formula for calculating stability is

I = Fan-out / (Fan-out + Fan-in)

where Fan-out refers to the number of outgoing dependencies and Fan-in refers to the number of incoming dependencies.

Modules with a high value of I (close to 1) have many outgoing dependencies and few incoming dependencies, making them more unstable and easier to change. On the other hand, modules with a low value of I (close to 0) have few outgoing dependencies and many incoming dependencies, making them more stable and harder to change.

Software stability refers to the difficulty of making changes to a component. This can be affected by factors such as size, complexity, and clarity. One way to make a component hard to change is to have many other components depending on it. Such a component with many incoming dependencies is considered stable because any changes made to it would require a lot of work to reconcile with all the dependent components. The key rule to follow is to depend in the direction of stability, meaning that components with more incoming dependencies should be depended on by components with fewer incoming dependencies.

The key takeaway is to depend on more stable components, as they are less likely to change and cause ripple effects throughout the system.

Visually looking at this we can say the first labelled component is maximally stable as it has no outgoing dependencies but has a lot of incoming dependencies. Inversely the second labelled component is unstable.

We want to strike a balance between stability and changeability in our software system. If all components were highly stable, the system would become rigid and resistant to change, making it difficult to adapt to new requirements or fix issues. On the other hand, if all components were highly changeable, the system would become unstable and unreliable, making it difficult to ensure its correctness and maintainability over time.

Therefore, we need to design our component structure so that some components are more stable and others are more changeable, depending on their role in the system. And we should always try to depend in the direction of stability so that changes to unstable components don’t ripple through the entire system and cause unintended consequences.

The previously stable component, as shown below, is no longer stable as it now has an outgoing dependency on an unstable component. Remember our rule, always depend in the direction of stability.

We can employ the Dependency Inversion Principle to fix this violation by introducing an abstraction in between as shown below.

Below is a more real-world example of respecting the dependencies rules, by ensuring that our business rules, the policy, do not depend on the details but instead the details depend on the policy.

Clean Architecture

Finally, let’s talk about Clean Architecture. Let’s jump right in with some visuals. Were are going to introduce some concentric circles that represent key areas of our software and show how Clean Architecture expects dependencies on these to work.

Starting with the most stable, we would have what is critical to our operations, the entities that hold critical business rules.

Next, the Use Cases represent the application-specific business rules that are responsible for orchestrating the flow of information and actions between different components in the system. These rules are typically specific to a given application and may involve many moving parts, including entities, interfaces, and other components.

ℹ️ The direction of the arrows points to the direction of a dependency. That is to say A -> B, should be read as A depends on B, or that A knows of B. Inversely, this means B does not depend on A and is not aware of the existence of A. This is not to be confused with the flow of control, which will be discussed separately.

In the architecture diagram, the Use Cases are shown as being dependent on the entities, which contain the critical business rules that underpin the entire system. Notice the direction of the arrow, which indicates that the Use Cases depend on the entities, but the entities do not depend on the Use Cases.

This direction of the arrow is important because it represents the direction of stability in the system. As we move in the diagram from lower-level components to higher-level ones, the components become more abstract and less volatile. This means that they are less likely to change and are therefore more stable.

Conversely, as we move out of the diagram from higher-level components to lower-level ones, the components become more concrete and more volatile. This means that they are more likely to change and are therefore less stable.

In general, the arrows in the architecture diagram should always point in the direction of stability, with higher-level components depending on lower-level ones. This helps to ensure that the system remains stable and resilient to change over time.

Next up, we have the Interface Adapters.

These adapters are responsible for converting incoming formats from the outside world into formats that are convenient for the use cases, and vice versa. They are the bridge between the higher-level use cases and the lower-level details of the outside world, such as database implementations, SDKs, APIs, user interfaces, I/O, and more.

As with the other components, the direction of dependencies in the Interface Adapters layer should always point in the direction of stability. In this case, the adapters should depend on the interfaces defined by the use cases, and not the other way around.

The Interface Adapters layer serves as a buffer between the highly volatile and specific external technologies and the stable use cases, allowing the system to evolve without being too tightly coupled to any particular external technology.

In the outermost layer of Clean Architecture, we have the outside world which represents the low-level details such as the database, SDKs, APIs, User Interfaces, and IO. The arrows always point in the direction of stability, with the more inward direction indicating higher-level software and the more outward direction indicating lower-level software.

It’s important to note that the source code dependencies must only point inward towards stability, and violating this rule can result in a code smell. When the name of something in an outer circle is mentioned in an inner circle, or when data formats from an outer circle are used in an inner circle, it can be a sign that this rule is being violated.

What about the flow of control? The flow of control should always follow the flow of dependencies. If a use case needs to call an external system or service, it should not do so directly because that would violate the Dependency Rule. Instead, the use case should call an interface or port defined in the outer layer of the architecture, and the implementation of that interface in the outer layer should handle the interaction with the external system or service.

Similarly, if an external system or service needs to call into the system, it should do so through an interface defined in the outer layer. This technique ensures that all dependencies in the architecture flow inwards, towards the more stable and abstract layers, and that no component in an inner layer depends directly on a component in an outer layer.

Overall, the goal is to use dynamic polymorphism to create source code dependencies that allow for flexibility and extensibility while conforming to the Dependency Rule, regardless of the direction of the flow of control.

But I am doing this already

This may look familiar even if you are new to it, but that can be very misleading

The layers may look like the classic n-tier layers, but the flow of dependencies is different

If you are not explicitly implementing clean architecture or at the very least if you are not concerned about solving the same challenges addressed by the clean architecture you are probably doing the classic n-tier.

N-tiered architecture violates the Dependency Rule because the business logic layer knows about the data access and other infrastructure concerns, which means that high-level policies are dependent on low-level details. This violates the flow of dependencies and can lead to an unstable and difficult-to-maintain system. Clean Architecture, on the other hand, promotes a strict flow of dependencies where high-level policies depend only on abstractions and low-level details depend on high-level policies. This creates a more stable and maintainable system that is easier to change over time

Onion Architecture and Ports & Adapters

Different architectural patterns can be used to achieve the same goal of separating concerns and dependencies in a software system. The onion architecture, ports and adapters, and clean architecture are examples of such patterns.

©️ Image by Mark Seeman on his blog

While they may differ in their visualization and terminology, they share the same underlying principles and concepts. It’s perfectly fine to appreciate and have a preference for one over the other.

I prefer the clean architecture as it provides a clear visualization of the different layers and their dependencies, which is backed by Uncle Bob’s book. However, I also find the terminology used in Ports and Adapters to be more explicit in describing the different components and their roles. Therefore, I may subtly introduce such terminology in my explanation going forward.

Specifics on some of the software layers

What else belongs in the domain core?

Interfaces
Aggregate Roots
Value Objects
Entities
Domain Services/ Use Cases
Domain Exceptions
Domain Events
Event Handlers
Validators
Enums

What else belongs in the infrastructure?

Repositories
Db Context classes
API Clients
File System Access
Cloud Service Access
Email Clients
SMS Clients
System Clocks
SDK types
Identity

Summary

Following these simple rules we have discussed so far is not a challenging task, and it can save you a lot of trouble in the future.

By dividing your software into layers and abiding by the Dependency Rule, you will establish an inherently testable system, with all the advantages this entails.

If any external parts of the system become outdated, such as the database implementation or the web framework, you can easily replace those obsolete elements with a minimum of fuss.

Furthermore, you can lift and shift modules with ease. Your critical business rules, or your application business rules, can be reused in similar applications that may use different low-level details, such as varying technology implementations. For example, the same application core in a web API can be reused in a desktop application, as is.

The same applies to the External agency implementations, the infrastructure, or adapters. The infrastructure implementations derived from clean architecture, such as API clients, would have no business rules knowledge and thus can be used in entirely different applications, as it.

The million-dollar question is whether you can break these rules and still be following clean architecture principles. I firmly believe that you cannot, particularly the dependencies flow. The benefits of this architecture stem from its purity. This is not to say that it is a rigid architecture; there is a lot of flexibility built into it. However, the Dependency Rule is an absolute must.

©️ Photo by Tingey Injury Law Firm on Unsplash

⚠️ The above is a very bold remark but very good advice. Once you have a firm understanding of this and have respected the rules enough to gain experience you will stumble upon interesting scenarios that may require trade-offs, like the classic EF Core problem in . NET. You can make a trade-off and break the rule(s) but tread with care.

What if I grossly violate the rules anyway, what could go wrong?

Then you choose pain or sorts:

You choose to marry frameworks
You choose to marry the UI technology
You choose to marry the database
You choose to end up with something that is not testable
You choose to deal with occasional cyclic dependency issues
You choose developer spelunking

Must I remind you of your speluncaphobia, agoraphobia and mazeophobia? You can’t handle spelunking.
You choose to forfeit the ability to lift and shift
You choose productivity that is ever-decreasing with each new feature.

🛑 That is a lot of baggage to take on

Code Demo (C#)

To see Clean Architecture at play let us run with a scenario where we are creating a voucher system for a retail company. Initially, the requirement is to be able to generate vouchers that can be applied for free delivery. It should be possible to validate and redeem these vouchers, with the latter only being possible once.

We want to be able to store these vouchers but we are not sure about the details yet so we want to defer this requirement. As for generating the vouchers we have identified shortid as the library that we shall be using. For this example, let us assume this is a complex integration.

The demo examples will make use of the terminology from Ports and Adapters

Port: an interface between the application and the external world, which defines a set of operations or services that the application provides or requires.

Adapter: a component that implements a specific port interface, adapting it to the application’s internal architecture (This is infrastructure!).

Let us start with identifying the inner circle, the domain entities.

public record struct Voucher(string Code, VoucherType Type, Discount Discount, bool Shareable, bool Redeemed)
{
    public static Voucher NewVoucher(string code, VoucherType type, DiscountType discountType, float value,
        bool shareable)
    {
        var discount = Discount.Create(discountType, value);

        return new(
            code,
            type,
            discount,
            shareable,
            false
        );
    }

    public void Redeem() => Redeemed = false;
}

public readonly record struct Discount(DiscountType DiscountType, float Value)
{
    public static Discount Create(DiscountType discountType, float value) => new(
        discountType,
        value
    );
}

public enum DiscountType
{
    Unknown,
    Amount,
    Percentage
}

public enum VoucherState
{
    Unknown,
    AlreadyRedeemed,
    Valid,
    NotFound
}

public enum VoucherType
{
    Unknown,
    Delivery
}

The Voucher structure defines a public record with five properties: Code, Type, Discount, Shareable, and Redeemed. The Code property is a string that holds the unique identifier for a voucher, Type holds the type of voucher, Discount holds the discount applied by the voucher, Shareable indicates if the voucher can be shared, and Redeemed is a boolean indicating if the voucher has been used or not. Additionally, the Voucher structure has a static method named NewVoucher that takes in a code, type, discountType, value, and shareable parameter and returns a new Voucher object.

The Discount structure defines a public read-only record with two properties: DiscountType and Value. The DiscountType property is an enum that defines the type of discount (Amount or Percentage), and Value is the amount of the discount applied.

There are three enums defined in this code: DiscountType, VoucherState, and VoucherType. DiscountType has three values: Unknown, Amount, and Percentage, which are used to define the type of discount applied by a voucher. VoucherState has four values: Unknown, AlreadyRedeemed, Valid, and NotFound, which represent the state of a voucher. VoucherType has two values: Unknown and Delivery, which define the type of voucher.

In addition to the structures and enums, there is a method defined within the Voucher structure named Redeem(). This method sets the Redeemed property of a Voucher object to true.

Overall, this code provides us with a basic framework for working with vouchers, discounts, and their state as required by our client.

Next let us move to the next circle, the application. We shall require use cases that honour the expected behaviour, namely:

Generating a new voucher
Redeeming a voucher
Validating a voucher

Before diving into the details of these use cases, we can already identify that the use cases will need to trigger infrastructure. For example, the use cases will need to read, create or update voucher entries that we are storing and the generate voucher use case specifically will need to generate a new voucher code.

If we were to have a direct dependency on infrastructure, this would validate our dependencies rule, which states, always depend in the direction of stability. We can apply Dependency Inversion in our favour to make the application own and define the contracts that should then be honoured by the infrastructure. Our use cases can then depend on and consume these contracts, hence not depending on something volatile, and in turn, our infrastructure can depend on and implement these contracts.

Let’s look at what these contracts could look like

public interface IVoucherGenerator
{
    Task<string> GenerateNewCode(CancellationToken cancellationToken = default);
}

public interface IVoucherRepository
{
    Task Upsert(Voucher voucher, CancellationToken cancellationToken = default);

    Task<bool> TryGetVoucher(string code, out Voucher? voucher, CancellationToken cancellationToken = default);
}

The first interface IVoucherGenerator defines a single method GenerateNewCode that takes in an optional CancellationToken and returns a Task<string>. This method is responsible for generating a new voucher code and returning it as a string. The CancellationToken parameter is optional and allows for the cancellation of the task if needed.

The second interface IVoucherRepository defines two methods Upsert and TryGetVoucher. The Upsert method takes in a Voucher object and an optional CancellationToken and returns a Task. This method is responsible for inserting or updating a voucher in the repository. The CancellationToken parameter is optional and allows for the cancellation of the task if needed.

The TryGetVoucher method takes in a string representing a voucher code, an out parameter voucher of type Voucher? and an optional CancellationToken. This method is responsible for retrieving a voucher from the repository if it exists, assigning it to the voucher parameter and returning true. If the voucher does not exist, it returns false. The CancellationToken parameter is optional and allows for the cancellation of the task if needed.

Overall, these interfaces provide the necessary contracts for voucher generation and repository operations. By implementing these interfaces in classes, the code can be decoupled and modular, allowing for flexibility and maintainability. The IVoucherGenerator and IVoucherRepository interfaces can be used together to generate, store and retrieve vouchers, allowing the rest of the codebase to interact with vouchers without worrying about the implementation details of voucher generation and storage.

Now that we have defined the contracts, let’s look at the use cases.

public record GenerateVoucherRequest(VoucherType VoucherType, DiscountType DiscountType, float Value, bool Shareable)
{
    public static GenerateVoucherRequest ForFreeDelivery() => new(
        VoucherType.Delivery,
        DiscountType.Percentage,
        100,
        true
    );
}

public class GenerateVoucherUseCase
{
    private readonly IVoucherGenerator _voucherGenerator;
    private readonly IVoucherRepository _voucherRepository;

    public GenerateVoucherUseCase(IVoucherGenerator voucherCodeGenerator, IVoucherRepository voucherRepository)
    {
        _voucherGenerator = voucherCodeGenerator ?? throw new ArgumentNullException(nameof(voucherCodeGenerator));
        _voucherRepository = voucherRepository;
    }

    public async Task<Voucher> Handle(GenerateVoucherRequest request, CancellationToken cancellationToken = default)
    {
        var voucherCode = await _voucherGenerator.GenerateNewCode(cancellationToken);

        var voucher = Voucher.NewVoucher(voucherCode,
            request.VoucherType,
            request.DiscountType,
            request.Value,
            request.Shareable);

        await _voucherRepository.Upsert(voucher, cancellationToken);

        return voucher;
    }
}

public record RedeemVoucherRequest(string Code);

public record RedeemVoucherResponse(bool IsValid, Voucher? Voucher)
{
    public static RedeemVoucherResponse Valid(Voucher voucher) => new(
        IsValid: true,
        voucher
    );

    public static RedeemVoucherResponse Invalid(Voucher? voucher) => new(
        IsValid: false,
        voucher
    );
}

public class RedeemVoucherUseCase
{
    private readonly IVoucherRepository _voucherRepository;

    public RedeemVoucherUseCase(IVoucherRepository voucherRepository) => _voucherRepository = voucherRepository;

    public async Task<RedeemVoucherResponse> Handle(RedeemVoucherRequest request,
        CancellationToken cancellationToken = default)
    {
        var exists = await _voucherRepository.TryGetVoucher(request.Code, out var existingVoucher, cancellationToken);

        if (!exists)
            return RedeemVoucherResponse.Invalid(voucher: null);

        if (existingVoucher!.Value.Redeemed)
            return RedeemVoucherResponse.Invalid(existingVoucher);

        await RedeemVoucher(existingVoucher.Value);

        return RedeemVoucherResponse.Valid(existingVoucher.Value);
    }

    // If this logic is used elsewhere, ideally this would be another use case
    private async Task RedeemVoucher(Voucher existingVoucher)
    {
        existingVoucher.Redeem();

        await _voucherRepository.Upsert(existingVoucher);
    }
}

public record ValidateVoucherRequest(string Code);

public record ValidateVoucherResponse(VoucherState State, Voucher? Voucher = null)
{
    public static ValidateVoucherResponse NotFound() => new(
        VoucherState.NotFound
    );

    public static ValidateVoucherResponse Valid(Voucher voucher) => new(
        VoucherState.Valid,
        voucher
    );

    public static ValidateVoucherResponse AlreadyRedeemed(Voucher voucher) => new(
        VoucherState.AlreadyRedeemed,
        voucher
    );
}

public class ValidateVoucherUseCase
{
    private readonly IVoucherRepository _voucherRepository;

    public ValidateVoucherUseCase(IVoucherRepository voucherRepository) => _voucherRepository =
        voucherRepository ?? throw new ArgumentNullException(nameof(voucherRepository));

    public async Task<ValidateVoucherResponse> Handle(ValidateVoucherRequest request,
        CancellationToken cancellationToken = default)
    {
        var exists = await _voucherRepository.TryGetVoucher(request.Code, out var existingVoucher, cancellationToken);

        if (!exists)
            return ValidateVoucherResponse.NotFound();

        return existingVoucher!.Value.Redeemed
            ? ValidateVoucherResponse.AlreadyRedeemed(existingVoucher.Value)
            : ValidateVoucherResponse.Valid(existingVoucher.Value);
    }
}

The above defines three use cases related to voucher generation, redemption, and validation.

The GenerateVoucherUseCase class implements a use case for generating a new voucher. The class takes in an IVoucherGenerator and an IVoucherRepository object through its constructor. The Handle method of the class takes in a GenerateVoucherRequest object and an optional CancellationToken. The method generates a new voucher code using the IVoucherGenerator object, creates a new Voucher object using the Voucher.NewVoucher static method, and stores the voucher in the repository using the IVoucherRepository.Upsert method. The method returns the generated voucher.

The RedeemVoucherUseCase class implements a use case for redeeming a voucher. The class takes in an IVoucherRepository object through its constructor. The Handle method of the class takes in a RedeemVoucherRequest object and an optional CancellationToken. The method uses the IVoucherRepository.TryGetVoucher method to check if the voucher exists in the repository and is not already redeemed. If the voucher is valid, the method calls the Redeem method of the Voucher object, sets the Redeemed property to true, and stores the voucher in the repository using the IVoucherRepository.Upsert method. The method returns a RedeemVoucherResponse object indicating whether the voucher was successfully redeemed.

The ValidateVoucherUseCase class implements a use case for validating a voucher. The class takes in an IVoucherRepository object through its constructor. The Handle method of the class takes in a ValidateVoucherRequest object and an optional CancellationToken. The method uses the IVoucherRepository.TryGetVoucher method to check if the voucher exists in the repository and is not already redeemed. The method returns a ValidateVoucherResponse object indicating whether the voucher is valid, already redeemed or not found.

That concludes the application concerns. Next, let us move to the infrastructure by implementing the contracts defined in the application.

internal class VoucherRepository : IVoucherRepository
{
    private readonly InMemoryStore<Voucher> _inMemoryStore = new();

    public Task Upsert(Voucher voucher, CancellationToken cancellationToken = default)
    {
        _inMemoryStore.Add(voucher);

        return Task.CompletedTask;
    }

    public Task<bool> TryGetVoucher(string code, out Voucher? voucher, CancellationToken cancellationToken = default)
    {
        voucher = _inMemoryStore.TryGet(vouchers => vouchers.SingleOrDefault(x => x.Code.Equals(code)));

        return Task.FromResult(!(voucher is null));
    }
}

internal class InMemoryStore<T>
{
    private readonly ConcurrentBag<T> _store = new();

    public void Add(T item) => _store.Add(item);

    public T TryGet(Func<IEnumerable<T>, T> selector) => selector(_store.AsEnumerable());
}

public class PersistenceAdapter
{
    private ServiceProvider _serviceProvider;

    public PersistenceAdapter()
    {
        SetupAdapterContainer();
    }

    public void Initialize(IServiceCollection services) =>
        services.AddScoped(x => _serviceProvider.GetService<IVoucherRepository>());

    public void SetupAdapterContainer()
    {
        var services = new ServiceCollection();
        services.AddSingleton<IVoucherRepository, VoucherRepository>();

        _serviceProvider = services.BuildServiceProvider();
    }
}

The above defines an in-memory store and an adapter class for persistence operations.

The VoucherRepository class implements the IVoucherRepository interface and uses an in-memory store to add and retrieve voucher objects. The class has a private instance of InMemoryStore<Voucher> class and two public methods: Upsert and TryGetVoucher.

The InMemoryStore<T> class is a generic class that provides an in-memory store for any type of object. The details are not of particular importance. The key thing here is that we could defer the decision of implementing a data store, which could have been a database, file storage, blob storage, etc, by implementing a trivial in-memory store. Once we have enough context to make this decision, all we need to do is implement a new concrete implementation of the interface and once tested and ready, simply swap the previous for the new in our bootstrapping.

The PersistenceAdapter class is a simple adapter class that initializes and sets up the IVoucherRepository object using the VoucherRepository class.

Overall, these classes provide a basic framework for implementing persistence operations.

internal class CodeGenerator : IVoucherGenerator
{
    private readonly VoucherGeneratorAdapterOptions _options;

    public CodeGenerator(VoucherGeneratorAdapterOptions options)
    {
        _options = options ?? throw new ArgumentNullException(nameof(options));
    }

    public Task<string> GenerateNewCode(CancellationToken cancellationToken = default) =>
        Task.FromResult(ShortId.Generate(
            new GenerationOptions
            {
                Length = _options.Length,
                UseNumbers = _options.UseNumbers,
                UseSpecialCharacters = _options.UseSpecialCharacters
            }));
}

public class VoucherGeneratorAdapterOptions
{
    public int Length { get; set; }
    public bool UseNumbers { get; set; }
    public bool UseSpecialCharacters { get; set; }
}

public class VoucherGeneratorAdapter
{
    private ServiceProvider _serviceProvider;

    public VoucherGeneratorAdapter(VoucherGeneratorAdapterOptions options)
    {
        SetupContainer(options);
    }

    public void Initialize(IServiceCollection services) =>
        services.AddScoped(x => _serviceProvider.GetService<IVoucherGenerator>());

    private void SetupContainer(VoucherGeneratorAdapterOptions options)
    {
        var services = new ServiceCollection();
        services.AddSingleton(options);
        services.AddSingleton<IVoucherGenerator, CodeGenerator>();

        _serviceProvider = services.BuildServiceProvider();
    }
}

The defines a code generator class and an adapter class for generating voucher codes.

The CodeGenerator class implements the IVoucherGenerator interface and uses a third-party library called ShortId to generate short, unique, and random voucher codes. The class takes in a VoucherGeneratorAdapterOptions object through its constructor, which specifies the length of the code, whether to use numbers, and whether to use special characters. The class has a single public method, GenerateNewCode, that returns a new voucher code as a string wrapped in a Task object.

The VoucherGeneratorAdapter class is an adapter class that initializes and sets up the IVoucherGenerator object using the CodeGenerator class.

Overall, these classes provide a basic framework for generating voucher codes using the ShortId library.

Let us look at how this can be consumed, by observing some bootstrapping and a sample ASP.NET Core controller.

private void SetupAdapter(IServiceCollection services)
{
    var settings = Configuration.Get<ApplicationOptions>();
    var persistenceAdapter = new PersistenceAdapter();
    var voucherGeneratorAdapter =
        new VoucherGeneratorAdapter(settings!.VoucherGeneratorOptions.VoucherVoucherGenerator);
    persistenceAdapter.Initialize(services); voucherGeneratorAdapter.Initialize(services);
}

public class ApplicationOptions
{
    public VoucherGeneratorOptions VoucherGeneratorOptions { get; set; }
}

[ApiController]
[Route("[controller]")]
public class VouchersController : ControllerBase
{
   private readonly ILogger<VouchersController> _logger;

    public VouchersController(ILogger<VouchersController> logger) => _logger = logger;

    [HttpGet(Name = "/vouchers/delivery/generate")]
    public async Task<IActionResult> Get([FromServices] GenerateVoucherUseCase useCase,
        CancellationToken cancellationToken)
    {
        var request = GenerateVoucherRequest.ForFreeDelivery();

        using (_logger.BeginScope(new Dictionary<string, object>
               {
                   ["query"] = request
               }))
        {
            var response = await useCase.Handle(request, cancellationToken);

            return Created(string.Empty, response);
        }
    }
}

The code shows how to consume the Adapters/Infrastructure in an ASP.NET Core application. The SetupAdapter method sets up the necessary adapters and initializes them with the services required for the application to work. The ApplicationOptions class is used to define options for the voucher generator adapter. The VouchersController is a sample ASP.NET Core controller that demonstrates the flow of control.

We have all the moving parts, what about the project structure? This is quite flexible, and a decision you can make based on your project context or preferences and can leverage onto other patterns and practices you may be using like CQRS (Command Query Responsibility Segregation), DDD (Domain Driven Design), Vertical Slices etc. The key thing is however to continue to respect the dependencies’ rules. Let us look at an example project structure.

We can observe that the Domain is maximally stable as it has no outgoing dependencies but has incoming dependencies, which is desirable as this is our policy. Next, we have the Application that knows of the Domain, followed by two adapters VoucherGenerator and Persistence that know of the Domain and, hence transitively know of the Domain. Lastly, we have a Web Host that knows of both adapters, as it needs to bootstrap them, and therefore transitively knows of everything. This know-all scenario is expected for Application Hosts and Tests.

Let’s observe this again but with transitive references shown.

The resulting solution structure looks as follows.

We can visualize what we have in code as follows

If we draw concentric circles based on that visualization with the various software levels, we would end up with the following

That means we have satisfied Clean Architecture in our implementation 😃.

Tests

What about tests?

Tests are an essential part of software development and they follow the Dependency Rule just like any other component of the system. They always depend inward towards the code being tested and nothing within the system depends on the tests. This makes them the outermost layer of Clean Architecture.

If the system architecture is centred around the use cases and the frameworks are kept at arm’s length, it should be possible to unit-test all the use cases without any frameworks in place. This is because tests are concrete and detailed, and they follow the Dependency Rule, making them easy to control.

One advantage of tests is their flexibility in skipping boundaries. This allows developers to choose the type of tests to author, depending on the needs of the system. The combination of Inversion of Dependencies and Dependency Injection gives developers full control of this.

In the case of the demo sample, developers could opt for integration tests that test at the API level. They could choose to use real dependencies for their infra implementation, mock them, or use doubles like an in-memory implementation. The same applies to unit tests, where developers can employ mocking or doubles to make the process easier.

Clean Architecture doesn’t necessarily empower developers with things like end-to-end tests, but it also doesn’t hinder them. Developers can still write contract tests, behaviour-driven tests, targeted integration tests, and more.

When a clean architecture is not followed, tests may not be integrated into the design as a first-class citizen. This leads to difficulties in testing and fragile tests. In such cases, tests are likely to be strongly coupled to the system being tested, making it challenging to maintain and change the system.

A clear indication of a poorly designed test system is the strong coupling between tests and the system components. Such tightly coupled tests must change along with the system. Even a minor modification to a component can cause many dependent tests to break or require modifications.

It is essential to understand that tests are not separate from the system being tested. Instead, they are integral parts of the system and must be well-designed to provide the benefits of stability and regression testing. Therefore, it is crucial to adhere to clean architecture principles to ensure that tests are appropriately integrated into the system design. By doing so, testing can be more effective, and changes to the system can be made with greater ease and confidence.

Summary

Give Clean Architecture a try and evaluate for yourself the claims it makes. Overall, however, we can summarize as follows:

Clean Architecture is good for long-term software life
Clean Architecture is good for stability and maintainability
Clean Architecture is good testability
The stability, maintainability and testability of your software are good for your long-term health

Azure Pipelines Standards and Practices

Sun, 19 Feb 2023 00:00:00 GMT

Photo by JJ Ying on Unsplash

Azure Pipelines - Standards and Practices

✅ DO create pipelines in YAML committed in the relevant project repository and not via the classic UI approach or CLI.

✅ DO add shared variables to a variable group to avoid duplicated variables across various pipelines.

Add only truly global variables, that is any for which you would require the same value for all pipelines that use this without the need to customize for a single pipeline. An API key to a third-party service or a license key is a good example.

✅ DO leverage integrations to services like Azure Key Vault for managing user secrets and using those in pipeline runs.

✅ DO store secret variables with Keep this value as secret enabled. The default would be for Azure DevOps to store variables as raw text.

Always ensure your secrets are stored securely and aren’t hardcoded in your pipeline.

✅ DO favour parameters over variables for scenarios where you have a finite list of values as parameters will make use of a radio button group or dropdown for convenience.

✅ DO maintain an extensive retention policy for pipeline runs. This is important to track run history over an extensive period for example to see if test run performance is regressing, linting cleanups are on track, and the addition of tests in the solution is healthy.

✅ DO have a clear naming convention for your pipelines, making it easy to identify the type of pipeline (CI or CD), the target environment, and the project being built.

Here’s an example of an Azure Pipelines naming standard:

Build pipelines: build-{projectName}
Release pipelines: release-{projectName}-{environmentName}
Deployment pipelines: deploy-{projectName}-{environmentName}
Infrastructure pipelines: infra-{infrastructureType}-{projectName}-{environmentName}

Here’s a breakdown of each component of the naming standard:

{projectName}: the name of the project being built, deployed, or tested.
{environmentName}: the name of the environment being deployed or released to.
{infrastructureType}: the type of infrastructure being deployed or managed, such as aks for Azure Kubernetes Service or appsvc for Azure App Service. {testType}: the type of test being run, such as unit, integration, or end-to-end.
{platform}: the platform being tested on, such as linux-x64 or windows-x64.

By following a consistent naming standard like this, it’s easier to quickly identify and locate the pipelines you need within Azure DevOps.

✅ DO use templates to share pipeline configuration across projects. This ensures consistency in pipeline design and makes it easier to maintain pipelines.

✅ DO implement checks such as quality gates, code coverage requirements, and security scans as part of the pipeline.

✅ DO have a clear and concise description of what the pipeline is doing and include any relevant links to documentation or related systems.

✅ DO split long-running jobs across multiple stages. This promotes visibility and enables progress reporting as each stage is completed. Make sure any dependencies between stages are required to avoid unnecessary waits in the runs.

✅ DO make use of the appropriate pool directive for agents to help reduce build times by ensuring the build does not wait for an agent to become available.

✅ DO ensure that your deployment scripts are idempotent, i.e., running the script multiple times should not produce unexpected/different results.

✅ DO favour explicit versions of SDKs and other tooling in your pipelines. For example explicit node versions, .NET SDK versions or Bicep versions.

✅ DO make use of the appropriate pool directive for agents to help reduce build times by ensuring the build does not wait for an agent to become available.

ℹ️ Microsoft frequently updates these images with the latest available tooling, bugfixes and patches. You can also specify a specific version for reproducibility reasons, especially when you want to maintain compatibility with certain libraries or frameworks.

✅ DO have separate pipelines for CI/CD. This separation makes it easy to change the deployment strategy without affecting the build process.

✅ DO modularize your pipelines into templates for stages, jobs or steps once they start to become large or very focused. This makes things more manageable and readable, also opening up reuse potential.

Once you have the same logic copied across multiple pipelines this is often the smell that you could do with a template.

🎇 DO consider making use of caching to speed up pipelines by avoiding unnecessary computation or downloads. For example, by hashing a node package lock JSON file as a cache key, you can avoid doing a node install when the packages have not changed and are available in a cache.

🎇 DO consider making use of pipeline artefacts to make resources from one pipeline run available to another for download to speed up pipelines by doing certain computations or downloads once.

A key deciding factor between caching and artefacts is whether or not the artefacts are optional or mandatory. If optional use a cache, else artifacts.

✅ DO that you only publish required artefacts to keep your pipelines lean.

✅ DO that you only download required artefacts to keep your pipelines lean.

🎇 DO consider comments in your pipelines for things that may not be easy to understand. However, avoid commenting on every single line and assume a basic understanding of Azure Pipelines as a prerequisite for any reader.

🎇 DO consider built-in tasks, particularly platform-independent ones to increase the portability of your pipelines over using things like PowerShell tasks.

🎇 DO consider condition-based stages to optimize your pipelines by not executing stages that are not required based off set conditional logic.

✅ DO ensure you have monitoring and alerting in place for your pipelines, so you can be notified when something goes wrong.

ℹ️ A healthy build and deployment process is only as good as the monitoring and alerting in place. Ensure that you have appropriate notifications and reporting in place so that you can quickly identify and remediate any issues that arise

⛔ DO NOT expose secrets via logs in the pipeline runs or to any unauthorized users.

⛔ DO NOT use pipeline variables to store sensitive data such as secrets, keys, or passwords. Instead, use Azure Key Vault or a similar secure storage service.

⛔ DO NOT create overly complex pipelines with unnecessary steps or dependencies. This can lead to longer build times and an increased risk of failure. Keep pipelines simple and focused on the task at hand.

⛔ DO NOT use scripts or code snippets in your pipeline without proper testing and validation. This can introduce security vulnerabilities or errors that are difficult to diagnose

Examples

Here is a single pipeline example that accounts for most of the recommendations in this document with comments

# Build pipeline for a .NET Core project
# Uses a pool of Microsoft-hosted agents
# Stores secrets in Azure Key Vault
# Includes linting, unit testing, code coverage, and security scanning checks
# Uses templates for pipeline configuration
# Has a clear description of what the pipeline is doing and relevant links
# Splits long-running jobs across multiple stages
# Uses the appropriate agent pool to reduce build times
# Has a separate pipeline for CI and CD
# Has monitoring and alerting in place for the pipeline

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

stages:
- stage: Build
  displayName: 'Build stage'
  jobs:
  - job: Build
    displayName: 'Build job'
    steps:
    - template: templates/dotnet-core-build.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Build completed $(Build.BuildId)
      displayName: 'Finish Build'
    # The linting check step
    - template: templates/eslint-check.yml
      parameters:
        workingDirectory: '$(Build.SourcesDirectory)'

- stage: Test
  displayName: 'Test stage'
  jobs:
  - job: Test
    displayName: 'Test job'
    steps:
    - template: templates/dotnet-core-test.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Test completed $(Build.BuildId)
      displayName: 'Finish Test'

- stage: Publish
  displayName: 'Publish stage'
  jobs:
  - job: Publish
    displayName: 'Publish job'
    steps:
    - template: templates/dotnet-core-publish.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Publish completed $(Build.BuildId)
      displayName: 'Finish Publish'

# Variables that should be shared across pipelines can be stored in variable groups
# Secrets should be stored in Azure Key Vault or a similar secure storage service
variables:
- group: MyVariableGroup

# Templates can be used to share pipeline configuration across projects
# In this example, we have separate templates for .NET Core builds, tests and publish
# These templates can be reused across different projects with minor modifications
templates:
- template: templates/dotnet-core-build.yml
- template: templates/dotnet-core-test.yml
- template: templates/dotnet-core-publish.yml

# Use a clear and concise description of what the pipeline is doing
# Include relevant links to documentation or related systems
# This will help team members understand what the pipeline does and how it works
# It also provides a starting point for troubleshooting if issues arise
# The example below includes links to the project's wiki, issue tracker, and documentation
resources:
  repositories:
  - repository: self
    triggers:
    - main
    - development
  - repository: git://github.com/MyOrganization/my-project.wiki.git
    name: wiki
  - repository: git://github.com/MyOrganization/my-project/issues.git
    name: issues
  - repository: git://github.com/MyOrganization/my-project/docs.git
    name: docs

# Specify an appropriate agent pool to reduce

✅ DO create pipelines in YAML committed in the relevant project repository and not via the classic UI approach or CLI.

✅ DO add shared variables to a variable group to avoid duplicated variables across various pipelines.

Add only truly global variables, that is any for which you would require the same value for all pipelines that use this without the need to customize for a single pipeline. An API key to a third-party service or a license key is a good example.

✅ DO leverage integrations to services like Azure Key Vault for managing user secrets and using those in pipeline runs.

✅ DO favour parameters over variables for scenarios where you have a finite list of values as parameters will make use of a radio button group or dropdown for convenience.

✅ DO store secret variables with Keep this value as secret enabled. The default would be for Azure DevOps to store variables as raw text.

✅ DO have a clear naming convention for your pipelines, making it easy to identify the type of pipeline (CI or CD), the target environment, and the project being built.

Here’s an example of an Azure Pipelines naming standard:

Build pipelines: build-{projectName}
Release pipelines: release-{projectName}-{environmentName}
Deployment pipelines: deploy-{projectName}-{environmentName}
Infrastructure pipelines: infra-{infrastructureType}-{projectName}-{environmentName}

Here’s a breakdown of each component of the naming standard:

{projectName}: the name of the project being built, deployed, or tested.
{environmentName}: the name of the environment being deployed or released to.
{infrastructureType}: the type of infrastructure being deployed or managed, such as aks for Azure Kubernetes Service or appsvc for Azure App Service. {testType}: the type of test being run, such as unit, integration, or end-to-end.
{platform}: the platform being tested on, such as linux-x64 or windows-x64.

By following a consistent naming standard like this, it’s easier to quickly identify and locate the pipelines you need within Azure DevOps.

✅ DO use templates to share pipeline configuration across projects. This ensures consistency in pipeline design and makes it easier to maintain pipelines.

✅ DO implement checks such as quality gates, code coverage requirements, and security scans as part of the pipeline.

✅ DO have a clear and concise description of what the pipeline is doing and include any relevant links to documentation or related systems.

✅ DO split long-running jobs across multiple stages. This promotes visibility and enables progress reporting as each stage is completed.

✅ DO make use of the appropriate pool directive for agents to help reduce build times by ensuring the build does not wait for an agent to become available.

✅ DO split long-running jobs across multiple stages. This promotes visibility and enables progress reporting as each stage is completed.

✅ DO ensure that your deployment scripts are idempotent, i.e., running the script multiple times should not produce unexpected/different results.

✅ DO make use of the appropriate pool directive for agents to help reduce build times by ensuring the build does not wait for an agent to become available.

ℹ️ Microsoft frequently updates these images with the latest available tooling, bugfixes and patches. You can also specify a specific version for reproducibility reasons, especially when you want to maintain compatibility with certain libraries or frameworks.

✅ DO have separate pipelines for CI and CD. This separation makes it easy to change the deployment strategy without affecting the build process.

✅ DO ensure you have monitoring and alerting in place for your pipelines, so you can be notified when something goes wrong.

ℹ️ A healthy build and deployment process is only as good as the monitoring and alerting in place. Ensure that you have appropriate notifications and reporting in place so that you can quickly identify and remediate any issues that arise

⛔ DO NOT use pipeline variables to store sensitive data such as secrets, keys, or passwords. Instead, use Azure Key Vault or a similar secure storage service.

⛔ DO NOT use scripts or code snippets in your pipeline without proper testing and validation. This can introduce security vulnerabilities or errors that are difficult to diagnose

Examples

Here is a single pipeline example that accounts for most of the recommendations in this document with comments

# Build pipeline for a .NET Core project
# Uses a pool of Microsoft-hosted agents
# Stores secrets in Azure Key Vault
# Includes linting, unit testing, code coverage, and security scanning checks
# Uses templates for pipeline configuration
# Has a clear description of what the pipeline is doing and relevant links
# Splits long-running jobs across multiple stages
# Uses the appropriate agent pool to reduce build times
# Has a separate pipeline for CI and CD
# Has monitoring and alerting in place for the pipeline

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

stages:
- stage: Build
  displayName: 'Build stage'
  jobs:
  - job: Build
    displayName: 'Build job'
    steps:
    - template: templates/dotnet-core-build.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Build completed $(Build.BuildId)
      displayName: 'Finish Build'
    # The linting check step
    - template: templates/eslint-check.yml
      parameters:
        workingDirectory: '$(Build.SourcesDirectory)'

- stage: Test
  displayName: 'Test stage'
  jobs:
  - job: Test
    displayName: 'Test job'
    steps:
    - template: templates/dotnet-core-test.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Test completed $(Build.BuildId)
      displayName: 'Finish Test'

- stage: Publish
  displayName: 'Publish stage'
  jobs:
  - job: Publish
    displayName: 'Publish job'
    steps:
    - template: templates/dotnet-core-publish.yml
      parameters:
        solution: ${{ variables.solution }}
        buildPlatform: ${{ variables.buildPlatform }}
        buildConfiguration: ${{ variables.buildConfiguration }}
    - script: echo Publish completed $(Build.BuildId)
      displayName: 'Finish Publish'

# Variables that should be shared across pipelines can be stored in variable groups
# Secrets should be stored in Azure Key Vault or a similar secure storage service
variables:
- group: MyVariableGroup

# Templates can be used to share pipeline configuration across projects
# In this example, we have separate templates for .NET Core builds, tests, and publish
# These templates can be reused across different projects with minor modifications
templates:
- template: templates/dotnet-core-build.yml
- template: templates/dotnet-core-test.yml
- template: templates/dotnet-core-publish.yml

# Use a clear and concise description of what the pipeline is doing
# Include relevant links to documentation or related systems
# This will help team members understand what the pipeline does and how it works
# It also provides a starting point for troubleshooting if issues arise
# The example below includes links to the project's wiki, issue tracker, and documentation
resources:
  repositories:
  - repository: self
    triggers:
    - main
    - development
  - repository: git://github.com/MyOrganization/my-project.wiki.git
    name: wiki
  - repository: git://github.com/MyOrganization/my-project/issues.git
    name: issues
  - repository: git://github.com/MyOrganization/my-project/docs.git
    name: docs

# Specify an appropriate agent pool to reduce

GDPR technological implementation research (AWS Cloud)

Sun, 19 Feb 2023 00:00:00 GMT

Photo by Towfiqu barbhuiya on Unsplash

GDPR technological implementation research (AWS Cloud)

The General Data Protection Regulation (GDPR) is a European Union (EU) regulation that came into effect in 2018. It is designed to protect the privacy and personal data of EU citizens by setting strict guidelines for how data should be collected, processed, and stored. The GDPR has far-reaching implications for organizations that handle personal data, including those that use cloud services like Amazon Web Services (AWS). In this research, we aim to explore how the GDPR is translated into engineering, with a specific focus on the AWS Cloud platform. Our goal is to provide technical recommendations and best practices that will help AWS users achieve GDPR compliance and ensure the privacy and security of personal data.

Research Goals

The purpose of this research is to provide technical recommendations and best practices for achieving GDPR compliance on the AWS Cloud platform. Specifically, I aim to answer the following questions:

What practices are followed around data handling both at rest and in transit on AWS Cloud?
How does GDPR impact what information we can store and how we store it, including in databases, files, logs, etc. on AWS Cloud?
How does GDPR impact what support agents or developers can do when they need to get client data locally to reproduce issues on AWS Cloud?

By answering these questions, I hope to provide AWS Cloud users with a better understanding of how to implement GDPR requirements in their environments and ensure the privacy and security of personal data. The technical recommendations and best practices I provide are based on my analysis of the GDPR and how it can be implemented on AWS Cloud, as well as on industry standards and best practices for data security and privacy.

Useful resources

What is GDPR, the EU’s new data protection law?

Privacy and Data Protection by Design

Technical

Consider client-side encryption of data before transferring to S3, using AWS SDKs that support encryption and decryption operations of objects.
Adopt tools in AWS that aid with better security and governance posture.
- AWS Security Hub
  
  AWS Security Hub is a security service offered by Amazon Web Services (AWS) that provides a comprehensive view of security alerts and compliance status across an AWS account. The General Data Protection Regulation (GDPR) is a regulation in EU law on data protection and privacy that requires organizations to take measures to protect personal data.
  
  AWS Security Hub can help with GDPR compliance in several ways:
  - Identifying potential security threats: AWS Security Hub provides a centralized view of security alerts from different AWS services, such as Amazon GuardDuty and AWS Inspector, which can help identify potential security threats and vulnerabilities that may impact the security of personal data.
  - Enabling compliance checks: AWS Security Hub provides compliance checks against a variety of industry standards and regulations, including GDPR. These checks can help identify non-compliant resources and provide recommendations for remediation.
  - Automating compliance monitoring: AWS Security Hub can automate compliance monitoring by continuously checking AWS resources for compliance with GDPR requirements, such as data access control, data retention policies, and data protection measures.
  - Facilitating incident response: AWS Security Hub can also help organizations respond to security incidents and breaches by providing alerts and notifications when security events occur, as well as by integrating with other AWS security services such as AWS CloudTrail and AWS Config.
- Amazon Guard Duty
  
  Amazon GuardDuty is a threat detection service offered by Amazon Web Services (AWS) that continuously monitors for malicious activity and unauthorized behaviour within an AWS environment. GuardDuty can help with GDPR compliance by providing enhanced security for personal data stored or processed within the AWS environment.
  
  Here are some specific ways that Amazon GuardDuty can help with GDPR compliance:
  - Detecting threats to personal data: Amazon GuardDuty can help detect threats to personal data stored or processed within an AWS environment. By analyzing log data and network traffic, GuardDuty can identify potential threats such as compromised instances, malware, or unauthorized access to sensitive data.
  - Providing real-time alerts: GuardDuty can provide real-time alerts when it detects suspicious activity, allowing organizations to quickly investigate and respond to potential threats.
  - Automating threat detection: GuardDuty can automatically detect and alert on potential threats, reducing the need for manual monitoring and analysis.
  - Enabling compliance with GDPR security requirements: GuardDuty can help organizations comply with GDPR security requirements by providing visibility into security threats and unauthorized behaviour that may impact personal data. GuardDuty can also be integrated with other AWS security services, such as AWS Security Hub and AWS CloudTrail, to provide a comprehensive view of security and compliance posture.
  - Offering simplified compliance reporting: GuardDuty provides a range of pre-built compliance reports that can help organizations demonstrate their compliance with GDPR and other regulations.
  In summary, Amazon GuardDuty can help organizations meet GDPR compliance requirements by providing enhanced security for personal data, automating threat detection and response, and simplifying compliance reporting.
- Amazon Inspector
  
  Amazon Inspector is a security assessment service offered by Amazon Web Services (AWS) that helps identify security vulnerabilities and compliance issues within an AWS environment. Amazon Inspector can help with GDPR compliance by providing a range of security assessments that can help organizations identify and remediate security issues that may impact personal data.
  
  Here are some specific ways that Amazon Inspector can help with GDPR compliance:
  - Identifying security vulnerabilities: Amazon Inspector can help identify security vulnerabilities within an AWS environment, such as missing security patches, insecure network configurations, or weak access controls. By identifying these vulnerabilities, organizations can take steps to remediate them and reduce the risk of unauthorized access to personal data.
  - Enabling compliance assessments: Amazon Inspector can assess an organization’s compliance with a variety of industry standards and regulations, including GDPR. These assessments can help identify non-compliant resources and provide recommendations for remediation.
  - Providing a range of assessment templates: Amazon Inspector provides a range of pre-built assessment templates that can help organizations assess the security of their AWS resources, including EC2 instances, network configurations, and applications.
  - Automating security assessments: Amazon Inspector can automatically perform security assessments regularly, reducing the need for manual security testing and analysis.
  - Facilitating incident response: Amazon Inspector can help organizations respond to security incidents and breaches by identifying vulnerabilities and misconfigurations that may have contributed to the incident. This information can help organizations remediate the underlying security issues and prevent similar incidents in the future.
  In summary, Amazon Inspector can help organizations meet GDPR compliance requirements by identifying security vulnerabilities and compliance issues within an AWS environment, automating security assessments, and facilitating incident response.
- AWS Systems Manager
  
  AWS Systems Manager is a management service offered by Amazon Web Services (AWS) that helps manage and automate operational tasks within an AWS environment. AWS Systems Manager can help with GDPR compliance by providing visibility and control over the configuration of AWS resources, enabling organizations to ensure that personal data is processed and stored in a secure and compliant manner.
  
  Here are some specific ways that AWS Systems Manager can help with GDPR compliance:
  - Managing AWS resource configurations: AWS Systems Manager can help manage the configuration of AWS resources such as EC2 instances, databases, and network configurations. This can help ensure that personal data is stored and processed in a secure and compliant manner, with appropriate access controls, data retention policies, and encryption.
  - Automating security patching: AWS Systems Manager can automate the process of applying security patches and updates to AWS resources. This can help ensure that systems are up-to-date and secure, reducing the risk of security vulnerabilities that may impact personal data.
  - Enabling compliance checks: AWS Systems Manager provides compliance checks against a variety of industry standards and regulations, including GDPR. These checks can help identify non-compliant resources and provide recommendations for remediation.
  - Facilitating incident response: AWS Systems Manager can help organizations respond to security incidents and breaches by providing tools for remote management and automation. This can help organizations quickly and efficiently remediate security issues and prevent further damage to personal data.
  - Providing centralized management: AWS Systems Manager provides a centralized management console for managing AWS resources, making it easier for organizations to manage large and complex environments.
  In summary, AWS Systems Manager can help organizations meet GDPR compliance requirements by managing and automating operational tasks within an AWS environment, enabling compliance checks, and facilitating incident response.
AWS Macie

AWS Macie is a security service offered by Amazon Web Services (AWS) that provides automated data discovery, classification, and protection of sensitive data in an AWS environment. AWS Macie can help with GDPR compliance by identifying and classifying personal data stored within an AWS environment, and by providing tools to protect that data.

Here are some specific ways that AWS Macie can help with GDPR compliance:
- Identifying personal data: AWS Macie can automatically discover and classify personal data stored within an AWS environment, including data such as names, addresses, phone numbers, and email addresses. This can help organizations better understand what personal data they have, where it is stored, and how it is being used.
- Monitoring data access: AWS Macie can monitor data access and usage, providing visibility into who is accessing personal data and how it is being used. This can help organizations ensure that personal data is being processed and stored in a secure and compliant manner.
- Enabling data protection: AWS Macie provides tools for protecting personal data, such as data loss prevention (DLP) policies that can help prevent data exfiltration or unauthorized access to personal data.
- Automating compliance checks: AWS Macie can automatically perform compliance checks against a variety of industry standards and regulations, including GDPR. These checks can help identify non-compliant resources and provide recommendations for remediation.
- Facilitating incident response: AWS Macie can help organizations respond to security incidents and breaches by providing tools for identifying and remediating data breaches or other security incidents involving personal data.
- In summary, AWS Macie can help organizations meet GDPR compliance requirements by automatically discovering and classifying personal data, monitoring data access and usage, enabling data protection, automating compliance checks, and facilitating incident response.
Create a plan to keep up with changes in TLS protocols and ensure support for the most secure HTTPS protocol. When vulnerabilities are discovered, a new TLS versioned protocol is released, and it is important to keep up with these updates.
The purpose of the collection and processing of personal data should be clearly defined and documented. Be critical about reusing data for other purposes particularly if this is not communicated to the user.
- Customer data requests have to adhere to this. For instance, if communicated indicated that specific data would stay in a given region that should not be violated and it should be clear for what duration the data would be available in another form and the intended use. Scenarios like this should be covered by the privacy policy documentation.
Ask the following questions regarding data retention
- What storage timescale do we communicate and adhere to?
- Is there a need for regular deletion of data beyond that timescale?
- Is there a need for anonymizing data beyond that timescale?
Pay attention to potential gaps in the encryption of data, and keep data on encrypted media. For example, on data requests, the receiving machine should have encryption of drives and the data should not be transferred through means that could unreasonably expose it, like sharing via another cloud provider or messaging application. These gaps need to be identified and addressed.
Research your data collection mechanisms to identify when it necessitates that in the EU there needs to be a consent form shown to users. This however is usually not required when data is stored only for the operation of our application.
There is a need for regular risk assessments around the security of data. The cloud is very dynamic and adopting tooling to help keep up is recommended.
Adopt AWS config to assess, audit, and evaluate the configuration of AWS resources to ensure compliance (compliance-as-code framework), including
- Ensuring that configuration does not drift out of compliance. For example, someone accidentally turns off the encryption of a drive
- Keep up with potential security weaknesses
Choose conformance packs to get out-of-the-box best practices. General recommendation examples
Reduce the risk of secrets being exposed
Consider AWS Secret Manager with best practices enforced
Adopt compliance as code in any other viable areas. For example, adopting this as part of the IaC to make sure non-compliant changes will not pass a gate and will not be rolled out.
You need to be able to produce, delete and audit data easily as this can be demanded.
- Consider cloud trail
- Carefully consider the logging approaches in use and their retention policies
Anonymize data that is requested for debugging.

A simple approach would be to identify what data to anonymize like email addresses, phone numbers, names, etc., and have some automation that runs to apply any number of techniques like masking, encrypting, randomizing, and faking before that data is given to a support agent.

Organizational

There must be organisational measures in place to ensure that personal data is not compromised by accident or deliberately

Particularly with data transfer requests, ensure that whoever received the data does so responsibly and terminates the data after the intended usage is concluded.
Implement awareness and training of any policies and obligations internally to anyone who has access to data, environments that contain data, the configuration of environments, and any other updates that can cause compliance issues like exposure through code updates.
Limit who has access to data.

Internally, organizations should implement data handling and security protocols that align with GDPR. This includes establishing clear guidelines on data access, regular training programs on data protection, and emergency response plans for potential data breaches. For instance, regular workshops on handling PII (Personally Identifiable Information) and conducting mock data breach exercises can significantly enhance GDPR compliance.

Contract related

Ensure a clear outline of the following in your privacy policy documents
- The subject matter and duration of data processing
- The nature and purpose of the processing
- The type of personal data and categories of data
The data controller’s obligations and rights
Summary of required policy documents
- Information security policy
- Business continuity plan
- Privacy policy
- Data retention policy

Other recommendations

Map out all personal data you collect and categorize it
- Identify if any of that data needs to be anonymized
SQL Server allows for tagging data with classifications with useful tags out of the box and supports recommendations (which unfortunately seems to get royally wrong at times, and can be overly sensitive as shown below). These classifications are stored as extended properties on the columns so this can be scripted as part of DB creation and targeted in other scripts.
Delete/stop collecting any data you do not need. This approach aligns with Article 25 of the GDPR, which states that the controller shall implement appropriate technical and organizational measures for ensuring that, by default, only personal data which are necessary for each specific purpose of the processing are processed.
Communicate intent of use and ensure that future use adheres to that. For example, if you collect phone numbers for 2-factor authentication, you cannot later use them to send bulk messages for some other reason or share those details with an external party.
Identify sensitive data that needs to always be encrypted. For example, credit card details that would typically fall under that category
Anonymize production data used in other environments

Conclusion

If a software developer is looking to research GDPR, there are a few steps they could take:

Start by reading the official GDPR text to gain a foundational understanding of the regulation. This will help to familiarize the developer with the key terms and concepts of GDPR. The official text of GDPR can be found on the European Union’s website.
Explore available online resources and training materials, such as webinars, podcasts, and online courses. These can help to provide more detailed information on specific aspects of GDPR, as well as practical guidance on implementing GDPR in software development.
Engage with GDPR experts and legal professionals who can provide guidance and advice on GDPR compliance. This can help to ensure that the developer has a clear understanding of how GDPR applies to their specific situation and can provide guidance on how to implement GDPR-compliant software solutions.
Research best practices for GDPR compliance in software development. There are several resources available online that guide how to design, develop, and implement software solutions that comply with GDPR.
Participate in forums and online communities to share knowledge and experiences with other software developers who are also working on GDPR compliance. This can help to provide insights into the challenges and solutions of GDPR-compliant software development.

Overall, it’s important for software developers to have a comprehensive understanding of GDPR and to stay up to date on any updates or changes to the regulation. By taking these steps, a software developer can research GDPR thoroughly and gain the knowledge and skills needed to design and develop software solutions that comply with GDPR requirements.

Navigating GDPR compliance within the AWS Cloud requires a multifaceted approach that encompasses utilizing AWS’s robust security services, internal organizational measures, and clear contract-related policies. By understanding the specific requirements of GDPR and leveraging AWS tools effectively, organizations can not only comply with EU data protection laws but also enhance their overall data security posture. This research serves as a starting point for AWS users to build a GDPR-compliant environment, ensuring the privacy and security of personal data in the cloud.

⚠️ Much of GDPR falls under contract so it can be difficult to tell if the technological interpretation is complete, this would require guidance from a qualified and certified GDPR officer.

General Software Development Standards and Practices

Sun, 19 Feb 2023 00:00:00 GMT

Photo by Call Me Fred on Unsplash

General Software Development Standards and Practices

General Standards and Practices

Documentation Principles

✅ DO Recognize that certain tasks such as setting up a new environment, establishing infrastructure, or registering services are often repeated. These recurring tasks can benefit immensely from thorough documentation. Advocate for a Run Book approach, where the documentation is designed to guide the reader—whether it’s someone else or your future self—through each step of the process. This ensures that anyone following the guide can easily replicate the process without unnecessary trial and error.

✅ DO create a reference of all decisions of architectural relevance by employing Architecture Decision Records (ADRs). An architecture decision record is a short text file in a format similar to an Alexandrian pattern. (Though the decisions themselves are not necessarily patterns, they share the characteristic balancing of forces.) Each record describes a set of forces and a single decision in response to those forces. Note that the decision is the central piece here, so specific forces may appear in multiple ADRs.

✅ DO acknowledge that well-documented standards and practices are essential for any team that wants to be efficient, effective, and reliable. They provide a shared understanding of how things should be done, which helps to avoid confusion and errors. They also make it easier to onboard new team members and improve/maintain quality. It is important to do this for all operations, for exampled coding standards, infrastructure management standards, DevOps standards, expectations on team ceremonies and roles etc. Ensure that the team has the relevant processes and gates for reviewing things such as Pull Request, second pair of eyes on infrastructure changes etc., while avoiding red tape and rigidity. Save time by also identifying standards and policies that can be automated with things like linting.

These approaches align well with the principle of prioritizing working software over extensive documentation. It doesn’t require an inordinate amount of time to document these processes. However, the quality of the documentation should not be compromised. High-quality, concise documentation can save time and effort in the long run, making it a worthwhile investment.

Programming Principles

✅ DO use consistent coding styles. Consistency in coding styles can help make your code more readable and easier to maintain, so it’s important to establish clear coding standards and guidelines that all team members can follow.

In a recent project, the adoption of consistent coding styles, enforced through tools like ESLint for JavaScript, resulted in a significant reduction in merge conflicts and easier code reviews. This uniformity allowed new team members to quickly acclimate and contribute effectively.

✅ DO use version control. Version control tools like Git can help you manage and track changes to your codebase over time, making it easier to collaborate with other developers and keep your codebase organized.

✅ DO use continuous integration and delivery. Continuous integration and delivery (CI/CD) tools can help you automate your build, testing, and deployment processes, making it easier to deliver high-quality code more quickly and efficiently.

✅ DO write automated tests.: Writing automated tests can help you ensure that your code is functioning correctly and that changes you make to your code don’t introduce new bugs or errors.

A healthy test coverage helps to ensure that your library is both correct and robust, and can help to prevent regressions and other issues from creeping in as you add new features or make changes. It also helps to build confidence in your library among users, who can see that it has been thoroughly tested and validated.

A compelling case for automated testing was observed in a complex application where continuous integration, coupled with rigorous testing, caught a critical bug before deployment. This proactive detection saved significant time and resources that would have been spent in post-release nightmares.

✅ DO follow best practices for performance. Optimizing performance can be a key consideration in many software development projects, so it’s important to be mindful of best practices for performance, such as reducing the number of database queries or minimizing the use of expensive operations.

✅ DO prioritize security. Security should be a top priority in all of your software development efforts, and you should always be mindful of potential security risks and vulnerabilities in your code.

✅ DO embrace dependency injection. Dependency injection is a technique for managing dependencies in your code and can help you create code that is more modular, flexible, and testable.

✅ DO practice defensive programming. Defensive programming is a mindset that emphasizes anticipating and guarding against potential errors, bugs, and other issues in your code. This can help you create more reliable and resilient code.

✅ DO use consistent naming conventions. Consistent naming conventions can help make your code more readable and easier to understand, so it’s important to establish clear naming conventions and stick to them throughout your codebase.

✅ DO apply a library developer mindset on any task that you pick up.

This means thinking about the code you’re writing as a potential reusable library that can be used by others in the future, rather than just a one-off solution to a particular problem.

To apply this principle effectively, you should consider whether the code you’re writing has good potential for reuse. If so, it may be worth extracting it into a library.

To determine whether code has good reuse potential, you should consider a few key factors. One important factor is the level of abstraction in the code - code that is highly abstracted and modular is generally easier to reuse in different contexts. You should also consider the specific problem domain that the code is addressing - code that solves a common problem or performs a common task may be more likely to be reused than code that addresses a very specific, niche problem.

When deciding whether to extract code into a library, it’s also important to consider the maintenance burden of the library. If you extract code into a library, you’re committing to maintaining that library over time, fixing bugs, updating it to work with new versions of dependencies, and so on. This can be a significant investment, so you should be sure that the potential benefits of having a reusable library outweigh the costs of maintaining it.

By applying a library developer’s mindset to all programming tasks, you can create code that is more modular, more reusable, and more maintainable in the long term. This can help to reduce duplication of effort, improve code quality, and accelerate development speed overall.

✅ DO Leave the campground cleaner than you found it. This means that you should always strive to improve the codebase and associated assets (such as documentation and tests) when working on a project, rather than simply adding new features or functionality.

To apply this rule effectively, there are several actions you can take:

Observe documentation that is out of date, then update it. This helps to ensure that others who use your codebase can understand it and use it effectively. It also helps to maintain the quality of the project overall and can reduce confusion and errors in the future.
Observe code that violates your team’s standards and practices or principles, then address it. Consistency is key in software development, and adhering to established coding standards and principles can make the codebase easier to read and maintain over time.
Observe missing test coverage in critical areas, then add it. Tests are a crucial component of software development and can help to ensure that the codebase is robust and reliable. Adding missing test coverage in critical areas can help prevent regressions and other issues from creeping into the codebase.
Observe technical debt, then address it. Technical debt is a term used to describe code that has been written quickly or without proper consideration, and which may require additional work or refactoring in the future. Addressing technical debt can help to improve the quality of the codebase overall, and can make it easier to maintain and extend in the future.

Of course, there may be situations where applying the Boy Scout rule would be a big distraction, requiring a lot of attention and time. In such cases, it is justifiable to move on, however, it’s important to track these issues in your backlog or issue-tracking system, so that they can be addressed at a later time.

By following the boy scout rule, you can help to create a codebase that is clean, maintainable, and extensible, and that can be a valuable asset for your team and your organization.

✅ DO apply care to not introduce new technical debt. It’s important to apply care to not introduce new technical debt into your codebase. It can be tempting to take shortcuts or make compromises under pressure, but these decisions can lead to long-term problems and make it difficult to maintain or evolve the codebase over time.

One way to avoid introducing technical debt is to seek a second opinion, particularly from someone who is disconnected or immune to the current pressure. This can help to provide a fresh perspective and ensure that decisions are made with the long-term health of the codebase in mind.

To help identify potential sources of technical debt, consider the following factors that can impact the quality and maintainability of your codebase:

Performance: If code is poorly optimized or inefficient, it can lead to slow or unreliable performance, which can impact the user experience and scalability of the application.
Evolvability: Code that is inflexible or tightly coupled can be difficult to evolve, leading to problems when new features or changes are required.
Observability: If code is difficult to observe or monitor, it can be challenging to diagnose and fix issues when they occur.
Security: Code that is not properly secured or has vulnerabilities can put the application and its users at risk.
Maintainability: Code that is difficult to understand or maintain can be time-consuming and expensive to work with, making it more difficult to fix issues or add new features.
Scalability: Code that is not designed to scale can limit the ability of the application to handle increasing amounts of traffic or data.
Cost: Code that is inefficient or wasteful can be expensive to operate and maintain over time.
Robustness: Code that is not properly tested or validated can be prone to errors or failures, which can impact the reliability and usability of the application.

By paying attention to these factors and avoiding shortcuts or compromises, you can help to ensure that your codebase is maintainable, scalable, and resilient and that it can serve as a valuable asset for your organization over the long term

The decision to go into debt alters the course condition of your life. You no longer own it. You are owned - Dave Ramsey

✅ DO embrace within reason functional concepts for code that is easier to reason about, less prone to bugs, and easier to consume and test.

Immutability: Immutability is the idea that once a variable is assigned a value, that value cannot be changed. This can make code easier to reason about, as you don’t have to worry about unexpected changes to variables.
Referential transparency/Purity: Referential transparency is the idea that a function’s output depends only on its input, and not on any external state. This can make functions easier to test, as you don’t have to worry about the external state affecting the results of your tests.
First-class functions: First-class functions are functions that can be treated like any other value in the language. This means you can pass functions as arguments to other functions, return functions from functions, and store functions in variables.
H.O.F (Higher Order Functions): Higher-order functions are functions that take other functions as arguments, or return functions as output. This allows you to build more complex and flexible functionality by composing simpler functions.
Disciplined state: Disciplined state means that state changes are done in a controlled and predictable manner. This can help to reduce bugs related to state changes and make code more predictable and easier to reason about.
Closures: Closures are functions that capture variables from their surrounding environment. This can be used to create functions with “memory” or to build more complex functionality by composing simpler functions.
Lazy evaluation: Lazy evaluation is the idea that values are only computed when they are needed. This can help to reduce unnecessary computation and make code more efficient.

By embracing functional concepts in your code, you can create code that is easier to understand, test, and maintain, and that is less prone to bugs and unexpected behaviour. However, it’s important to use these concepts within reason and to balance them with other design principles and patterns, as appropriate for your specific use case

✅ DO be mindful of code style as a blocker. Does the style and structure of your code make it harder to read and maintain?

The style and structure of your code can have a significant impact on its readability, maintainability, and overall quality.

When code is poorly structured or styled, it can be difficult for others to understand and maintain. This can lead to slower development cycles, increased bugs, and higher technical debt. In contrast, well-structured and well-styled code is easier to read and maintain, making it faster and less costly to develop and maintain.

To avoid code style as a potential blocker, it’s important to adhere to best practices and style guides for your programming language and framework. This means writing code that is consistent in terms of naming conventions, indentation, comments, and other formatting elements.

In addition, you should take care to structure your code in a way that is easy to read and maintain. This might mean breaking down complex functions into smaller, more focused functions, or organizing code into well-defined modules or classes.

By being mindful of code style and structure, you can help to ensure that your code is easy to read and maintain, which can accelerate your development process and improve the quality of your code overall.

✅ DO enforce simplicity.

The virtue of simplicity cannot be overstated - simpler interfaces are easier to use for all kinds of users, and require less code to create, providing an obvious performance advantage.

Design decisions can be difficult, and often require user research to be done well. However, keeping things simple is always the right thing to do. This means not just simplifying or making things seem simple, but truly striving for simplicity in all aspects of your code and interfaces.

When creating code and interfaces, aim to do a little and do it well, for as many people as you can. This means creating interfaces that are straightforward to understand, with clear and intuitive functionality. It also means creating code that is focused and purposeful, with minimal complexity or unnecessary features.

By enforcing simplicity in your code and interfaces, you can create software that is not only easier to use and understand, but also faster and more performant. You can also reduce the likelihood of bugs and technical debt, making your code easier to maintain and extend over time.

So remember, when it comes to software development, simpler is almost always better. Strive for simplicity in everything you create, and focus on doing a little and doing it well, for as many people as you can.

Bugs hide behind less important but easily noticeable things or complexity

✅ DO avoid overly WET code, otherwise known as write everything twice, write every time, we enjoy typing, or waste everyone's time. This means that you should strive to avoid duplicating code unnecessarily, as it can lead to more work and maintenance in the long run.

However, it’s also important to recognize that there are times when duplication may be necessary or even preferable to the wrong abstractions. This means that in some cases, it may be better to have duplicated code that is easier to understand and maintain, rather than trying to create complex abstractions that are more difficult to work with.

Duplication can also be a useful tool for further understanding how to make the right abstraction. By seeing where and how code is being duplicated, you can identify common patterns and determine where an abstraction may be appropriate.

In general, the key is to find a balance between avoiding unnecessary duplication and creating abstractions that are too complex or difficult to work with. This means being mindful of where duplication is occurring and why, and seeking to create abstractions that are simple, intuitive, and easy to understand and maintain.

By avoiding overly WET code and finding the right balance between duplication and abstraction, you can create code that is both efficient and maintainable, and that can be a valuable asset for your organization over the long term.

✅ DO ensure that all logs are emitted to console as structured and compact json logs. This follows the 12 factor Apps approach to logging and is ideal for log targets to analyze the logs. DO NOT ship logs directly to a log target, in the case of k8s a side car agent is responsible for scrapping pod logs and ensuring that they get shipped out to the sink and any other heavy lifting like dealing with network issues, buffering etc.> A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage log files. Instead, each running process writes its event stream, un-buffered, to stdout.

✅ DO pay close attention to the size of log or metrics payloads. Overly large data payloads should be avoided. As a safety net the config limits the depth of destructuring and caps the string length, but care is still required as this can be bypassed with context and scopes.

🛑 DO NOT repeat yourself. This principle states that every piece of knowledge or information within a system should have a single, unambiguous, and authoritative representation.

When information is duplicated or repeated across a system, it can lead to inconsistencies, errors, and maintenance headaches. It can also make it difficult to update or modify the system, as changes may need to be made in multiple places.

To avoid repetition and adhere to the DRY principle, it’s important to identify common patterns and extract them into reusable functions, modules, or components. This can help to reduce duplication and make code easier to understand and maintain.

By following the DRY principle, you can create systems that are more reliable, easier to maintain, and less prone to errors and other issues. It can also make it easier to modify and extend the system over time, as changes can be made in a single place and propagated throughout the system.

Apply care to strike the right balance with DRY, premature abstractions can be worse especially if attained by speculating future usages.

🛑 DO AVOID leaky abstractions. A leaky abstraction exposes details or complexities that should be hidden from the consumer, making the abstraction more difficult to use and maintain.

A classic example of a leaky abstraction is an implementation of an HttpClient wrapper that leaks HTTP details out to the consumer. In this case, the HttpClient wrapper should abstract away the details of the HTTP protocol, making it easier to use and maintain. However, if the wrapper leaks HTTP details to the consumer, it becomes less useful and more difficult to work with.

To avoid leaky abstractions, it’s important to carefully consider the scope and purpose of each abstraction you create. The abstraction should be designed to hide details and complexities, not to expose them to the consumer.

GIT

✅ DO add at least two branch suffixes as part of new branch names, an indication of your name and change type to have the name syntax [name]\[change-type]\[devops-code], i.e jane\bug\10111. By using this naming convention, you can create a branching hierarchy that is easy to navigate and manage. The name makes it easy for one to target specific branches, for example, to clean up their branches. The change types make it easy to navigate branches by the type of change, such as a new feature implementation, bug fix, architecture changes, or documentation-only changes.

Here are the tags that you can use for the change type:

Tag	Description
feature	New feature implementation
bug	Bug fix
architecture	Architecture changes
doc	Documentation-only changes
pending	Work-in-progress branch that may likely be long-living

✅ DO clean up remote branches after they are merged. An easy way to guarantee this is to configure a Pull Request to delete the branch automatically on merge.

By cleaning up remote branches in this way, you can keep your repository organized and easy to navigate, and reduce clutter and confusion. It can also help to improve the overall quality and efficiency of your development process, as it reduces the need for manual cleanup and maintenance tasks.

✅ DO use the imperative mood in the subject line

Using the imperative mood in the subject line helps to ensure that your message is clear, concise, and action-oriented. It also makes it easier for others to understand what changes are being made and why.

For example, bad commit messages might include vague or uninformative subject lines, such as tweaks to package-info files or polishing. These messages don’t provide much information about what changes were made or why making it more difficult for others to understand the changes being made.

In contrast, good commit messages use the imperative mood in the subject line to clearly describe what changes were made and why. For example, fix failing CompositePropertySourceTests, rework @PropertySource early parsing logic, or add tests for ImportSelector meta-data. These messages provide clear and concise information about the changes being made, making it easier for others to understand the context and purpose of the changes.

The idea is that instead of a commit message saying what has been done it should be read as this is what will happen when the commit is applied.

.NET

✅ DO target the highest .net support surface area you can reasonably achieve and aim for a health test coverage when creating libraries.

Targeting the highest .NET support surface area means that you should aim to make your library compatible with the widest possible range of .NET applications and platforms. This will make your library more useful to a larger number of developers, and increase its potential for adoption.

Achieving a high level of .NET support surface area can be challenging, as it may require incorporating features and APIs from multiple versions of .NET. However, it’s important to strike a balance between achieving high support surface area and maintaining compatibility with the specific versions of .NET that are relevant to your users.

In addition to targeting a high support surface area, it’s also important to aim for healthy test coverage. This means writing comprehensive unit tests, integration tests, and other automated tests to ensure that your library functions correctly and reliably in a variety of scenarios.

Conclusion

In conclusion, adhering to established software development standards and practices is not just about maintaining code quality; it’s about building a foundation for sustainable and efficient software development. From the preciseness of documentation to the consistency in coding and strategic use of .NET frameworks, these practices pave the way for robust, scalable, and maintainable software systems. As we’ve seen through various examples, whether it’s through streamlined workflows in Git or the flexibility afforded by .NET’s broad support surface area, these standards are more than guidelines—they are pillars that uphold the integrity and success of our software development endeavors.

Testing Best Practices

Sun, 19 Feb 2023 00:00:00 GMT

Photo by ThisisEngineering RAEng on Unsplash

Testing Best Practices

Unit tests are an essential part of software development that help ensure that code works as expected, both initially and in the future. To maximize the value of unit tests, it is important to follow best practices.

This document provides a comprehensive set of testing best practices for developers to follow to create high-quality and low-maintenance tests that aid in building robust and maintainable codebases.

General

✅ DO aim for your tests to meet the following positive heuristics

Run fast

Tests should run quickly so that developers can run them frequently and not be slowed down by the testing process. Slow tests can be a barrier to TDD and other agile practices and can slow down feedback cycles
Not break often or be flaky

Tests should be reliable and not break often, especially due to reasons unrelated to the code being tested, such as timing issues, race conditions, or other environmental factors. This can lead to decreased trust in the testing process and can be frustrating for developers.
Not be high maintenance

Tests should be easy to maintain, update and refactor as code changes, so they don’t become a burden to the development process. This is especially important in large and complex codebases where maintaining the tests themselves can become a time-consuming and error-prone task.
Easy to read and understand

Tests should be easy to read and understand for developers who did not write the code being tested. This helps new developers on a team quickly understand what is being tested and how it’s being tested.
Have the same style and diligence as the code being tested

Tests should follow the same coding standards and best practices as the code being tested so that developers can maintain and update the tests as easily as they can the code.
Be deterministic

Tests should be deterministic, meaning that they always produce the same output given the same input. This helps ensure that the tests are reliable and can be trusted.
Survive major refactorings

Tests should survive major refactorings, meaning that they continue to work as expected even when large parts of the codebase are changed. This is important to ensure that developers can continue to rely on the tests even as the codebase evolves.
High coverage of important functionality

Tests should have high coverage of important functionality, helping to catch regressions and ensuring that changes don’t cause unintended side effects.
A good balance between unit tests, integration tests, and end-to-end tests, each serving its purpose in validating different aspects of the system.

🛑 DO AVOID the following negative heuristics of tests:

⚒️ Heavy work

A heavy reliance on manual testing, with developers spending a lot of time manually testing the system and debugging issues.
🧪 Logic in tests

Tests should focus on the behaviour of the code being tested rather than its implementation details. Including too much logic in tests makes them less readable, less maintainable, and more difficult to understand.
🪄 Magic values

Tests should use clearly defined, meaningful values instead of arbitrary or hard-coded values. Magic values can lead to confusion and make it difficult to understand what the test is verifying.
😔 Not writing tests because they are difficult or something is untestable

Not writing tests is a common problem among developers, especially when they are working on complex or legacy codebases. However, skipping tests because they are difficult or deemed untestable can lead to major issues down the line, such as regressions or bugs that go unnoticed.
🔫 Shotgun surgery

This can lead to unhealthy practices like reading private methods with reflection to test functionality that should be public, which can lead to brittle and difficult-to-maintain tests.
🌝 Eager test

Tests that are too eager expose a lot of irrelevant details about the system under test (SUT), which can distract the test reader from what the test is verifying. These details can change often and require frequent updates to the test code.
😬 Happy path

Happy path testing only tests the paths where things go well, but often misses potential issues and edge cases. It’s important to test both the happy path and other potential paths to ensure the system under test behaves as expected in all situations.
🦥 Slow Poke

Tests that take too long to run can be a drain on productivity and can lead to frustration among developers. These types of tests can often be optimized by using techniques such as parallelization or avoiding unnecessary setup.
🦣 Giant

Large tests are difficult to maintain and can cause problems if they fail unexpectedly. It’s best to break tests down into smaller, more manageable units that are easier to understand and maintain.
🤡 Mockery

Overuse of mocking can lead to tests that are difficult to read and understand, and can also make it more difficult to identify problems when they occur.
🔎 Inspector

Tests that violate the black box approach make assumptions about the inner workings of the system under test, which can lead to brittle and difficult-to-maintain tests.
🪰 Generous leftover

Tests that depend on run order or data that is shared between tests can be difficult to maintain and debug, especially when they fail unexpectedly.
🦸 Local hero

Tests that only pass in specific environments are not reliable and can lead to false positives. Tests should be designed to run in any environment and not be dependent on specific configurations.
🐜 Nitpicker

Tests that focus on irrelevant specifics that are likely to change frequently can be difficult to maintain and can lead to false positives.
📢 Loudmouth

Tests that clutter the console with diagnostic messages can make it difficult to identify problems when they occur.
🤥 The liar

Tests that skip the red-green-refactor process, use mocked SUTs, or are not specific enough can lead to false positives and make it more difficult to identify problems when they occur.
🆓 Free ride

Assertions that piggyback on an existing test when they should be split out can lead to false positives and make it more difficult to identify problems when they occur.
🔮 Mystery guest

Tests that use magic elsewhere or do not indicate the cause and effect between act, arrange, and assert can be difficult to understand and maintain.
🤫 Useless test Useless tests are those that do not provide any value to the project, and only exist to satisfy a code coverage metric. Such tests might pass or fail randomly, and require manual debugging to figure out what caused them to fail. This results in a waste of time and effort for the team. Always make sure that your tests are relevant and valuable to the project.
🪲 Flaky test

Flaky tests fail under specific conditions like time of day, day of the week, or hardware configuration. These tests are usually caused by race conditions, timing issues, or random data. Flaky tests are frustrating for developers because they are hard to reproduce and debug. It is crucial to fix flaky tests as soon as possible and ensure that your tests run reliably in all environments.
💹 High maintenance tests

Tests that are high maintenance are those that are fragile and require constant updates as the codebase changes. UI-driven tests are a common cause of high-maintenance tests. Overly complex tests, tests that are tightly coupled to the implementation, and tests with a lot of setups are also more likely to be high maintenance. High-maintenance tests consume a lot of resources and can make it difficult to refactor the codebase.
🐡 Lost tests

Lost tests are those that do nothing and have no value. These tests might have been good at some point but have since been refactored to oblivion or were never implemented in the first place. Lost tests add noise to your test suite, making it harder to determine which tests are meaningful. Always make sure that your tests are relevant to the current state of the project.
🔏 Using uncleansed data

Using prod data as-is in tests can expose sensitive information and pose a security risk. Always use sanitized or synthetic test data when running tests, to ensure that sensitive information is not exposed.

⚙️ Ignoring business needs or issues

Tests that focus only on the technology and ignore business needs or issues can result in software that does not meet business requirements. It is essential to involve stakeholders and understand the requirements of the software before writing tests. Tests should focus on ensuring that the software meets business needs and requirements.
🕔 Involving testing late

Testing late in the development process can result in a lot of bugs being discovered late in the development cycle, which can be expensive and time-consuming to fix. It is essential to involve testing early in the development process to catch bugs as soon as possible. Writing tests alongside the code can also result in more effective tests and a more stable codebase.
🤐 Secret catcher

Secret catchers rely on exceptions happening in the code and being caught by the test runner. This approach can be brittle and difficult to maintain, as it relies on specific error messages and exceptions that can change over time. It is better to use a more proactive approach to testing, ensuring that the software meets the desired behaviour under normal conditions.
🔢 Line hitter

Tests that are only written to increase the code coverage metric, without any consideration for the quality of the tests, are known as line hitters. Line hitter tests do not provide any real value to the software and can be misleading. It is important to write meaningful tests that ensure the software meets business needs and requirements.

✅ DO use test doubles (such as mocks and stubs) to isolate the code under test from its dependencies. Test doubles can make it easier to write focused tests that only test the behaviour of the code under test.

✅ DO use code coverage metrics to identify untested code. Code coverage can be a useful tool for identifying parts of the codebase that are not being tested, which can help guide the creation of new tests.

✅ DO use property-based testing to generate large amounts of test data automatically. Property-based testing can be a useful technique for testing complex or edge-case behaviour.

✅ DO use continuous integration (CI) system to run tests automatically on each commit. A CI system can help catch regressions early and ensure that the test suite is always up to date.

✅ DO use version control systems to track changes in the tests. This makes it easy to roll back to a previous version in case of issues.

✅ DO use code reviews to ensure that tests are well-written and adhere to best practices.

✅ DO prioritize test cases based on the risk and impact of failure. This helps ensure that critical functionality is always tested.

✅ DO use test-driven development (TDD) to guide the development process. TDD can help ensure that the code is thoroughly tested and that new features are implemented correctly.

✅ DO use the classic AAA (Arrange Act Assert) and RGR (Red Green Refactor).

The AAA and RGR patterns are two common testing patterns that help to write clear, maintainable tests. The AAA pattern is a three-step pattern that consists of Arranging the test, Performing an Action, and Asserting the results. The RGR pattern is a three-step pattern that consists of writing a failing test (Red), making the test pass (Green), and then Refactoring the code. Following these patterns helps to ensure that the tests are clear, maintainable, and reliable.

ℹ️ When RGR is ignored there is the possibility of creating tests with false positives, also known as the liar.

✅ DO familiarize yourself with Fixtures and best practices around their usage.

A fixture is a fixed state that the system is in for a test. Fixtures help to ensure that tests are reproducible and predictable. For example, if a test relies on a particular file being in a certain state, the test can set up a fixture to ensure that the file is in the correct state for the test to run. It is important to familiarize yourself with fixtures and best practices around their usage to ensure that tests are reliable and reproducible.

✅ DO consider live unit testing tooling and practices.

Live unit testing is a tool that runs unit tests automatically as the code is being developed. This tool can help to identify issues quickly and efficiently. By running the tests automatically, developers can focus on writing code and receiving immediate feedback on the code’s behaviour. Consider using live unit testing tooling and practices to improve the speed and reliability of testing.

✅ DO use data generation sparingly and only in cases where specific data is not of relevance.

Data generation is a tool that helps to create test data automatically. It can help generate large amounts of data or data that would be difficult to create manually. However, it is important to use data generation sparingly and only in cases where specific data is not relevant to the test.

ℹ️ Apply caution with tooling like AutoFixture.

✅ DO write more tests. In particular, areas that require tests at all times are

Algorithms
Intricate business logic
Helpers and Utilities
Resolved bugs

ℹ In addition to having more test coverage that helps with more confidence to change existing code and ensuring that previously discovered regressions do not occur again, tests are very good at identifying issues, especially coupling. Code that violates certain core principles that ensure a healthy and maintainable code base is very difficult to test, writing more tests allows us to see this upfront.

✅ DO validate private methods by unit testing public methods.

Validating private methods through their corresponding public methods ensures that all the code within a unit is being tested, including any private code that is called by the public methods. This practice helps prevent changes in private methods from breaking the functionality of public methods. To validate private methods through their public methods, a developer needs to create test cases that call the public methods with input data that exercise the private methods. The output of the public methods is then validated to ensure that the private methods have produced the expected results.

✅ DO keep an eye on test metrics. These should answer questions about how fast tests run, whether there are regressions, how often tests fail, whether there are any tests that stand out as flaky, and what the code coverage is.

Test metrics can provide valuable insights into the effectiveness of your testing approach. Metrics can help answer questions such as: How long do tests take to run? Are tests failing more often than usual? Are certain tests consistently failing or passing? How much code is covered by tests? Tracking these metrics can help identify areas for improvement, such as slow tests or areas of code that need more coverage. It’s important to keep an eye on test metrics regularly and adjust your testing approach as needed to improve the quality and speed of testing.

✅ DO consider a data generator like Bogus paired builders.

Data generators like Bogus can be very helpful for creating test data. Test data generation can be time-consuming, especially if the data needs to be complex or varied. By using a data generator, a developer can easily create a large amount of realistic data to use in testing. Paired builders are a common pattern for generating data that have relationships or dependencies between different objects. They can help create data that is more representative of real-world scenarios and make it easier to write tests that cover a variety of use cases.

Over time the dependency list can grow and it is quite cumbersome to have to modify tests to add new mocks
Quite often most of the mocks do not require setup so you do not need to explicitly create the mock, it can be auto-created
The code can be made more succinct and easier to read

ℹ Often when a service is updated to add a new dependency, this indicates new behaviour and likely means you would add new tests, not modify existing ones. With this approach, you do not have to update the tests.

✅ DO name mocks vs mock object instances accordingly. Mocks from AutoMoq, new Mock<T>() or Mock.Of<T> should have the suffix mock but not the object instances from mock.Object i.e userMock for the mock and user for the object. This distinction can go a long way in the readability of tests.

The mocks should have the suffix Mock, while the object instances should not have this suffix. This helps in the readability of tests, making it clear which objects are mocks and which are not. For example, you can name the mock as userMock and the object instance as a user.

✅ DO make use of the name sut for the system under test. This small detail shines particularly when the test is heavy in the arrange stage, it needs to be clear what the sut is.

It is a good practice to use the name sut for the system under test in your tests. This makes it clear which object is being tested, especially in a test with a large setup. It also helps to avoid anti-patterns as it forces a test to have one SUT. The same standardization applies to what is expected with the name expected and results for any sut call results

ℹ️ This is in contrast to the actual entity name and actual result type i.e. deal. That approach means a lot of unique names and does not help in a large code base. As for the SUT, there is an understated benefit of immediately being able to identify what is the system under test, especially in a test with a large setup. This can also avoid anti-patterns as it forces a test to have one SUT. The same standardization applies to what is expected with the name expected and result for any sut call results.

✅ DO make use of the name result, for the result to be asserted in a test.\

It is recommended to use the name result for the variable that holds the result of the system under test in your tests. This makes it clear what the variable represents and which variable is being asserted.

✅ DO abstract things out to keep arrange phase clean.

It is a good practice to abstract things out to keep the arrange phase of your tests clean. This helps to reduce duplication and keep the tests concise. However, when doing this, make sure that it is still clear what is going on entirely from reading the test. It is very easy to introduce the mystery guest while doing this.

✅ DO favour naming tests in the format given then should or given when then should. Long test names are perfectly fine.

The recommendation to name tests in the format given then should or given when then should is to make the intent of the test clear and to make it easy to understand what the test is doing. The given section sets up the initial conditions for the test, the when section describes the action being taken, and the then section describes the expected outcome.

Using this naming convention can make it easier to write tests and can make it easier to read and understand the tests later on. The long test names may seem cumbersome, but they can make the tests more readable and self-documenting.

For example, consider the following test name: GivenAListOfIntegers_WhenSortingInDescendingOrder_ThenShouldReturnSortedListInDescendingOrder

This test name clearly states what the test is doing: it is given a list of integers, sorting them in descending order, and then it checks that the resulting list is sorted in descending order.

Using this naming convention consistently throughout the project can make it easier for developers to find and understand tests, and can make the codebase more maintainable

⛔ DO NOT use tools that do not adhere to mocking best practices like NSubstitute.

It is important to use mocking tools that adhere to best practices, as otherwise, your tests may be unreliable and difficult to maintain. NSubstitute is one such tool that does not follow best practices, so it is recommended to avoid it.

⛔ DO NOT test the inner details of components, take a black box approach. Inner details should be able to change without necessarily affecting a consumer. When tests rely on these details they become brittle and high maintenance with minimal to no value added.

It is important to take a black box approach to test components, as this ensures that your tests will be more resilient to changes in the implementation. If your tests rely on inner details, they may become brittle and difficult to maintain.

⛔ DO NOT aim for 100% code coverage instead aim for high-quality and low-effort tests. Focusing on code coverage regressions is however reasonable. Emphasis on this metric can lead to line hitter tests that developers create just to achieve a metric without really adding much value.

While high code coverage is important, it should not be the sole focus of your testing efforts. Instead, aim for high-quality tests that are easy to write and maintain. Focusing too much on code coverage can lead to tests that are low-value and difficult to maintain

⛔ DO not use the MemberData attribute for primitive types, only use this for complex types that cannot use InlineData. InlineData makes it easy to see the test and its test data easily, you lose this with a separate method for member data.

[Theory]
[InlineData(null, false)]
[InlineData("", false)]
[InlineData(" ", false)]
[InlineData("value", true)]
public void Sample(string input, bool expected){}

// IS BETTER THAN
[Theory]
[MemberData(nameof(TestData))]
public void Sample(string input, bool expected){}

⛔ DO NOT write tests with multiple responsibilities, instead a test should have a single focus. Multiple unrelated assertions are a red flag.

When writing tests, it’s important to keep in mind that each test should have a single responsibility and focus. This means that multiple unrelated assertions within a single test should be avoided. When a test has multiple responsibilities, it becomes more difficult to understand and maintain and may lead to false positives or false negatives.

Instead, tests should be written with a clear, single focus. This allows for better isolation of failures and makes it easier to diagnose and fix issues when they arise. It also helps to keep the tests readable and understandable, making it easier for other developers to understand what the test is testing and what the expected outcome is.

.NET

✅ DO make use of `AutoMoq to create instances of an object that have dependencies in tests.

The effort of mocking services is quite high given the typical number of dependencies
Mocking service dependencies manually in tests creates fragile tests that will need updating each time those dependencies change. Ideally tests should change when behavior of interest changes.

Problem scenario:

  var sut = new ValueService(
      Mock.Of<IJobService>(),
      Mock.Of<ICache>(),
      dataContext,
      Mock.Of<IServiceProvider>(),
      Mock.Of<IClaimsPrincipalProvider>(),
      new NullLoggerFactory(),
      Mock.Of<ILocalisedExceptionProvider>()
  );

In this case the only mock we care about is actually the data context and so the rest can be auto mocked.

  var autoMock = AutoMock.GetLoose(c => c.RegisterInstance(dataContext));

  var sut = autoMock.Create<ValueService>();

Additional cases:

  var autoMock = AutoMock.GetLoose(c =>
  {
      // if you need a real instance of JobService, automock will transitively mock JobService's deps as well
      c.RegisterType<JobService>().As<IJobService>();
      c.RegisterInstance(dataContext);
  });

  // setup can be done on the mocks using Moq's api as
  autoMock.Mock<IJobService>().Setup(...);
  autoMock.Mock<IJobService>().Verify(...);

✅ DO replicate the folder structure of the project being tested in your tests. This makes it easy to navigate between the two and helps ensure that tests are placed in logical locations.

✅ DO favour creating test projects for each project being tested. This allows tests to be parallelized at a project level, which can help speed up test execution time. Using a single test project can create a bottleneck and slow down the testing process.

✅ DO name test files with suffix tests

Naming test files with a suffix such as Tests help to indicate that the file is related to testing and makes it easier to find all test files in the codebase. The naming convention of using a Tests suffix is widely used and understood by developers in the .NET community, making it a best practice to follow.

For example, suppose you have a class DictionaryExtensions in your production code that provides additional functionality for working with dictionaries. In that case, it would be a good practice to create a separate file DictionaryExtensionsTests.cs, which contains unit tests for the DictionaryExtensions class. This way, when you are navigating through the codebase, it is easy to identify and locate the tests for a given class or module.

🎃 DO be aware of a GOTCHA with working with the EF core in-memory data context. This implementation chooses to enforce referential integrity for navigation properties. If you create an object that has a non-nullable navigation property and leaves this as null and then add it into the data context in your test, any attempt to read this back out will not return anything. All non-nullable navigation properties will have to be non-null as well for this to work. Quite the annoyance for tests but currently with no workaround.

This is an important consideration when using the EF Core in-memory data context for testing. If you encounter this issue, you will need to make sure that all non-nullable navigation properties are set to non-null values.

Jest and React Testing Library

✅ DO group related tests under a describe if multiple tests are in one file such as with tests for utils.

When multiple tests are in one file, it’s a good idea to group them using the describe function. The describe function creates a block that groups together several related tests, making it easier to read and understand the test suite.

describe('Utils', () => {
  test('should do something', () => {
    // test code
  });

  test('should do something else', () => {
    // test code
  });
});

✅ DO name test files with suffix tests i.e utils.test.ts, breadCrumbs.test.tsx

src/
  utils.js
  utils.test.js
  components/
    breadcrumbs.js
    breadcrumbs.test.js

✅ DO favour data driven tests with test.each over duplicated tests

test.each([
    [new Date(2000, 11, 12), false],
    [new Date(1970, 11, 12), false],
    [new Date(1969, 11, 12), true],
    [new Date(1950, 11, 12), true],
])("isDateBefore_1970 returns relevant boolean", (date: Date, expected: boolean) => {
    expect(isDateBefore_1970(date)).toBe(expected);
});

🎃 When setting up a mock with jest.mock, ideally you may want to set up one of the properties based on a variable so that you can use the same value in your assertions. Jest will however fail the test complaining that mocks cannot access outside variables. This is in place to avoid dirty mocks, however, you can bypass this by simply naming the variables with the mock suffix i.e mockUser

const mockUnknown = "#9E9E9E";

jest.mock("../../redux/store", () => ({
    getState: (): RecursivePartial<RootState> => ({
        app: {
            statusColors: {
                unknown: mockUnknown,
            },
        },
    }),
}));

⛔ DO NOT use ReactTestUtils. Use React Testing Library which is designed to enable and encourage writing tests that use your components as the end users do. ReactTestUtils has features like mock elements, which do not simulate how a user will use the components and encourages very bad testing approaches.

ReactTestUtils is a utility library that was introduced early in the development of React. It provides various methods for testing React components. However, using ReactTestUtils is no longer recommended for testing React applications. The library encourages testing the internal implementation details of the components, rather than their behaviour, which can result in tests that are tightly coupled to the implementation and are therefore more brittle and difficult to maintain.

Instead, the recommended approach for testing React components is to use React Testing Library. React Testing Library is a lightweight testing library that is designed to enable and encourage writing tests that use components as the end users do. Its API is designed to be intuitive and mirrors how users interact with components in the browser. For example, instead of directly accessing a component’s internal state and props, you interact with the component through its rendered output and the DOM nodes that it produces.

React Testing Library is focused on testing the behaviour of the components, rather than their implementation details. This approach makes it easier to write tests that are maintainable and do not break as often. Additionally, the library provides a better developer experience and makes it easier to write tests that are expressive and easy to read.

Examples

Here are some examples to demonstrate some of these practices at play in unison.

.NET

using System;
using Xunit;
using Moq;
using AutoMoq;
using Bogus;

public class OrderServiceTests
{
    // Naming convention for test method is in the format of
    // given [scenario], when [action], then [expected outcome]
    [Fact]
    public void GivenValidOrder_WhenSubmittingOrder_ThenOrderIsSavedAndConfirmationIsReturned()
    {
        // Arrange
        var sut = new OrderService();

        // Create a fake order using the Bogus data generator
        var fakeOrder = new Faker<Order>()
            .RuleFor(o => o.Id, f => f.Random.Guid())
            .RuleFor(o => o.CustomerName, f => f.Name.FullName())
            .RuleFor(o => o.ProductName, f => f.Commerce.ProductName())
            .RuleFor(o => o.Quantity, f => f.Random.Int(1, 10))
            .Generate();

        // Use AutoMoq to create an instance of the OrderRepository with the required dependencies mocked
        var mocker = new AutoMoqer();
        var mockRepository = mocker.Create<OrderRepository>();

        // Set up the mock repository to return the fake order when the Save method is called
        mockRepository.Setup(r => r.Save(It.IsAny<Order>())).Returns(fakeOrder);

        // Inject the mock repository into the SUT
        sut.Repository = mockRepository.Object;

        // Act
        var result = sut.SubmitOrder(fakeOrder);

        // Assert
        // Check that the Save method was called on the mock repository with the correct argument
        mockRepository.Verify(r => r.Save(fakeOrder), Times.Once);

        // Check that the confirmation message is not empty and contains the order ID
        Assert.True(!string.IsNullOrEmpty(result.ConfirmationMessage));
        Assert.Contains(fakeOrder.Id.ToString(), result.ConfirmationMessage);

        // Check that the returned order is not null and has the same ID as the fake order
        Assert.NotNull(result.Order);
        Assert.Equal(fakeOrder.Id, result.Order.Id);
    }
}

In this test, we see the following best practices being applied:

The test method name follows the format given [scenario], when [action], then [expected outcome].
The sut variable is used to clearly identify the System Under Test.
The result variable is used to clearly identify the result of the action being tested.
Mock objects are used to isolate the SUT from its dependencies.
AutoMoq is used to make it easier to create mock objects with dependencies.
The MemberData attribute is not used for primitive types in favor of InlineData.
The test is kept focused with a single responsibility.
The Fact attribute is used to mark the method as a test.
The OrderServiceTests class follows the naming convention of adding Tests as a suffix to the class being tested.
A shared data generator like Bogus is used to create fake orders for testing.
The test code is kept organized with clear comments and whitespace.
The Mock.Of<T> factory method is used for mock objects with no additional setup required.
The AutoMoq library is used to make tests more concise and easier to read.
The result variable is used to clearly identify the result of the action being tested.
The test file follows the naming convention of adding Tests as a suffix to the file being tested.
The InlineData attribute is used to supply test data directly to the test method.
The test method is kept focused with a single responsibility.

// Import the function being tested
import { formatName } from '../utils';

// Describe block for the group of tests
describe('formatName function', () => {
  // Single responsibility test case
  it('should format a first and last name correctly', () => {
    // Arrange
    const firstName = 'John';
    const lastName = 'Doe';

    // Act
    const formattedName = formatName(firstName, lastName);

    // Assert using fluent assertions
    expect(formattedName)
      .to.be.a('string')
      .and.to.equal('Doe, John')
  });

  // Data driven tests with test.each
  test.each`
    firstName      | lastName        | expected
    ${'John'}      | ${'Doe'}        | ${'Doe, John'}
    ${'Mary'}      | ${'Smith'}      | ${'Smith, Mary'}
    ${'James'}     | ${'Brown'}      | ${'Brown, James'}
  `('should format $firstName $lastName correctly', ({ firstName, lastName, expected }) => {
    // Arrange and Act
    const formattedName = formatName(firstName, lastName);

    // Assert using fluent assertions
    expect(formattedName).to.equal(expected);
  });

  // Clear and concise test descriptions with given-when-then format
  it('should capitalize the first letter of first and last names', () => {
    // Arrange
    const firstName = 'john';
    const lastName = 'doe';

    // Act
    const formattedName = formatName(firstName, lastName);

    // Assert using fluent assertions
    expect(formattedName).to.equal('Doe, John');
  });

  // Descriptive naming for test files and functions
  it('should return an empty string if no parameters are passed', () => {
    // Arrange and Act
    const formattedName = formatName();

    // Assert using fluent assertions
    expect(formattedName).to.equal('');
  });
});

This test incorporates best practices such as using clear and concise test descriptions in the given-when-then format, having descriptive naming for test files and functions, using data-driven tests with test.each, using single-responsibility tests, and utilizing fluent assertions for more readable and maintainable test code.

Automating developer machine setup

Sat, 27 Aug 2022 09:00:00 GMT

Photo by charlesdeluvio on Unsplash

Optimizing new developer machine setup

This article will outline how I tackled an inefficiency around the setting up of a new developer machine whenever someone new joins the team or someone gets a new machine. The existing process at the time was based on some text-based instructions that someone would have to follow manually, downloading applications and clicking through prompts, customizing tooling configuration, setting up tooling plugins, and manually verifying that all was set up well. This process had many flaws

On updating the instructions it was difficult to verify that the update was correct and fits in with the rest of the steps
The process was tedious and error-prone. It was easy to forget a step or struggle with out-to-date instructions or screenshots based on updates to tooling from the time the instructions were written.
The process took time. While focusing on other onboarding processing and actioning the setup in between a new developer typically required a whole week to complete the setup.

As an engineering team we could do better and what we needed was to automate the process, but how? To best determine that we needed the criteria for automation and this was the requirement

The automation should run within a time constraint of an hour considering an average internet connection for downloading required applications and SDKs
Existing tooling should be leveraged. For example SDKs like node, dotnet, etc. ship with CLI tooling that could be utilized. I would further explore automation tools for setting up applications like Ansible, Chef, Chocolatey, etc.
After machine setup, verification should be run to ensure that the setup was a success.
The process should largely be customizable, giving developers flexibility over the tooling they use and how it is configured
The setup should ideally have one entry point. A requirement to run multiple commands while better would still be counter to the goals

From this, I further looked at what the automation details would be. Here are some examples of things a team may want to tackle:

Installing required tooling like SDKs, IDEs, etc
Pulling down the base repository on the correct branch
Installing code dependencies via tools like npm and NuGet
Configuring git settings like autocrlf to team standard
Setting up customized docker containers like Redis
Validating required tool version specifics like .net, node, npm
Validating the setup via executing certain steps in your repository, like restoring dependencies, linting, running tests, creating builds, etc.
Running custom migrations like database migrations with DbUp or Entity Framework
Verifying that all required tooling has been installed

After thorough research, given my use case, I settled for the following:

Leverage Chocolatey for installing required software and plugins with a script that a developer can customize
Leverage SDKs in use for further setups and verifications like dotnet and npm commands
To ensure a single entry point while utilizing various tooling, a simple JavaScript script would be leveraged to orchestrate everything. Other scripting options like Powershell would have been viable, but JavaScript was specifically chosen as the skills to maintain this already existed in the team.

Here is an example chocolatey script setup-packages.config

<?xml version="1.0" encoding="utf-8"?>

<packages>

  <!-- mandatory -->

  <package id="nodejs" version="18.8.0" />

  <package id="dotnet-6.0-sdk" />

  <package id="netfx-4.8-devpack" />

  <package id="vscode" />

  <package id="vscode-eslint" />

  <package id="vscode-codespellchecker" />

  <package id="visualstudio2022enterprise" />

  <package id="visualstudio2022-workload-netweb" />

  <package id="nswagstudio" />

  <package id="docker-desktop" />

  <package id="sql-server-2019" />

  <package id="sql-server-management-studio" />

  <package id="git" />

  <package id="lens" />

  <package id="kubernetes-helm" />

  <package id="kubernetes-cli" />

  <package id="awscli" />

  <package id="eksctl" />

  <package id="terraform" />

  <package id="putty" />


  <!-- Optional, can substitute with own tools of choice -->

  <package id="gitextensions" />


  <package id="vscode-vsliveshare" />

  <package id="vscode-gitlens" />

  <package id="vscode-csharp" />


  <package id="googlechrome" />

  <package id="lastpass-chrome" />

  <package id="postman" />

  <package id="chocolateygui" />

  <package id="powertoys" />

  <package id="7zip" />

  <package id="paint.net" />

  <package id="microsoft-windows-terminal" />

  <package id="oh-my-posh" />

  <package id="obsidian" />


  <package id="devtoys" />

  <package id="pulumi" />

  <package id="mremoteng" />

</packages>

Here is an example implementation of an orchestration script setup.js

import { spawnSync } from 'child_process';

import {
  MessageLevel,
  colorMessage,
  versionValidators,
  executeCommand,
  executeInteractiveCommand,
  indent,
} from './setup-module.mjs';

import process from 'process';

import { rename, copyFile, existsSync } from 'fs';

import packagesJson from './package.json';

const engines = packagesJson.engines;

console.log(colorMessage(MessageLevel.Warning, '\n⚠ prerequisites:'));

console.log(
  colorMessage(
    MessageLevel.Info,

    'Chocolatey should be installed on this machine for this process to complete successfully',
  ),
);

console.log(colorMessage(MessageLevel.Info, 'The docker daemon should be running'));

spawnSync('pause', { stdio: 'inherit', shell: true });

console.log(colorMessage(MessageLevel.Info, '▶️ Starting repository verification\n\n'));

executeCommand('git --version', 'Verifying if Git is installed');

console.log(colorMessage(MessageLevel.Info, '▶️ Updating repository'));

executeCommand('git checkout develop', 'Fetching latest changes');

executeCommand('git pull', 'Checking out develop branch');

console.log(colorMessage(MessageLevel.Info, '▶️ Configuring Git'));

executeInteractiveCommand('git config --global core.autocrlf true', 'Set core.autocrlf=true');

console.log(
  colorMessage(
    MessageLevel.Info,

    '▶️ Installing applications with chocolatey with no prompts. Check results.txt for status.',
  ),
);

executeCommand('choco install setup-packages.config > result.txt -y', 'Trigger chocolatey');

console.log(colorMessage(MessageLevel.Info, '▶️ Verifying installations'));

executeCommand(
  "choco feature enable --name='useEnhancedExitCodes' -y",

  "Enable chocolatey feature '' to get enhanced exit codes",
);

executeCommand('dotnet --version', 'Verifying if dotnet is installed');

executeInteractiveCommand('nswag --versions', 'Verifying if NSwag Studio is installed');

executeCommand('code --version', 'Verifying if Visual Studio Code is installed');

executeCommand(
  'choco find visualstudio2022enterprise -r -l',

  'Verifying if Visual Studio 2022 Enterprise is installed',
);

executeCommand(
  'choco find visualstudio2022-workload-netweb -r -l',

  'Verifying if ASP.NET and web development workload for Visual Studio 2022 is installed',
);

executeCommand(
  'choco find sql-server-2019 -r -l',

  'Verifying if Microsoft SQL Server 2019 Developer Edition is installed',
);

executeCommand(
  'choco find sql-server-management-studio -r -l',

  'Verifying if Microsoft SQL Server Management Studio is installed',
);

executeCommand(
  'choco find docker-desktop -r -l',

  'Verifying if Docker Desktop is installed',

  'https://www.docker.com/products/docker-desktop/',
);

console.log(colorMessage(MessageLevel.Info, '▶️  Setting up Custom Redis container'));

executeCommand('docker run --name dev-redis -p 6379:6379 -d redis', 'Creating Redis container');

console.log(colorMessage(MessageLevel.Info, '▶️  Validating tooling versions'));

const versionValidationErrors = Object.keys(engines)

  .map(function (engine) {
    if (!versionValidators[engine]) {
      throw new Error(`Engine '${engine}' is unsupported.`);
    }

    return versionValidators[engine](engines[engine]);
  })

  .filter(Boolean);

if (versionValidationErrors.length) {
  console.error(
    colorMessage(
      MessageLevel.Danger,
      '🚨 There were errors validating the compatibility of this computer:',
    ),
  );

  console.error(`\n${indent}${versionValidationErrors.join('\n    ')} \n\n`);

  process.exit(1);
}

console.log(colorMessage(MessageLevel.Info, '▶️ Verifying .NET Solution'));

executeCommand('dotnet restore Sample.sln --force', 'Restoring .net dependencies');

executeCommand('dotnet build Sample.sln --no-restore', 'Building .net solution');

executeCommand('dotnet test Sample.sln  --no-build --no-restore', 'Running .net tests');

console.log(colorMessage(MessageLevel.Info, '▶️ Updating database with DbUP'));

process.chdir('Sample.DbUpgrader');

executeCommand('dotnet run', 'Executing DB Upgrader');

console.log(colorMessage(MessageLevel.Info, '▶️ Verifying Node Application'));

process.chdir('../Sample');

console.log(colorMessage(MessageLevel.Info, '▶️ Setting up Node Application developer .env file'));

if (existsSync('.env')) {
  console.log(
    colorMessage(
      MessageLevel.Subtitle,
      `${indent}Creating back up of existing .env file '.env.old'`,
    ),
  );

  rename('.env', '.env.old', err => {
    if (err) throw err;

    process.exit(1);
  });
}

console.log(colorMessage(MessageLevel.Subtitle, `${indent}Creating new .env file from seed`));

copyFile('.env.seed', '.env', err => {
  if (err) throw err;

  process.exit(1);
});

executeCommand('npm install', 'Installing Node Application dependencies');

executeCommand('npm run test', 'Running Node Application tests');

executeCommand('npm run lint', 'Running Node Application linting');

executeCommand('npm run build', 'Creating production Node Application build');

console.log(colorMessage(MessageLevel.Success, `✅  Verification completed successfully`));

Supporting setup-module.js

import { spawnSync, execSync } from 'child_process';

import process from 'process';

import semver from 'semver';

const versionErrorMessage = (name, actual, desired, url) =>
  `This computer has ${name} ${actual} installed, but ${desired} is required ${url}`;

export const indent = '    ';

export const MessageLevel = {
  Success: 'success',

  Danger: 'danger',

  Info: 'info',

  Subtitle: 'subtitle',
};

const ColoredMessage = {
  [MessageLevel.Success]: { open: '\u001b[32;1m', close: '\u001b[0m' },

  [MessageLevel.Warning]: { open: '\u001b[33;1m', close: '\u001b[0m' },

  [MessageLevel.Danger]: { open: '\u001b[31;1m', close: '\u001b[0m' },

  [MessageLevel.Info]: { open: '\u001b[36;1m', close: '\u001b[0m' },

  [MessageLevel.Subtitle]: { open: '\u001b[2;1m', close: '\u001b[0m' },
};

const validateVersion = (desired, command, name, url) => {
  console.log(colorMessage(MessageLevel.Info, `${indent}▶️ Validating ${name}`));

  let actual;

  try {
    actual = execSync(command).toString().trim();
  } catch (error) {
    return `There was an error running the command '${command}':\n ${error.message}`;
  }

  const valid = semver.satisfies(actual, desired);

  if (valid)
    console.log(
      colorMessage(MessageLevel.Success, `${indent}✅  Success: Validating ${name} \n\n`),
    );

  return valid ? null : versionErrorMessage(actual, desired, name, url);
};

export const colorMessage = (modifier, message) => {
  return `${ColoredMessage[modifier].open} ${message} ${ColoredMessage[modifier].close}`;
};

export const versionValidators = {
  dotnet: function (desired) {
    return validateVersion(
      desired,
      'dotnet --version',
      'dotnet',
      'https://dotnet.microsoft.com/en-us/download',
    );
  },

  dotnet_format: function (desired) {
    return validateVersion(
      desired,
      ' dotnet format --version',
      'dotnet format',
      `Please update npm by running 'dotnet tool install -g dotnet-format --version "${desired}.*"`,
    );
  },

  node: function (desired) {
    return validateVersion(desired, 'node --version', 'node', 'https://nodejs.org');
  },

  npm: function (desired) {
    return validateVersion(
      desired,
      'npm --version',
      'npm',
      `Please update npm by running 'npm install --global npm@${desired}`,
    );
  },
};

export const executeCommand = (command, title, url = '', ignoreError = false) => {
  console.log(colorMessage(MessageLevel.Info, `${indent}▶️ ${title}`));

  const result = spawnSync(command, { stdio: 'inherit', shell: true });

  if (result.status !== 0 && !ignoreError) {
    console.error(colorMessage(MessageLevel.Danger, `${indent}🚨 Failure: ${title} ${url} \n\n`));
    process.exit(result.status);
  }

  console.log(colorMessage(MessageLevel.Success, `${indent}✅  Success: ${title} \n\n`));
};

export const executeInteractiveCommand = (command, title, url = '') => {
  console.log(colorMessage(MessageLevel.Info, `${indent}▶️ ${title}`));

  const result = spawnSync(command, { shell: true }).stderr.toString().trim();

  if (result) {
    console.error(colorMessage(MessageLevel.Danger, `${indent}🚨 Failure: ${title} ${url} \n\n`));
    process.exit(result.status);
  }

  console.log(colorMessage(MessageLevel.Success, `${indent}✅ Success: ${title} \n\n`));
};

package.json for indicating setup dependencies and setting dependency version requirements via the engines with semantic versioning support.

{
  "name": "Setup",
  "description": "",
  "repository": "",
  "private": true,
  "type": "module",
  "engines": {
    "dotnet": ">=6",
    "dotnet_format": ">=6",
    "node": "18.8.0",
    "npm": "7.0.0"
  },
  "dependencies": {
    "semver": "7.1.3"
  }
}

This can be triggered with the command npm install; node --experimental-json-modules setup

This implementation met all the requirements criteria and in summary, was a success due to the following:

Process runs in just under an hour
On update, this can be retested by spinning up a clean virtual machine and triggering the whole process
Process is highly configurable, particularly the chocolatey configuration
Verification can be run on demand to ensure that a developer machine meets requirements, particularly around things like SDK versions
One would no longer have to wait weeks/months before using a certain tool or command to realize their machine is not correctly set up and lose hours trying to diagnose this. Verifications are now done on demand.

Adopt AutoMock with Autofac and Moq

Sat, 27 Aug 2022 00:00:00 GMT

Photo by Sander Sammy on Unsplash

Adopt AutoMock in dotnet tests for Autofac and Moq

Context

While the effort of manually mocking services in .NET tests can be high, understanding the underlying mechanism of AutoMock can greatly reduce this complexity. AutoMock, when integrated with Autofac and Moq, not only simplifies the process but also provides a more intuitive approach to creating mocks. Here’s a deeper exploration of how AutoMock works

Mocking service dependencies manually in tests creates fragile tests that will need updating each time those dependencies change. Ideally, tests should change when the behaviour of interest changes.

Problem scenario


var sut = new SomeService(
        Mock.Of<IDependency1>(),
        Mock.Of<IDependency2>(),
        Mock.Of<IDependency3>(),
        Mock.Of<IDependency4>(),
        dataContext,
    );

In this hypothetical case, the only mock we care about is the data context and so the rest can be auto-mocked.

Recommendation

Adopt Autofac’s Moq helper automock as a way to create service instance SUTs (System Under Test). Unlike Autofixture this will not mock the SUT as well but will only locate the ctor and auto mock dependencies, maintaining trust in the SUT.

Application scenario


var autoMock = AutoMock.GetLoose(c => c.RegisterInstance(dataContext));

var sut = autoMock.Create<SomeService>();

Now if SomeService were to have an additional dependency injected via the ctor which does not affect the behaviour we are testing, there is no need to update the test(s) as this will just get auto-mocked, ideally what we would have done manually anyway.

Additional cases


var autoMock = AutoMock.GetLoose(c =>

{

    //If you need a real instance of a dependency, auto mock will transitively mock the concrete implementation's dependencies as well

    c.RegisterType<Dependency1>().As<IDependency1>();

    c.RegisterInstance(dataContext);

});


//Setup and verifications can be done on the mocks using Moq's API as

autoMock.Mock<IDependency1>().Setup(...);

autoMock.Mock<IDependency1>().Verify(...);

A great thing about Auto mock is that it exposes the Moq API, meaning everything else is working with Moq.

Benefits

Over time the dependency list can grow and it is quite cumbersome to have to modify tests to add new mocks
Quite often most of the mocks do not require setup so you do not need to explicitly create the mock, it can be auto-created
The code can be made more succinct and easier to read

ℹ Often when a service is updated to add a new dependency, this indicates new behaviour and likely means you would add new tests, not modify existing ones. With this approach, you do not have to update the tests.

Is Onboarded To Portal

Adopt run books

Sat, 27 Aug 2022 00:00:00 GMT

Photo by Calle Macarone on Unsplash

Adopt run books

A Run Book / Operations Manual for known maintenance scenarios. These should act as a technical support knowledge base. Run books, often underutilized, serve as a beacon of clarity and order in this chaotic landscape. They are comprehensive, step-by-step guides that detail the processes and actions necessary for handling routine operational tasks and resolving incidents.

Example scenarios include:

Routine maintenance like database backups
Incident response like a security breach
Software deployments

ℹ Any known manual tasks or issue remedies should be documented as a run book including any scripts used to automate technical workflows or investigations of production issues.

⚠️ Please keep these run books up to date. If you run through one and notice something different or require additional steps please do account for this. If you solve a new technical issue do document this as a run book. This avoids solving the same problems over and over.

Best Practices

Ensure each step in the run book is clearly described and easy to follow. Avoid jargon and overly technical language unless absolutely necessary.
Keep run books up-to-date with the latest procedures and changes in the infrastructure. Outdated run books can lead to ineffective or harmful actions.
Regularly test the procedures in your run books to ensure they work as expected. This is especially crucial for critical incident response run books.
Encourage team members to provide feedback on the run books based on their experiences. Continuous improvement is key to maintaining effective run books.
Incorporate run books into training programs for new staff to ensure they are familiar with established procedures and protocols.

By adopting these practices, organizations can maximize the effectiveness of their run books, leading to more efficient and reliable operations and incident management.

Template

# Description

## Situation

What is the situation that this run book addresses?

## Remedy

What steps need to be taken to remedy this?

Example

How to assume an AWS IAM role via the CLI

You need to assume a role from the AWS CLI

Situation

Assume/Impersonate a role that has the rights you desire using aws sts assume-role command

Solution


aws sts assume-role --role-arn "arn:aws:iam::[Account Id]:role/[Role Name]" --role-session-name [Session Name]

Depending on your workflow, ensure that the AccessKeyId, SecretAccessKey, and SessionToken output from that command are stored where they need to be. Also, ensure that the region is set to the desired target. A simple approach is to create a new section in the credentials file and mark that as the default with the required values.

Example:


[default]

aws_access_key_id = [Access Key]

aws_secret_access_key = [Secret Key]

On completion, a session token will be added to the credentials file:

[default]

aws_access_key_id = [Access Key]

aws_secret_access_key = [Secret Key]

aws_session_token=[Security Token]

⚠️ Assuming a role provides a short-lived token, if you suddenly lose access again, check that the token has not expired. In which case you will need to assume the role again.

How to assume an AWS IAM role via the CLI

Sat, 27 Aug 2022 00:00:00 GMT

Photo by engin akyurt on Unsplash

Assuming an AWS IAM role via the CLI is a crucial part of managing your AWS resources securely. This guide will walk you through the process, step by step, and provide best practices to ensure a smooth experience.

Approach

Assume/Impersonate a role that has rights you desire using aws sts assume-role command

aws sts assume-role --role-arn "arn:aws:iam::[Account Id]:role/[Role Name]" --role-session-name [Session Name]

Depending on your workflow, ensure that the AccessKeyId, SecretAccessKey, and SessionToken output from that command are stored where they need to be securely. Also, ensure that the region is set to the desired target. A simple approach is to create a new section in the credentials file and mark that as the default with the required values.

Example:

[default]
aws_access_key_id = [Access Key]
aws_secret_access_key = [Secret Key]

On completion, a session token will be added to the credentials file:

[default]
aws_access_key_id = [Access Key]
aws_secret_access_key = [Secret Key]
aws_session_token=[Security Token]

⚠️ It’s important to note that assuming a role provides a short-lived token. If you suddenly lose access, check that the token has not expired, in which case you will need to assume the role again.

Usage scenarios

Cross-Account Access: Assume a role in another AWS account to access resources securely.
Temporary Permissions: Provide temporary elevated permissions to users or services without sharing long-term access keys.

Conclusion

Assuming an AWS IAM role via the CLI is a powerful way to enhance the security of your AWS resources. By following these steps and best practices, you can efficiently manage your AWS access and permissions.

For more detailed information and advanced use cases, refer to the official AWS IAM documentation.

Resolve Cryptographic Exceptions in IIS

Sat, 27 Aug 2022 00:00:00 GMT

Photo by Etienne Girardet on Unsplash

Resolve Cryptographic Exception in IIS

Situation

An IIS deployed application is throwing a CryptographicException with the message System cannot find the specified file. Not a very useful message now is it?!

Remedy

This is most likely because the Windows Cryptographic Service Provider was trying to store or load a key for a certificate in the user store, and since a profile was not available, a cryptographic context was not available. This means the Process model has the Load User Profile set to false.

⚠️ The Load User Profile setting only applies to user accounts. Service Accounts like NETWORK SERVICE and ApplicationPoolIdentity have special handling.

What exactly happens when I set Load User Profile in the IIS pool?

The user profile is loaded. This includes their cryptographic store, environment variables such as %TEMP%, and other ones.

To enable the setting

Go to the IIS Manager
Go to the application pool instance
Click on Advanced settings
Under the Process model, set Load User Profile to true
Recycle the application pool and restart the linked web application for changes to reflect

Adopting decision records

Sat, 27 Aug 2022 00:00:00 GMT

Photo by Maksym Kaharlytskyi on Unsplash

Architecture Decision Records (ADRs)

An Architecture Decision Record (ADR) is a software design choice documentation that addresses a functional or non-functional requirement that is architecturally significant.

Having this record-keeping allows the team to easily go back in time to understand the current status and be better able to pivot with complete knowledge. These prove very useful for new joiners as well. Another key thing is that when the team decides on something and someone is tasked with putting it down as a record, often that could identify gaps or misalignment and allow addressing this early on particularly when examples are involved.

Example

Adopt SQL Temporal Tables for time travel

Created: 2023-01-01

Updated: 2023-01-05

Participants: Entire Tech Team

Status

Accepted

Context

We are building an Analytics database that needs to keep track of historical data, and we need to decide on the best way to implement this feature.

Alternatives

…
SQL Temporal Tables: This approach involves using the built-in SQL temporal table feature to automatically track the history of a table. This approach allows for easy querying of historical data and does not require additional storage as the history is stored in the same table as the current data.

Decision

We will use the SQL Temporal Tables approach for tracking history.

Reasoning

SQL Temporal Tables allow for easy querying of historical data. It also allows storing the history of a table transparently and is easy to use.

Pros

Temporal tables allow for easy tracking and querying of historical data.
…

Cons

Temporal tables can add complexity to a database schema.
…

Authoring recommendations

✅ DO ensure that the records are immutable. The decisions made in a previously published ADR should not be altered in a way that changes the decision, a new record should be made in the interest of maintaining the history.

✅ DO ensure that each record is specific to one decision only.

✅ DO ensure that all decisions are consistent with other team and organization agreed-upon principles and standards.

✅ DO reference other detailed documentation and supporting artefacts to avoid duplications.

✅ DO ensure that you document a clear argument as this is as important as the decision. Outline the position, including things like implementation cost, ownership cost, resource requirements, etc.

✅ DO follow a minimalist approach and document only the issues that need addressing or attention.

Template

NUMBER - Title

Created: [YYYY-MM-DD] when the decision was created

Updated: [YYYY-MM-DD] when the decision was last updated

Participants: [list involved participants/team]

Status

Context

Describe the context and problem statement, e.g., in free form using two to three sentences. You may want to articulate the problem in form of a question

Decision

This is the decision that was made if any alternatives were considered, list them here

Consequence

This is the consequence of the decision

Conclusion

In summary, ADRs are instrumental in maintaining a clear and accessible record of architectural decisions, aiding in team alignment, retrospective analysis, and onboarding new members.

Reason for Outage tracking

Sat, 27 Aug 2022 00:00:00 GMT

Photo by Антон Дмитриев on Unsplash

Reason For Outage (RFO) tracking

An RFO (Reason for Outage) is a body of knowledge for tracking known outages to learn from and to use as a reference when going back in time. This creates an actionable history of things that go wrong in production environments and can help in decision-making and resolution of potential issues.

Outages include

Resources that become unavailable. This could be due to many issues like network, infrastructure, faulty applications
Failed deployments
Manual application restarts necessitated by something going wrong

Outages do not include

Maintenance application/infrastructure restarts that are expected
Application downtimes due to deployments. This is not ideal however and can be avoided by adopting a different deployment strategy.

# Reason for Outage

# Outage

ISO Date

Description

Resolution

Example:

# Reason for Outage

# Surge in application Kubernetes pods restart

2022-08-27

There was an increase in the number of Kubernetes pod restarts starting at ... and flagged by DataDog at ... . See the screenshot below.

The surge in pod restarts was due to a faulty configuration of ... that led to an increase in memory usage and eventually some Out of Memory Exceptions that were resulting in pod restarts. See details on the ticket ... about the misconfiguration and resolution.

DataDog monitors were also updated by applying ... to ensure that when something like this happens we will be notified within a shorter window.

Secure AWS instance metadata

Sat, 27 Aug 2022 00:00:00 GMT

Photo by regularguy.eth on Unsplash

Secure AWS EC2 instance metadata

Instance metadata is data about your instance that you can use to configure or manage the running instance. Instance metadata is divided into categories, for example, host name, events, and security groups. As a practice, the exposing of data about an instance via an internal endpoint this is not limited to just AWS. A Cloud Metadata Attack attempts to abuse a misconfigured web server in order to access the instance metadata maintained by cloud service providers such as AWS, GCP, and Azure. All of these providers provide metadata via an internal unrouteable IP address 169.254.169.254 - this can be exposed by incorrectly configured web servers and accessed by using this IP address in the Host header field. This makes instance metadata security-sensitive given that if attackers get access to this they have specifics about your instances such as OS, security groups etc. that could allow for more targeted attacks with a higher success rate.

Solution

Do not trust user data in web server configuration. For example in NGINX configs the $host variable which is set from the ‘Host’ header can be controlled by an attacker
Disable instance metadata. This is not always an option, for example, in AWS there are services such as Systems Manager that rely on this metadata to correctly identify instances.
Secure the instance metadata. That is what we shall be focusing on.

Secure instance metadata

This change cannot be done in the AWS console and requires the use of the AWS CLI. Impersonate a role that has the rights to make this change if required or attain user login from the CLI.

Run a command to update the instance metadata as required. For example to enable the HTTP endpoint on a given instance:

aws ec2 modify-instance-metadata-options --instance-id [INSTANCE_ID] --http-endpoint enabled

⚠️ It is desirable to keep the instance metadata enabled as this is required by various AWS services including Systems Manager which we require for automated scanning and patching of EC2 instances. This means the http-endpoint flag should always be set to enabled. To enforce security however, we set the http-tokens option to required which enforces sending a signed token header with any instance metadata retrieval request and hence requires an IAM role which is harder to be exploited. Additionally the http-put-response-hop-limit is also set to 1.

aws ec2 modify-instance-metadata-options --instance-id [INSTANCE_ID] --http-endpoint enabled --http-tokens required --http-put-response-hop-limit 1

To get the status of a given instance run the following command

aws ec2 describe-instances --instance-id [Instance ID]

# this return a lot of data so preferable pipe to file to view easily

aws ec2 describe-instances --instance-id [Instance ID] > result.txt

C# Standards and Best Practices

Wed, 24 Aug 2022 19:00:00 GMT

Photo by Gabriel Vasiliu on Unsplash

C# Standards and Best Practice

A list of recommendations around working with C# and dotnet that can potentially be adopted by a team as standards. I highly recommend teams should have very clear and explicit standards documented, similar to the one below that is agreed upon and enforced.

There are multiple ways to enforce standards

Pull request verification manual checks
Automated checks like linters, Roslyn analyzers, etc.
Code scanning tools like Sonar

✅ DO make use of EditorConfig with .NET and leveraging Roslyn analyzers. This brings a powerful combination that enforces coding standards and promotes consistent code quality across your projects.

Consistent Coding Standards: EditorConfig provides a standardized format to define and enforce coding conventions such as indentation style, line endings, and whitespace usage.
Team Collaboration and Onboarding: When multiple developers work on a project, maintaining a shared understanding of coding standards becomes critical. EditorConfig allows you to specify and communicate these standards effectively, promoting collaboration and reducing time spent on code reviews. New team members can quickly adapt to the project’s coding style by simply configuring their editor to read the shared EditorConfig file.
Automated Standards Validation: Integrating EditorConfig with Roslyn analyzers takes consistency to the next level. Roslyn analyzers are powerful static analysis tools that can automatically validate your code against a set of rules defined in the EditorConfig file.
Customizable and Extensible: Both EditorConfig and Roslyn analyzers are highly customizable to fit your project’s unique needs. Additionally, you can create custom analyzers to enforce project-specific guidelines, ensuring that your codebase adheres to your organization’s best practices.
Cross-Editor and Cross-Platform Support: EditorConfig is supported by a wide range of popular editors, including Visual Studio, Visual Studio Code, and JetBrains IDEs.

Example

root = true

[*]
guidelines = 120

# All files
[*]
indent_style = space
# cancellation tokens must be forwarded to methods that accept them
dotnet_diagnostic.CA2016.severity = error
# cancellation tokens must come last in method signatures
dotnet_diagnostic.CA1068.severity = error

On creating an editorconfig for the first time using Visual Studio, it will infer some rules by analyzing the code, which gives a good head start.

Working with secrets

⛔ DO NOT store secrets within the source code or source code directories. These can be accidentally committed by introducing surface area for leakage. Make use of user secrets that are managed outside of a repository.

⚠️ The Secret Manager tool doesn’t encrypt the stored secrets and shouldn’t be treated as a trusted store. It’s for development purposes only. The keys and values are stored in a JSON configuration file in the user profile directory.

⛔ DO NOT use production secrets for non-production environments. These should be considered highly sensitive and isolated. For example, do not use the same storage location for development purposes as production and share access keys even if you isolate them in other ways like folders, etc.

⛔ DO NOT deploy secrets with an application, ideally these should be accessible through a controlled means like environment variables. This principle is particularly important if you adhere to the [12 Factor App](The Twelve-Factor App (12factor.net)) way of managing an application.

⛔ DO NOT use the : character when working with secrets in flat files to create a hierarchy, make use of the __ sequence which is supported by all platforms.

Projects

Structure

Do adhere to a strict structure of managing projects and consistently/religiously apply this.

✅ DO ensure that projects are located in /src and their tests are located in /test, not just as projects in VS Solution Explorer but including the file structure. In addition to a clean separation, the projects and tests have separate Directory.Build.prop files and .editorconfig files. This will decouple rules around code from tests, meaning for instance that Roslyn’s analysis is specific to each group and dependencies are managed differently.

✅ Consider adding changes that affect multiple projects and need to be consolidated such as project version numbers and dependencies to Directory.Build.props or Directory.Build.targets.

directory.build.props

This file is used to define project properties and configuration settings. It is primarily used to customize the build process at the project level. You would use directory.build.props when you want to define common properties, references, or other configuration settings that apply to multiple projects within a directory or solution. For example, you can use directory.build.props to set the default output directory, set the target framework, define preprocessor symbols, specify common package references, or enable/disable build features across multiple projects. This file helps ensure consistency and reduces duplication in the build configuration.

<Project>
    <PropertyGroup>
        <!-- Set the target framework version for all projects -->
        <TargetFramework>net5.0</TargetFramework>
    </PropertyGroup>

    <!-- Set some default usings -->
    <ItemGroup>
        <Using Include="System" />
        <Using Include="System.Linq" />
        <Using Include="System.Threading" />
        <Using Include="System.Threading.Tasks" />
        <Using Include="System.Collections.Generic" />
    </ItemGroup>

    <!-- Custom Build Configuration -->
    <PropertyGroup>
        <!-- Custom properties specific to the build configuration -->
        <OutputPath>$(SolutionDir)Output\$(Configuration)\</OutputPath>
        <DefineConstants>DEBUG;TRACE</DefineConstants>
    </PropertyGroup>

    <ItemGroup>
        <!-- Specify default package references for all projects -->
        <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
    </ItemGroup>

    <!-- Custom Build Features -->
    <PropertyGroup>
        <!-- Enable/Disable custom build features -->
        <Optimize>true</Optimize>
        <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
    </PropertyGroup>

    <!-- Additional Properties -->
    <PropertyGroup>
        <!-- Additional project properties -->
        <MyCustomProperty>Custom Value</MyCustomProperty>
    </PropertyGroup>
</Project>

If you have multiple layers of director.build.props files, you can inherit from a base file, for example, if you have one at the solution root and a more specific one in the tests folder root that only applies to tests, you can do the following:

<Project>

 <!-- Inherit from one in solution root -->
    <Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))" />

    <ItemGroup>
        <PackageReference Include="FluentAssertions" Version="6.11.0" />
        <PackageReference Include="Microsoft.NET.Test.SDK" Version="17.6.2" />
        <PackageReference Include="Xunit" Version="2.4.2" />
        <PackageReference Include="Xunit.Runner.VisualStudio" Version="2.4.5">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
        <PackageReference Include="Moq" Version="4.18.4" />
        <PackageReference Include="Moq.AutoMock" Version="3.5.0" />
    </ItemGroup>

    <ItemGroup>
        <Using Include="Moq" />
        <Using Include="Xunit" />
        <Using Include="Moq.AutoMock" />
    </ItemGroup>

</Project>

directory.build.targets

This file is used to customize the build process by adding or overriding build targets. It allows you to insert custom build steps at specific points in the build process or modify the behaviour of existing build targets. You would use directory.build.targets when you need to perform additional actions during the build, such as code generation, resource updates, or file copying. You can also use it to override or extend the behaviour of existing build targets provided by the default build process.

For example, you can add a custom target to run a script before or after the build, modify the output files, or inject additional dependencies into the build process.

<Project>
    <!-- Custom Build Step -->
    <Target Name="CustomBuildStep" BeforeTargets="Build">
        <!-- Add custom build step to run before the build -->
        <Exec Command="echo Running custom build step..." />
    </Target>

    <!-- Copy Additional Files -->
    <Target Name="CopyAdditionalFiles" AfterTargets="AfterBuild">
        <!-- Copy additional files after the build completes -->
        <Copy SourceFiles="$(ProjectDir)ExtraFiles\*" DestinationFolder="$(OutputPath)" />
    </Target>
</Project>

Logging

✅ DO make use of Microsoft.Extensions.Logging as the primary API for logging. Do not use SDKs like Serilog or ApplicationInsights for this. 3rd party tooling can be changed at any time and in that case, should only be used for bootstrapping or wrapped around to avoid work on moving to something different.

ℹ For example you can adopt the use of Serilog for logger bootstrapping due to the power and flexibility it has with the supported sinks over Microsoft.Extension.Logging. Its use should then be limited only to that.

✅ DO make use of log scopes (log context) at all times. As an example, if a controller action has access to a correlation id for something that is being operated on, if this is added to the context and multiple call chains down the line some exception occurs and is captured there + logged, it will contain that correlation id which is very useful.


using (logger.BeginScope(new Dictionary<string, object>{

    ["CorrelationId"] = id,

}))

{

    // other method calls

}



using (logger.BeginScope("{Operation} running for department {Department}", operationName, departmentName))

{

    // other method calls

}

ℹ Do ensure that all context follows the key:value pair pattern. Scopes that only have values can be confusing.

ℹ Do consider making use of some form of middleware to extract and include common information that is useful in logs via the log context.

✅ DO ensure that log messages are as detailed and specific as possible.

✅ DO carefully consider the design of generic implementations of things like Job runners that are meant to be extended with additional implementations. These should have hooks to allow for adding generic logging or metrics tracking like timing operations in a single place as opposed to all implementations.

How ASP.NET WEB API middleware works is a good example of a hook that allows for intercepting a given pipeline. A more general approach would be to require implementations that override a virtual base class method that can be used to apply such a hook. The same scenario handled with implementing an abstract class would not offer such functionality.

✅ DO pay close attention to the data that is exposed in logs, especially when setting log scopes with object destructuring to avoid leaking sensitive information. Any secrets like passwords/API keys (even when encrypted) and personally identifiable information like email addresses should not end up in logs.

✅ DO pay close attention to the size of log or metrics payloads. Overly large data payloads should be avoided. As a safety net where possible set the config of your logger such as Serilog to limit the depth of destructuring and cap the string length, but care is still required as this can be bypassed with context and scopes.

Serilog example

{
  "Serilog": {
    "Using": ["Serilog.Sinks.Console"],
    "Destructure": [
      {
        "Name": "ToMaximumDepth",
        "Args": { "maximumDestructuringDepth": 4 }
      },
      {
        "Name": "ToMaximumStringLength",
        "Args": { "maximumStringLength": 100 }
      }
    ]
  }
}

✅ DO ensure that all logs are emitted to the console as structured and compact JSON logs. This follows the 12 factor Apps approach to logging and is ideal for log targets to analyze the logs.

✅ DO treat logging and metrics as auxiliary concerns, meaning they should not impact functionality i.e. by causing bottlenecks or unhandled exceptions. While you would not wrap log call around try/catch block you need to apply defensive coding esp. around the risk of null reference issues.

ℹ Logging to the console is one way to avoid bottlenecks and have agents scrape the logs and worry about shipping them. For deployments where file-based logging is required, in those cases find an approach that does not create a synchronous bottleneck on logging to IO. For example with Serilog, Serilog.Sinks.File sink can be configured to be wrapped around Serilog.Sinks.Async to reduce the overhead of logging calls by delegating work to a background thread as files are prone to I/O bottlenecks.

Serilog Example

{
  "Serilog": {
      {
        "Name": "Async",
        "Args": {
          "configure": [
            {
              "Name": "File",
              "Args": {
                "path": "Application.log",
                "rollingInterval": "Day",
                "rollOnFileSizeLimit": true,
                "retainedFileCountLimit": 5,
                "formatter": "Serilog.Formatting.Compact.CompactJsonFormatter, Serilog.Formatting.Compact"
              }
            }
          ]
        }
      }
    ]
  }
}

✅ DO bootstrap the logging and the application annotations (read by scrapping agents) in a way that ensures that at minimum logs/metrics are identifiable with what application sent the log, the level of the log, the source of the log (class/controller/action name, etc.), the deployment stack (i.e. k8s namespace) and other information like application region that are applicable. Additional information such as request hostname, accessed resource IDs, etc. can further enrich logs.

✅ DO tag logs that are not useful except in exceptional cases as Debug so that these by default are not shipped out to the sink. In the event of an exceptional need for these logs then the configuration can be updated to include Debug logs. Examples of these kinds of logs are generated SQL queries from EF Core and transactional logs such as indicating the start of a job.

⛔ DO NOT ship logs directly to a log target. In the case of docker-based deployments like Kubernetes, a sidecar agent is responsible for scrapping pod logs and ensuring that they get shipped out to the sink and any other heavy lifting like dealing with network issues, buffering, etc.

ℹ A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage log files. Instead, each running process writes its event stream, un-buffered, to stdout.

⛔ DO NOT hard code any logger configuration. These values should also be set in the application configuration to allow for flexibility in updating this in production environments without having to create new releases.

⛔ DO NOT use highly unique values for metric tags and log context like GUID IDs. This applies a strain on indexing of these logs and metrics and while tools in use may apply constraints on these or result in increased cost of usage. Instead use values you would likely be applying to bucket of data on like request method type, response status code, exception type, etc.

⛔ DO NOT use string interpolation with logs. Instead, adopt the message templates approach of structured logging.

A couple of reasons

Highly unique Logs are a strain on log indexes for log targets


_logger.LogInformation($"Deleting file: {job.FileContentId}");

//This creates unique logs for each file

Logs can be searched based on the message template


// IDEAL

_logger.LogInformation("Deleting file: {FileContentId}", job.FileContentId);



//This would allow searching by 'Deleting file: {FileContentId}'

Again very useful for observing our logs downstream as with most systems like Datadog, Prometheus, etc as searching by message template is more powerful and faster as opposed to substring searches.

Tests

An assumption that tests make use of xUnit is made here.

✅ DO aim for your tests to meet the following positive heuristics

run fast
not break often or be flaky
not be high-maintenance
easy to read and understand
have the same style and diligence as the code being tested
be deterministic
survive major refactorings

🛑 DO AVOID the following negative heuristics of tests:

🧪 Logic in tests
🪄 Magic values
😔 Not writing tests because they are difficult or something is untestable
🔫 Shotgun surgery (has to use unhealthy ways to perform tasks like reading private methods with reflection)
🌝 Eager test (exposes a lot of irrelevant details about SUT that distract the test reader from what affects the behaviour being tested)
😬 Happy path (only tests the paths where things go well)
🦥 Slow Poke (where you grab a coffee while they run)
🦣 Giant (super big test, the day it has issues it’s commented out)
🤡 Mockery (mocking on steroids)
🔎 Inspector (violates black box approach)
🪰 Generous left over (when separate tests do a baton exchange of data and often (but not always!) depend on run order)
🦸 Local hero (only passes in a specific environment)
🐜 Nitpicker (much like inspector but focuses on irrelevant specifics that are most likely to change easily/often)
📢 Loud mouth (clutters console with diagnostic messages)
🤥 The liar (skips red-green-refactor, mocked SUT, not specific enough leading to false positives)
🆓 Free ride (assertions that piggyback on an existing test when that should be split out)
🔮 Mystery guest (test reader is not able to see the cause and effect between act, arrange, and assert because stuff is magically done elsewhere)
🤫 Useless test (Shush) (Shush) (requires manual debugging to figure out what caused a test to fail)
🪲 Flaky test (fails under specific conditions like time of day, day of the week)
💹 High maintenance tests (UI-driven tests are a pathway to this hell)
🐡 Lost tests (they do nothing, maybe good refactored to oblivion, they never did anything in the first place)
🔏 Using uncleansed data (for example prod data as is in the test potentially exposing this)
⚙️ Ignoring business needs or issues (tests or test approaches that focus only on the technology)
🕔 Involving testing late
🤐 Secret catcher (relies on exceptions happening in the code and being caught by the test runner)
🔢 Line hitter (meant to appease the gatekeeper of the code coverage metric)

✅ DO make use of traits to segment tests, such as marking integration tests separately from unit tests. This brings significant advantages in managing and executing tests, especially in a continuous integration (CI) environment. By using traits, like the ones available in xUnit.NET, you can easily categorize and selectively run tests based on their characteristics. This helps with granular test execution as required, test result analysis and parallel test execution control.

[Trait("TestCategory", TestCategories.Integration)]
public class AnalysisDatabaseTests
{
}

public static class TestCategories
{
    public const string Integration = nameof(Integration);
}

In CI, the tests can be targetted with a filer, for example, the following command only runs integration tests

dotnet test --filter TestCategory=Integration

✅ DO write more tests. In particular, areas that require tests at all times are

Algorithms
Intricate business logic
Helpers and Utilities
Resolved bugs

ℹ In addition to having more test coverage that helps with more confidence to change existing code and ensuring that previously discovered regressions do not occur again, tests are very good at identifying issues, especially around the coupling. Code that violates certain core principles that ensure a healthy and maintainable code base is very difficult to test, writing more tests allows us to see this upfront.

✅ DO name mocks vs mock object instances accordingly. Mocks from automock, new Mock<T>() or Mock.Of<T> should have the suffix mock but not the object instances from mock.Object i.e. userMock for the mock and user for the object.

ℹ This distinction can go a long way in the readability of tests.

✅ DO make use of the name SUT for the system under test. This small detail shines particularly when the test is heavy in the arranging stage, it needs to be clear what the SUT is.

var sut = autoMock.Create<SecurityProvider>();

var token = sut.CreateToken(new SecurityTokenRequest());

token.Should().NotBeNullOrWhiteSpace();

✅ DO consistently use a builder for EF Core Data Contexts for creating a test data context. Having one place to create the data context will make it easier to maintain this over time should you need to make a change to the data context for all tests and remove boilerplate code.

    public static class InMemoryDataContextBuilder
    {
        public static SupportDataContext BuildSupportDataContext() {

            var options = new DbContextOptionsBuilder<SupportDataContext>()
                .UseInMemoryDatabase(Guid.NewGuid().ToString())
                .Options;
            options.Freeze();

            return new SupportDataContext(options);
        }

        public static DataContext Build()
        {
            var options = new DbContextOptionsBuilder<DataContext>()
                .UseInMemoryDatabase(Guid.NewGuid().ToString())
                .Options;
            options.Freeze();

            return new DataContext(options, new NullLoggerFactory());
        }

        public static DataContext Build(Func<(DbContextOptions<DataContext>, ILoggerFactory)> configure)
        {
            var (options,  loggerFactory) = configure();

            options ??= GetDefaultDbContextOptions();
            loggerFactory ??= new NullLoggerFactory();

            return new DataContext(options, loggerFactory);
        }

        private static DbContextOptions<DataContext> GetDefaultDbContextOptions()
        {
            var options = new DbContextOptionsBuilder<DataContext>()
                .UseInMemoryDatabase(Guid.NewGuid().ToString())
                .Options;
            options.Freeze();

            return options;
        }
    }

✅ DO consider a data generator like Bogus paired builders.

This will allow you to easily generate realistic and randomized test data. This is particularly useful when you need a large set of diverse data for your tests. Instead of manually creating test data, you can use Faker to generate it automatically, saving you time and effort

✅ DO make use of test builders for objects that potentially would get created over and over in tests.

// base implementation
    public abstract class Builder<T>
    {
        private readonly Dictionary<string, object> _properties = new();

        public abstract T Build();

        public static implicit operator T(Builder<T> builder) => builder.Build();

        public void Set<TProp>(Expression<Func<T, TProp>> action, TProp value)
            => _properties[((MemberExpression)action.Body).Member.Name] = value;

        public TProp Get<TProp>(Expression<Func<T, TProp>> action)
            => _properties.TryGetValue(((MemberExpression)action.Body).Member.Name, out var value)
                ? value as dynamic
                : default(TProp);
    }

// Example implementation
    public class UserBuilder : Builder<User>
    {
        public UserBuilder WithId(Guid id)
        {
            Set(x => x.Id, id);
            return this;
        }

        public UserBuilder WithName(string name)
        {
            Set(x => x.Name, name);
            return this;
        }

        public override User Build() =>
            new User
            {
                Id = Get(x => x.Id),
                Name = Get(x => x.Name),
            };
    }

    public class A
    {
        public static UserBuilder User => new();
    }


    public class Test
    {
        public void Test1()
        {
            User user = A.User
                .WithName("John Doe")
                .WithId(Guid.NewGuid());
                ...
        }
    }

ℹ Follow this principle for any object that gets created over and over in multiple tests, which should have a builder as one authoritative way to create it for tests.

🎃 DO be aware of a GOTCHA with working with the EF core in-memory data context. This implementation chooses to enforce referential integrity for navigation properties. If you create an object that has a non-nullable navigation property but leaves this as null and then adds it into the data context in your test, any attempt to read this back out will not return anything. All non-nullable navigation properties will have to be non-null as well for this to work. Quite the annoyance for tests but currently with no workaround.

⛔ DO not use the xUnit’s MemberData attribute for primitive types, only use this for complex types that cannot use InlineData. InlineData makes it easy to see the test and its test data easily, you lose this with a separate method for member data.

[Theory]

[InlineData(null, false)]

[InlineData("", false)]

[InlineData(" ", false)]

[InlineData("value", true)]

public void Sample(string input, bool expected){}



// IS BETTER THAN

[Theory]

[MemberData(nameof(TestData))]

public void Sample(string input, bool expected){}

Naming and structure

✅ DO name test projects in the format [name of project under test].tests ensuring that the suffix tests is always used. This makes it easy to identify test projects especially when you want to apply filters like in Directory.Builds.props or in build pipelines.

✅ DO name test files with the same name as the file being tested with the suffix Tests i.e. UtilsTests.cs

✅ DO mirror the folder structure of what is being tested in tests - /Services/Access/UserAccess.cs -> /Services.Tests/Access/UserAccessTests.cs

✅ DO favor naming tests in the format Given_Then_Should or Given_When_Then_Should. Long test names are perfectly fine provided they give a good insight into what the test is about.

ℹ This is particularly important as it makes it easy to navigate and find tests. Another key reason is those test runners (xUnit, Resharper), etc. parallelize tests at a project level, so if you have a test project with a ton of tests that becomes the critical path and you miss out on faster test runs.

✅ DO place tests above any auxiliary methods in a test file. All helper methods should be at the bottom of the tests.

⛔ DO NOT test multiple projects by adding all the tests in one project. The test projects ideally should mirror the project structure of what is being tested.

Implementation

✅ DO use Fluent Assertions for assertions over xUnit assertions. These are more readable, flexible, and powerful.

The XUnitToFluentAssertionsAnalyzer offers a Roslyn analysis rule enforcing this, so if you right xUnit assertion the build would fail.

XFA001: Use FluentAssertions equivalent

dotnet_diagnostic.XFA001.severity = error

✅ DO add generic test constructs such as builders and test data that can and should be reused in multiple test projects to a common shared project i.e. Tests.Common

✅ DO apply discipline within xUnit test projects by configuring the in-built Roslyn analyzers that ship with the core package as you would with any other standards. Particularly with new projects configure this to be very strict from the onset and gradually tweak this as you go along. Below is an example of a very strict configuration for xUnit.

[*.cs]
# xUnit1004: Test methods should not be skipped

dotnet_diagnostic.xUnit1004.severity = error



# xUnit1006: Theory methods should have parameters

dotnet_diagnostic.xUnit1006.severity = error



# xUnit1008: Test data attribute should only be used on a Theory

dotnet_diagnostic.xUnit1008.severity = error



# xUnit1013: Public method should be marked as a test

dotnet_diagnostic.xUnit1013.severity = error



# xUnit1014: MemberData should use nameof operator for member name

dotnet_diagnostic.xUnit1014.severity = error



# xUnit1025: InlineData should be unique within the Theory it belongs to

dotnet_diagnostic.xUnit1025.severity = error



# xUnit1026: Theory methods should use all of their parameters

dotnet_diagnostic.xUnit1026.severity = error



# xUnit2000: Constants and literals should be the expected argument

dotnet_diagnostic.xUnit2000.severity = error



# xUnit2003: Do not use equality check to test for null value

dotnet_diagnostic.xUnit2003.severity = error



# xUnit2004: Do not use equality check to test for boolean conditions

dotnet_diagnostic.xUnit2004.severity = error



# xUnit2007: Do not use typeof expression to check the type

dotnet_diagnostic.xUnit2007.severity = error



# xUnit2008: Do not use boolean check to match on regular expressions

dotnet_diagnostic.xUnit2008.severity = error



# xUnit2009: Do not use boolean check to check for substrings

dotnet_diagnostic.xUnit2009.severity = error



# xUnit2010: Do not use boolean check to check for string equality

dotnet_diagnostic.xUnit2010.severity = error



# xUnit2012: Do not use Enumerable.Any() to check if a value exists in a collection

dotnet_diagnostic.xUnit2012.severity = error



# xUnit2011: Do not use empty collection check

dotnet_diagnostic.xUnit2011.severity = error



# xUnit2013: Do not use equality check to check for collection size.

dotnet_diagnostic.xUnit2013.severity = error



# xUnit2015: Do not use typeof expression to check the exception type

dotnet_diagnostic.xUnit2015.severity = error



# xUnit2017: Do not use Contains() to check if a value exists in a collection

dotnet_diagnostic.xUnit2017.severity = error



# xUnit2018: Do not compare an object's exact type to an abstract class or interface

dotnet_diagnostic.xUnit2018.severity = error



# xUnit2019: Do not use obsolete throws check to check for asynchronously thrown exception

dotnet_diagnostic.xUnit2019.severity = error



# XFA001: Use FluentAssertions equivalent

dotnet_diagnostic.XFA001.severity = error

✅ DO favour data-driven tests where applicable over duplicated tests. This can be achieved with Theories using InlineData for compile time constants and MemberData for the rest.

[Theory]

[InlineData("yes", true)]

[InlineData("no", false)]

public void Sample input, bool expected)

{

    ...

}



[Theory, MemberData(nameof(TestCases))]

public Sample(string fileName, string expectedExtension)

{

    ...

}



public static IEnumerable<object[]> TestCases => new[]

{

    // input, expected

    new object[] {"test.txt", ".txt"},

    ...

};

✅ DO make it easy to tell apart the Arrange, Act, and Assert sections of your test and in particular to test what the system under test (sut) is.

✅ DO write tests that assert a single concept. While a single assert per test would be ideal, it is not realistic. Fluent assertions do have constructs in place to avoid situations where you have multiple related asserts that short circuit on the first failure.

[Fact]

public void Sample()

{

    var result = ...;



    // CLASSIC

    result.Id.Should().Be(id);

    result.Name.Should().Be(name);

    result.Value.Should().Be(value);



    // BETTER

    var expectedResult = ...;



    result.Should().BeEquivalentTo(expectedResult);

    // can omit properties

    result.Should().BeEquivalentTo(expectedResult, options => options.Excluding(x => x.Name));



    // ALTERNATIVE

    //Use this approach when necessary!

    using (var scope = new AssertionScope())

    {

        result.Id.Should().Be(id);

        result.Name.Should().Be(name);

        result.Value.Should().Be(value);

    }

}

ℹ This is also possible with collections

✅ DO use the most specific assertions, these give the most specific and useful failure messages as well.


// BAD

result.Should.Be(true);



// GOOD

result.Should.BeTrue();



// BAD

actual.Any().Should().BeTrue();

// Expected True, but found False.



// GOOD

actual.Should().NotBeEmpty();

// Expected collection not to be empty.

More examples here

ℹ There are Roslyn analyzers that ship with the core package to help enforce this.

✅ DO inject a NullLoggerFactory into tests unless you are interested in asserting the logs, then in that case use consider LoggerFactory double.

✅ DO make use of xUnit fixtures for setups that can be shared by an entire test file (Class Fixtures) or multiple test files (Collection Fixtures). The example below is a class fixture that bootstraps AutoMapper for use within a single test file.

    public class AutoMapperFixture<TProfile> where TProfile : Profile, new()
    {
        public AutoMapperFixture() =>
            Mapper = new MapperConfiguration(cfg => cfg.AddProfile<TProfile>())
                .CreateMapper();

        public IMapper Mapper { get; }
    }

    public class SampleTests : IClassFixture<AutoMapperFixture<MappingProfile>>
    {
        private readonly IMapper _mapper;

        public SampleTests(AutoMapperFixture<MappingProfile> fixture) => _mapper = fixture.Mapper;
    }

The following ⛔ DO NOT mix business logic with code that has direct access to infrastructure for example sending an email via SMTP, reading/writing a file, calling into system constructs such DateTime.Now, Task.Delay etc.

ℹ A core principle with tests is that they should have full control over the `sut which becomes problematic when you violate this rule. Infrastructure concerns should be implemented as adaptors and injected into code that requires it.


// PORT

public interface IDelayService

{

    Task Delay(TimeSpan delayDuration, Action continueWith);

}



// ADAPTOR

internal class DelayService : IDelayService

{

    public virtual async Task Delay(TimeSpan delayDuration, Action continueWith)

    {

        await Task.Run(async () => { await Task.Delay(delayDuration); })

            .ContinueWith(_ => continueWith())

            .ConfigureAwait(false);

    }

}



// TEST STUB

internal class DelayServiceStub : DelayService

{

    public AutoResetEvent DelayComplete = new AutoResetEvent(false);



    public override async Task Delay(TimeSpan delayDuration, Action continueWith)

    {

        await base.Delay(TimeSpan.Zero, continueWith);

        DelayComplete.Set();

    }

}

⛔ DO NOT have magic values in tests. Simple approaches like inlining variables to have a name can go a long way in test readability.


// BAD - why is this particular value of significance

result.Count.Should.Be(24);

⛔ DO NOT write tests with multiple responsibilities, instead a test should have a single focus. Multiple unrelated assertions are a red flag.

⛔ DO NOT mock the SUT. A key thing with the tests is to trust what is done in the act phase, which becomes hard when the SUT is being mocked as well.

⛔ DO NOT have conditional logic in tests. Rather make data-driven tests with theories or separate tests even.

Mocking

✅ DO use Moq to mock necessary dependencies within a test.

✅ DO consider when to use Mocks vs TestDoubles very carefully. Mocks should be a natural first choice, using doubles in cases where you cannot mock such as when dealing with library concrete classes, abstract classes, etc.

✅ DO use Mock.Of<> when no setup or assertions are required, over new Mock<>+ mock.object. Example:

// OK

var mock = new Mock<IDependency>();

var sut = new Sample(mock.Object);


// BETTER, more succinct

var sut = new Sample(Mock.Of<IDependency>());

Configuration

✅ DO validate app configuration on startup. Consider the use of FluentValidation for this. This will help you fail fast with regards to configuration immediately on a new deployment as opposed to hours/days later when an action that requires the configuration finally fires.

ℹ FluentValidation has great support out of the box for writing unit tests on validators, make sure to take advantage of this.

🛑 DO avoid reading configuration into a pre-initialized object. Creating an options class and hydrating it could lead to a situation where if the section is missing it will proceed silently, you will have an object with defaults for the properties and the app may not fail earlier or at all (by inconsistently handling cases with viable yet invalid config).

var appConfiguration = new ApplicationConfiguration();

configuration.GetSection(configurationName).Bind(appConfiguration);


// BETTER

var appConfiguration = configuration.GetSection(configurationName).Get<ApplicationConfiguration>();


// Now we would get null options if the section is missing and we can handle that case.

Language Features

✅ DO favour types that carry context over the context in the naming or otherwise assumed context. i.e. Favor the use of the TimeSpan type over a numerical value with a suffix in the name such CacheExpiryDays.

ℹ The TimeSpan type, in particular, is handled well in dotnet, for instance, you can use this with configuration and set your string config as '00:00:00' without having to worry about type conversions on reading config to a numeric type.

✅ DO consider using Enum Flags together with the relevant operator and Enum.HasFlag where possible over entities with multiple properties indicating Boolean states particularly when these are stored.

[Flags]

public enum UserFlags

{

    None = 0,

    SystemAccount = 1,

    Locked = 2,

    Disabled = 4,

    ...

}

✅ DO favour using directives instead of fully qualified type names. This helps to make the code cleaner.

// Avoid this
System.Collections.Generic.List<string> names = ...

// In favour of this
using System.Collections.Generic;

List<string> names = ...

✅ DO consider adding imports into the namespace when you have multiple imports from the current project. This will clean up the imports as they will no longer need to be fully qualified imports.

✅ DO make use of async overloads where available i.e. ToListAsync() over ToList().

✅ DO design code for async support upfront for Services/Usecases. Adding async support for a service method that has been reused in many places because there is now an await can be quite the refactor.

✅ DO favor Enumerable.Empty<T> over new new List<T> and Array.Empty<T> over new T[] {}. The static empty will return a compile-time constant as opposed to making a new empty allocation. This can be particularly troublesome when done in loops at scale and will put pressure on the garbage collector. This is also intent-revealing and makes it clear that the case is an empty list with no intent to add elements after

if (skip) return new List<T>();


if (skip) return Enumerable.Empty<T>();

✅ DO favour returning empty collections over nulls, particularly for data that crosses boundaries. An operation-like iteration over an empty collection is safe and does not lead to exceptions as would null objects.

✅ DO favour the use of file-scoped namespaces over block-scoped namespaces particularly to benefit from less indentation.

// ditch this
namespace SampleNamespace
{
    public class SampleClass
    {

    }
}

// for this
namespace SampleNamespace;

public class SampleClass
{

}

✅ DO make types static accordingly if there is no need for allocations, especially for things like utilities. This is also applicable to anonymous methods in C# 9 and forward and in particular makes it explicit that a lambda must not capture variables (should not be a closure).

✅ DO add a default None (semantically an unknown or invalid) of 0 to all Enums. This makes transmitting data over HTTP safer as unset Enum props will not default to a valid value by mistake.

public enum SampleEnum

{

    None = 0,

    ...

}

✅ DO favour using declarations over blocked scoped usings thereby reducing indentation and making code more readable.

// ditch this
        await using (var conn = new SqlConnection())
        {
            await conn.OpenAsync();
            await using (var cmd = conn.CreateCommand())
            {
                cmd.CommandText = ...;
                var result = await cmd.ExecuteScalarAsync();
            }
        }

// for this
        await using var conn = new SqlConnection();
        await conn.OpenAsync();
        await using var cmd = conn.CreateCommand();
        cmd.CommandText = ...;
        var result = await cmd.ExecuteScalarAsync();

⛔ DO NOT use negations in code unless necessary as those tend to be harder to read/comprehend

// This is easier to read
var response = isValid ? content : null;

// This is harder to read
var response = !isValid ? null : content;

⛔ DO NOT abuse null safety checks in a way that breaks the null semantics of a type in the interest of defensive coding. This can create confusion about what can be null and cascade from there as other developers extend the code.

var name = sample?.child?.child?.name;

// if any of these properties cannot be null then the defensive coding should not be applied.

⛔ DO NOT name async methods with an Async suffix, i.e. GetDataAsync.

ℹ Unless you are designing a library and adding async support side by side to the non-async (method overloads cannot differ only by return type) that is not warranted. In the CLR that was the exact use case for Microsoft and why they added that suffix leading to its adoption as a standard by many. However, given that most of us are either writing support on methods for one of the two but not both at the same time this rule should have been the exception really

⛔ DO NOT use async void. void returning async methods have a specific purpose: to make asynchronous event handlers possible. Async void methods have different error-handling semantics. When an exception is thrown out of an async Task or async Task<T> method, that exception is captured and placed on the Task object. With async void methods, there is no Task object, so any exceptions thrown out of an async void method will be raised directly on the SynchronizationContext that was active when the async void method started.

✅ DO make use of dictionaries and lookups in scenarios that call for them. In cases where data is loaded into a collection (list/array) and then enumerated for single/first or a group by some key like an id, that would best be a Dictionary for one key → one value or Lookup one key → multiple values and would give better performance at scale.

// Before

        var entities = await DataContext.Entity
            .AsNoTracking()
            .Where(a => a.Special)
            .ToListAsync();

        // later usage in a loop
        var targetName = entities.FirstOrDefault(x => x.Id == targetId)?.Name;

// Dictionary

        Dictionary<Guid, string> entities = await DataContext.Entity
            .AsNoTracking()
            .Where(a => a.Special)
            .ToDictionaryAsync(x => x.Id, x => x.Name);

        entities.TryGetValue(targetId, out var targetName);

// Lookup

        // Assuming multiple entities can have the same ID
        ILookup<Guid, string> entities = DataContext.Entity
            .AsNoTracking()
            .Where(a => a.Special)
            .ToLookup(x => x.Id, x => x.Name);

        var targetNames = entities[targetId].ToArray();

✅ DO favour the modern ways to test for nullability in C#. is and is not check for true nulls ignoring any operator overrides. By default the modern checks give a compiler error when the value is not nullable, the legacy gives a warning. This of course can be configured with Roslyn as desired.

// legacy

    if (item == null) { }
    if (item != null) { }


// modern

    if (item is null) { }
    if (item is not null) {  }

✅ DO simplify assignment on null variables with the single assignment target-typed new operator ?==. This is more succinct and the intent is easier to see on reading a single line.

// legacy

    if (request == null)
    {
        request = new Request
        {
            Special = true
        };
    }

// modern

    request ??= new Request
    {
        Special = true
    };

✅ DO split LINQ where clauses to make code more readable.

// before

    var result = await DataContext.Samples
        .AsNoTracking()
        .Where(x => !x.Deleted && x.Id == targetId && x.Special)
        .ToListAsync(cancellationToken);

// after

    var result = await DataContext.Samples
        .AsNoTracking()
        .Where(x => !x.Deleted)
        .Where(x => x.Id == targetId)
        .Where(x => x.Special)
        .ToListAsync(cancellationToken);

✅ DO split LINQ methods onto multiple lines for readability.


// before
    var customFields = await DataContext.Samples.AsNoTracking().Where(x => !x.Deleted).Where(x => x.Id == targetId).Where(x => x.Special).ToListAsync(cancellationToken);

// after

    var result = await DataContext.Samples
        .AsNoTracking()
        .Where(x => !x.Deleted)
        .Where(x => x.Id == targetId)
        .Where(x => x.Special)
        .ToListAsync(cancellationToken);

✅ DO favour the modern switch expressions over `switch statements. This makes for more succinct code that is easier to read and takes a functional approach.

// legacy

      string? recommendation;
      switch (state)
      {
          case TrafficLight.red:
              recommendation = "Stop";
              break;
          case TrafficLight.green:
              recommendation = "Go";
              break;
          case TrafficLight.amber:
              recommendation = "Proceed with caution";
              break;
          default:
              recommendation = "TrafficLight malfunction: proceed with caution";
              break;
      }

// modern

            string? recommendation = state switch
            {
                TrafficLight.red => "Stop",
                TrafficLight.green => "Go",
                TrafficLight.amber => "Procced with caution",
                _ => "TrafficLight malfunction: proceed with caution",
            };

✅ DO favour expression bodies for methods, properties, and ctors for single expressions for a more succinct syntax.

// legacy

    private Task<string> GetSample(string key)
    {
        return sampleService.GetSample(key);
    }

    protected async Task<List<SampleDto>> GetSample(Guid id)
    {
        return await DataContext.Sample
            .AsNoTracking()
            .Where(x => x.Id == id)
            .ToListAsync();
    }

// modern

    private Task<string> GetSample(string key) => sampleService.TranslateAsync(keyword);

    protected async Task<List<SampleDto>> GetSample(Guid id) =>
        await DataContext.Sample
            .AsNoTracking()
            .Where(x => x.Id == id)
            .ToListAsync();

✅ DO prefer pure functions. Given the same input, the same data should always be retrieved. Avoid mutations or any other side effects in functions. The effect will be functions that are easy to reason about, easy to consume with no assumptions, and certainly easier to test as well.

Web APIs

✅ DO name controllers with a Controller suffix and use plural names for the RESTful APIs, e.g. UsersController instead of UserController.

✅ DO inject a cancellation token from the controller action and drill this down to pass to relevant calls that are cancellable. If an async call is cancelled no additional CPU/Memory needs to be used and pending processes can be cancelled.

✅ DO make use of the FromServices attribute to localize dependency injection in controllers. This is most useful when you have a bloated ctor with dependencies that are only used by all controller actions. This removes injection bloat from the ctor and makes it clear where dependencies are used.

// before

    private readonly IDependency1 _dependency1;
    private readonly IDependency2 _dependency2;

    public SampleContoroller(IDependency1 dependency1, IDependency2 dependency2)
    {
        _dependency1_ = dependency1;
        _dependency2_ = dependency2;
    }

    [HttpGet("one")]
    public Task<SampleDto> One([FromQuery] Request request)
    {
        return _dependency1.Sample(request);
    }

    [HttpGet("two")]
    public Task<SampleDto> Two([FromQuery] Request request)
    {
        return _dependency2.Sample(request);
    }

// after

    [HttpGet("one")]
    public Task<SampleDto> One([FromQuery] Request request,
        [FromServices] IDependency1 dependency1) =>
        dependency1.Sample(request);

    [HttpGet("two")]
    public Task<SampleDto> Two([FromQuery] Request request,
        [FromServices] IDependency2 dependency2) =>
        dependency2.Sample(request);

[HttpGet("sample")]
    public Task<SampleDto> GetSample([FromQuery] SampleRequest request,
        [FromServices] ISampleService service,
        CancellationToken cancellationToken = default) =>
        service.Sample(request);


// Sample service
public Task<SampleDto> GetSample([FromQuery] SampleRequest request,
           CancellationToken cancellationToken)
        {
            ....
            var result = await DataContext.Samples()
                .AsNoTracking()
                .ToListAsync(cancellationToken);
        }

ℹ️Know when you have passed a point of no cancellation to ensure atomic actions

✅ DO use the type route constraint for actions. Among many, the following examples show the benefit of endpoint resolution.


[ApiController]

[Route("api/[controller]")]

public class SampleController : ControllerBase

{

    [Route("{id}")]

    public string Sample1(int id)

    {

        ...

    }



    [Route("{name}")]

    public string Sample2(string name)

    {

        ...

    }

}

Calling this with api/sample/10 and api/sample/sample will both give the AmbiguousMatchException. This can be resolved as:

[ApiController]

[Route("api/[controller]")]

public class SampleController : ControllerBase

{

    [Route("{id:int}")]

    public string Sample1(int id)

    {

        ...

    }



    [Route("{name:string}")]

    public string Sample2(string name)

    {

        ...

    }

}

⚠ Route constraints have features that can make them tempting to use for input validation such as ranges and regex. Don’t use constraints for input validation. If constraints are used for input validation, invalid input results in a 404 Not Found response. Invalid input should produce a 400 Bad Request with an appropriate error message. Route constraints are used to disambiguate similar routes, not to validate the inputs for a particular route. When creating params from the URL, required is a good exception to this rule as this should be useful in disambiguating routes.

Code Analysis

⛔ DO NOT suppress Roslyn analyzer rules, these are in place for a reason. Do consult other developers before suppressing rules, whether for specific code lines, files, or the entire solution to ensure that any changes that are related to standards have a team consensus.

GOTCHAS

✨ Apply caution when changing code surrounded by debugger directives. Ideally building with the debug or release config should not make a difference to the build’s succeeding. However with preprocessor directives if you change code that is only available for release, for instance, it could end up broken and you would not pick it up on your dev machine as you are building in debug. This would of course blow up later when you are making release builds. Be mindful of this scenario.

⚠ Historically Visual Studio has not been particularly smart with these scenarios, it could suggest for instance that you remove an unused using that is used in a preprocessor directive for a mode you are currently not running in.

✨ Apply caution with CopyToOutputDirectory option Always as this will break incremental builds and cause given projects to always build even when they have not been updated. Do make use of the option PreserveNewest. In addition, this option means less IO, particularly for large files which also speeds up builds.

ℹ️ Such cause is not obvious and an investigation must be conducted to find the glitch. To investigate, in the Visual Studio > Options > Projects and Solutions > SDK-Style Projects, set the Up to Date Checks value from None to Minimal. Now in the Output window, you will see logs that indicate when incremental builds are not working as expected and why

How you do something, is how you do everything

Wed, 24 Aug 2022 18:00:00 GMT

Photo by Àlex Rodriguez on Unsplash

How you do something is how you do everything

How often do we as individuals or collectively in groups put off certain things as inconsequential, and rush to deliver something that is “good enough”? How often do we put off a small-time investment to polish up on anything because it is not that important? This is a typical mindset and one that may not be entirely conscious if it has been done long enough to become a habit. Some examples:

That email that is sent out with grammar issues that creates a negative impression of you and the team you represent.
The lack of artifacts and documentation in a team for things like onboarding and offboarding of new colleagues or documentation of processes and practices.
The tardiness to the meeting that is only a few minutes, and no one should notice, right?
The hastiness when delivering feedback that leaves out the key details.
That one thing you keep telling yourself you will get to but never do.

Often, we put off doing things properly as either being insignificant or demanding too much time that we do not have yet. A lot of times we end up having to revisit that same thing again and again due to not getting it right the first time.

Over the years I have noticed that teams or individuals that don’t mind dropping the ball in one area believing it is not all that significant all tend to have the following traits:

Lack of unanimity on what areas are okay to drop the ball on and as such eventually things spiral out of control as everyone makes their own decisions on what to mediocre in.
Communication is not great in these teams, and it usually only takes one or two individuals to not do it well and be allowed to, for the whole team to struggle in this area.
Tasks that can be done right once and replicated with ease are instead rushed and then repeated haphazardly and without consistency.
Lack of consistency in delivery. A tell-tale sign is when you can look at what is delivered and easily tell which team member worked on what.
An ever-growing list of things that get put off that no one ever gets to.

A key observation for me has been that the situation increasingly gets worse over time and the same attitude starts to get applied in other areas that are not too apparent but have consequences regardless. The broken windows theory explains why things we put off ultimately spiral out of control by stating that any observable signs of disorder such as broken windows form an environment that encourages even more disorder. Consequently, by ignoring the “little” things you soon forfeit control of much more to this mindset. After all, if you allow your hedge to overgrow, would you notice when your lawn requires a trim?

So how do we manage this? Well, we tackle this much like Mayor Giuliani tackled the severe issues New York City was facing before 1993. Let’s glance at some statistics on what he accomplished first (source: The American Presidency Project)

In 1993, there were 11,545 major crimes per week; By 2001, that number dropped to 5,072.
Between 1993 and 2001, New York City experienced a 66% decline in murders.
During the same period, there was a 72% decline in shootings.
In addition to the decline in murders, New York saw a 45.7% decline in rapes, a 67.2% decline in robberies, a 39.6% decline in aggravated assault, a 68.2% decline in burglary, a 43% decline in larceny, and a 73.3% decline in motor vehicle theft.

He accomplished all this by “repairing all the broken windows”; tackling vandalism, loitering, public drinking, jaywalking, transport fare evasion, and more to create an environment that discouraged crime and disorder. Once the environment looked and felt good the residents treated it with regard and pride.

Similarly, this is how we can guarantee that we remain committed to delivering at the highest standard across the board. Once the quality is baked into our minds and there is no tolerance for dropping the ball you will notice consistency, a sort of synergy even within teams as more projects and challenges are taken on. These are a few heuristics to evaluate when this is going well:

Communication is healthy. We engage professionally and expressively cultivating strong relations. We return every phone call and reply to every message and email. We express ourselves with empathy and respect.
Delivery is done once and well. We do not scurry through delivery only to have to rework things at the expense of the next commitment.
Processes are regularly reviewed and evaluated to ensure that no broken windows are ignored. This is key to preserving this mindset at least long enough to make it a habit.
Doing things well does not have a big impact on the effort or time required to deliver, especially when repeated.

“How you do anything, is how you do everything” has become a favoured quote of mine, and really by contemplating on this often and acting I feel it has served me well. An early experience that helped me to always aspire to do all things well was participating in athletics back in my high school days. I aimed to improve my 100m sprint time and what I noticed in my attempts was that I could not “just” improve it. To achieve that I had to improve everything it took to do the dash, even if by a teeny bit. Surprisingly that meant a whole lot of things would require attention like how well my start was, my stamina, rest, and diet, and many more had to be improved. Interestingly some of the things I had initially put off as being insignificant turned out to have the greatest impact and the success in addressing those created a habit for me to pursue the other changes with the right attitude. Soon I was applying this mindset to my academics, relationships, parenting, and work with similar success.

Contemplate this and see what you can apply to yourself or your team. Here are a few questions that may guide you in that process:

Are you applying discipline and structure in all you do?
When you put off things are you being critical and objective about why or are you seeking a reason to justify the action?
If you continue to work in the manner and pace, you do now how will things look a year from now?
Are you putting off meetings and events because you are too busy?
Imagine a backlog of anything you must get done as a trashcan. Do you put off taking five minutes to empty the trashcan and ultimately must deal with a greater situation once the trash starts to spill over?

If you don’t do the “unimportant stuff” well, do not be surprised if you cease doing anything well at all. Once you tolerate mediocracy in one area you have simply tolerated it in all your doings, period. It is all about doing the little things well and consistently. Apply this in your work, personal life, and everything else in between to unlock great success and satisfaction. Of course, keep an eye on doing things well by avoiding cutting corners without necessarily aiming for perfectionism as that can easily burn you out.

In closing recognize this, repetition makes habits!

AWS Best Practices

Thu, 30 Jun 2022 10:00:00 GMT

Photo by Jon Tyson on Unsplash

AWS Best Practices

General

✅ DO come up with a complete and documented disaster recovery process for all the services and workloads you utilize in AWS. Regularly review and run through the process to ensure that it stays up to date and that the team has the skills to execute it. Have a clear definition of the time to recovery and point of recovery for your workloads.

IAM

✅ DO use access keys with restricted access to only what is required for a development setup and ensure that these are not linked to a user with AWS console access.

⚠️ The access keys used by developers are the ones at most risk as multiple people will have these on their machines, therefore, increasing the surface area of compromise.

✅ DO isolate resources by environment so that generated keys access only have access to the environment it is used in. For example, development keys should not be able to access production stacks.

⛔ DO NOT set up any AWS accounts with console access that do not have MFA enabled.

⛔ DO NOT login with a root account for day-to-day operations. IAM users/roles with specific permissions to the operation should be used instead. Root accounts should only be used for operations that cannot be done with any other account.

⛔ DO NOT login with the root user account without authorization. Only one team member should have the authorization to make use of the root account.

⚠️ While this is best practice the root user credentials should be shared with at least one other person should they be required in the absence of the primary.

ECR

✅ DO ensure that repositories are not publicly exposed.

ℹ️ There are exceptions. For example, a requirement to set up base build images for reuse would add unnecessary complexity to authenticate on each build. Given the is no company IP in creating a public build image that would be understandable.

✅ DO enable scan on push at repository level to automatically scan for vulnerabilities on any images pushed to a repository.

✅ DO consider a life cycle policy for cost optimization.

⚠️ This may not be ideal. The benefit of being able to get the artifacts by release number even months if not years from now may outweigh any cost implications of storing images so make this decision very carefully.

⚠️ Unfortunately the current implementation of the policies do not work well with semantic versioning making it difficult to automate a cleanup. There is an open request to enable regular expression to facilitate this.

✅ DO use semantic versioning for all images pushed to ECR and consistent naming.

✅ DO enable cross-region replication of ECR images.

✅ DO create and push only minimal images by removing all extraneous binaries or dependencies and do not use unfamiliar base images.

✅ DO create and push only minimal images by using multi-stage builds.

✅ DO regularly update the packages in container images. Consider scanning tools like Snyk to stay on top of this.

✅ DO scan and lint all Dockerfiles. Consider hadolint.

✅ DO ensure that all repositories have the immutability tag enabled to force tag updates on each push and restrict overwriting prior images.

⚠️ For reproducibility images must be immutable and for security purposes, this thwarts the possibility for attackers to push malicious versions by taking advantage of a tag that is already being pulled.

EC2

✅ DO ensure that instances are launched off of the list of team-approved AMIs.

✅ DO ensure that instances that have a burstable instance type have the credit specification set to enable unlimited mode.

ℹ️ If the average CPU utilization over 24 hours exceeds the baseline, you incur charges for surplus credits. Which is likely okay and desired for critical workloads.

✅ DO ensure that instances have termination protection enabled, which ensures that someone does not accidentally terminate an instance, an action that cannot be undone.

ℹ️ There are exceptions to this rule. Instances that are created from spot requests cannot have this setting enabled and fully managed instances such as those created by EKS for a k8s cluster should not have this setting enabled.

✅ DO CONSIDER spot instances for non-critical workloads such as running builds, pull requests, etc.

✅ DO make use of the bastion/jump server pattern for access to servers that do hold production workloads or client data.

✅ DO make sure a bastion/jump server allows RDP/SSH access to traffic only from specific whitelisted IP addresses.

✅ DO provide descriptive names to instances and tag them accordingly to make it easy to tell what each instance is for.

✅ DO make use of elastic IPs for any instances whose public/private IP address is required.

✅ DO monitor and respond to events of your instances such as CPU usage, disk space usage, etc.

✅ DO regularly review and test the EC2 disaster recovery process for instances and volumes.

✅ DO utilize and configure security groups wisely as the firewall to instances, providing only required access. Implement the least permissive rules for your security groups.

✅ DO make use of VPCs to logically isolate instances from the rest of the AWS cloud.

✅ DO create any key pairs using one of the two supported formats consistently, otherwise, keys can be cross-converted between PEM and PPK using tools like putty-gen.

✅ DO make use of the most specific AMI for a workload. AMIs with the least dependencies and features that come packaged/enabled are better as these would have a smaller attack surface area therefore always choose one that provides only what is required.

ℹ️ An example is the BottleRocket AMI over the AWS Linux 2 AMI for use in a Linux-based k8s cluster. BottleRocket is optimized for running docker and comes with very minimal for that.

✅ DO clean up unused instances by shutting them down or terminating them so as not to incur charges. Terminated instances cannot be resurrected.

✅ DO ensure that the account billing settings have the option to receive AWS Free Tier usage alerts always enabled.

These AWS Free Tier usage alerts allow AWS to notify you when you’re exceeding 85 per cent of your usage for the month. This of course can generate noise for non-expiring AWS free tier offerings.

⚠️ You must avoid browsing the internet on an instance by all means. If you must download something from the internet rather download it on your machine and copy it over. While the security group will act as a firewall for incoming traffic there is a caveat, if the incoming traffic is in response to a request it will come through, which is why one must avoid accessing the internet from any instances to avoid exposing the instance to attackers.

✅ DO regularly patch, update, and secure the operating system and applications on your instance.

✅ DO separate Amazon EBS volumes for the Operating System vs data and further group data on multiple volumes if possible i.e. database backups vs database logs.

⛔ DO NOT have any instances without encrypted EBS volumes or snapshots.

⛔ DO NOT create any snapshots or AMIs that are in a public catalogue.

⛔ DO NOT create security groups that open any traffic to the internet, including for developer access such as SSH and RDP. Any required access for servers that do not hold production workloads or client data should be via the session manager when possible.

⛔ DO NOT mishandle or lose access to access keys used to create instances. These should be stored securely as they are required to access the instances.

⛔ DO NOT disable EC2 instance metadata as this is required by various AWS services including Systems Manager for automated scanning and patching of EC2 instances. This means the http-endpoint flag should always be set to enabled. To enforce security, however, set the http-tokens option to required which enforces sending a signed token header with any instance metadata retrieval request and hence requires an IAM role which is harder to exploit. Additionally the http-put-response-hop-limit is also set to 1.

⛔ DO NOT make use of Internet Explorer or Chrome on EC2 instances as they are not very secure. Firefox should be the browser used on all instances.

✅ DO create a process for security hardening of an instance such as uninstalling unnecessary programs.

✅ DO make sure scaled/duplicated instances are not in the same Availability Zones as this would not offer much resilience when an entire zone is down.

✅ DO make use of the session manager over RDP or SSH access to instances. This avoids using and exposing an SSH key or having to open ports or a bastion host and makes use of AWS KMS for secure sessions.

Naming

✅ DO employ complete names but with standard acronyms and in all lower cases snake case:

| Entity | Prefix |

| -------------- | ----------------------- |

| Security Group | security-group-[region] |

| EC2 | ec2-instance-[region] |

Only include the region (i.e eu-west-1) for resources that can be created in multiple regions

TypeScript and React standards

Fri, 17 Sep 2021 10:00:00 GMT

Photo by Scott Rodgerson on Unsplash

I thought it might be worthwhile to share my very opinionated take on how best to work with TypeScript and React.

React Standards and Best Practice

✅ DO consider lint staged with your linting setup (npx lint-staged --verbose) to lint currently staged files. This is a more manageable way to ensure that you are not introducing lint violations. This is particularly useful when linting is added later and you have some cleanups to do as this will avoid you being overwhelmed with lint information about the whole project, only the files you have touched.

ℹ lint-staged with --verbose flag will run linting only on git staged files showing you both warning and errors.

✅ DO run eslint with the cache flag --cache enabled which will only lint files changed since the last run. This may have a big impact on developer setups. Continuous integration setups will benefit if you do not use clean build agents for each run which can greatly reduce pipeline run times.

✅ DO prefer the use of interfaces over type aliases. Use type when you need specific features offered by types. The key driver is that interfaces can always be extendable while types cannot. Because interfaces map more closely to how objects work these are a good default. However, if you cannot express a type with an interface and need to use union or tuple types then type aliases would be a better choice.

✅ DO check for console runtime errors and warnings as part of your manual testing before committing to a task, there could be useful things that show up there that are easily overlooked and often easier to act when you notice them when issues are introduced.

✅ DO use const and let over var, using let only where a mutation is expected. The scoping semantics of var leads to bugs.

ℹ Utilize linting to enforce low-hanging fruit standards like this.

ℹ️ In many cases you can avoid mutable variables altogether, hence let, by thinking in a more functional way when writing code.

✅ DO use !x instead of x==null. This avoids the confusion with x===null, the former is intentionally meant to catch both null and undefined but that situation is not exactly idiomatic and can accidentally get refactored or written incorrectly in the first place as the latter.

if (!x) {
  //'x' is falsey, so it will evaluate to false when undefined, null, or zero (0)
}

✅ DO favor the use of modern Javascript features over the more classic counterparts. Examples of features to be familiar with:

Template literals (string interpolation)
Rest and spread operators
Destructuring assignments
Object literals
Arrow functions
Yield
Ternary operator
Nullish coalescing operator
Promises
async and await

✅ DO familiarize yourself with the following array methods and avoid custom implementations or 3rd party libraries for these scenarios.

find
some
every
includes
map
filter
reduce

ℹ Favor functional map, filter, reduce, forEach, some etc over collection mutation

✅ DO favor the use of absolute imports over relative imports.

// given src/services/translate.ts

// an import from /src/../something.ts
import { translate } from '../../services/translate';

// changes to this regardless of where imported from
import { translate } from 'services/translate';

Can be achieved by adding the following to tsconfig.json

 "compilerOptions": {
  "baseUrl": "src"
}

If we want to tell local imports apart from the rest we can use aliasing

 "compilerOptions": {
    "baseUrl": "./",
    "paths": {
    "~component/*": ["src/components/_"],
     }
}

// import {Login} from ‘~component/Login;

✅ DO avoid direct mutations on arrays like calling sort. Instead, make copies with the spread operator or use slicing before calling these mutations.

✅ DO Favor arrow functions for consistent lexical scoping (no mix of lexical and dynamic scoping).

ℹ Apply caution with arrow functions on render with React, each render will see a new function and this can lead to unnecessary child renders.

See here for more detailed comparison between arrow functions and traditional functions.

✅ DO favor Sets for array operations like querying, uniqueness, etc.

✅ DO favor await of async methods over the traditional then and catch.

promise.then(result => console.log(result)).catch(result => console.log(result));

// looks closer to how we are doing it in C# as well
// less nesting -> easier to read
try {
  const result = await promise;
  console.log(result);
} catch (e) {
  console.log(e);
}

✅ DO use the object spread syntax over Object.assign to shallow-copy objects.

ℹ This can be enforced with eslint prefer-object-spread rule.

✅ DO familiarize yourself with the collection entries method and its use with destructuring.

var data = [1, 2, 3];

for (const [index, element] of data.entries())
    ...

✅ DO use double quotes for strings.

⛔ DO NOT use arrow functions for class methods or object literal (this will be scoped to the caller).

⛔ DO NOT use objects as maps. Make use of the JavaScript Map object when a dictionary-like type is required.

ℹ See here for a comparison of maps to objects.

⛔ DO NOT abuse optional chaining (Elvis Operator). Use it to show intent (that the object is nullable) and not as a safety net for cases where something would not be null.

// should be intentional
// falsy used gives impression the whole chain from the user can be nullable
// the typing of what is being operated on should reflect this as well
const streetName = user?.address?.street?.name;

⛔ DO NOT import all from a given module.

// BAD
import *

⛔ DO NOT make use of as unless absolutely required. Type assertions with as in TypeScript remove static type checking which can lead to issues.

interface User {
  name: string;
}

// BAD
// adding a new property on User for instance would not cause an error here
const getUser = () => {
  return {
    name: 'John Doe',
  } as User;
};

// GOOD
const getUser = (): User => {
  return {
    name: 'John Doe',
  };
};

⛔ DO NOT use Partial<T> unless absolutely required. This would be similar to abusing the as keyword to map types.

ℹ as should hardly be used unless ‘type inference’ is indeed intended for instance if you receive something from a 3rd party lib such as a value on an event and you know the type and want TS to infer that then such would arguably be a valid case for as but can still actually be avoided.

⛔ DO NOT favor the use of indexes when working with arrays if avoidable. Using an example with Axios responses, while type inference can help pinpoint when the wrong index is used, there are a lot of edge cases where this would not help out.

useHook(() => {
  promise.then(response => {
    let profile = response[0].data;
    let user = response[1];
  });
}, []);

// BETTER
useHook(() => {
  promise.then(response => {
    const [{ data: profile }, user] = response;
  });
}, []);

Feature best practices

✅ DO consider localizing all user-visible strings. If you go with localization then this should be done consistently.

✅ DO consider adding dirty checks to UIs that require explicit saving and provide user prompts on navigating away with changes.

Components

✅ DO create small composable components, much like lego pieces that plug together, over large monolithic components. This allows for easier to reason about components that are easy to reuse and test. This can also make responsibility boundaries very clear and in particular help isolate the state. This can also lead to more readable JSX.


// what does this do?
<div className={styles.userPanel}>
    <div>
        <div className={styles.detailsRow>
            <span className={styles.detailsColumnLeft}></span>
            </span>
        </div>
        <div className={styles.detailsRow}>
            <span className={styles.detailsColumnLeft}></span>
            <span className={styles.detailsColumnRight}></span>
            <span className={styles.detailsColumnRight}></span>
        </div>
    </div>
</div>
// contents above were removed for brevity, but imagine the mess!

// same component with composition
<ActivityPanel>
    <DetailsRow>
        <Name />
        <Completion />
    </DetailsRow>
    <DetailsRow>
        <DueDate />
        <CriticalPath />}
        <Buffer />}
    </DetailsRow>
</ActivityPanel>

✅ DO create safe defaults to state. This can make for cleaner code as opposed to say handling undefined in multiple areas. One less obvious problem when working with side effects such as network calls is that you may have your calls returning fast a majority of the time but could have edge cases were the component renders faster than the side effect and it may have unhandled cases of undefined.

// FROM
const [state, setState] = useState<Type[]>();

// TO
const [state, setState] = useState<Type[]>([]);

✅ DO identify generic parts of a component upfront that can be reused and code these generically enough to be shared.

✅ DO constrain what a user can enter in the UI to what the rest of the system can accept. For example, do not allow input that is larger than the database constraint, file uploads are larger than allowed in the backend. Validating such cases and showing specific and clean messages is a good UX practice.

✅ DO make use of TypeScript generics to expose typed props accordingly when you wrap around a component or accept a render component.

export interface SampleComponentProps<T> {
  sampleProp: ComponentType<T> | ElementType;
}

export const SampleComponent = <T extends {}>(props: SampleComponentProps<T>) => {};

// based on usage, will infer the type of T and expose component props
<SampleComponent smampleProp={Input} />;

ℹ The same principle should be applied to components that accept multiple data types. The inference of the type will add typing in places such as event handlers.

✅ DO ensure that the component file name is the same as the default export component.

✅ DO make use of CSS modules. Keep this consistent and in particular, do not make use of inline styles.

✅ DO favor functional components. They have a simpler syntax, no confusing lifecycle methods, constructors, or boilerplate and due to being succinct are more readable.

✅ DO favor multiple useEffect hook sections to make code more readable, over a single useEffect hook with multiple responsibilities.

ℹ A good heuristic for this refactoring is when you have dependencies that do not apply to everything in the useEffect, for instance, you could have a dependency on an id changing but in addition to fetching an entity with that id you fetch data that would not have changed or be dependant on the id in the same hook. This is also applicable with custom hooks that wrap around useEffect,

✅ DO consider virtualizing long lists. Unvirtualized long lists can make for a very horrible user experience with performance in many cases.

✅ DO destructure props as consts at the top of the component to guard against accidental mutations.

export const SampleComponent: FunctionComponent<SampleComponentProps> = memo((props) => {
    const { a, b, c, d } = props;
        ...
}

ℹ️ An alternative is to create read-only props. This can prove challenging when considering nested properties and one would need to be careful about collection types used as well.

ℹ️ In many scenarios knowing that something came from a prop vs say the state should not be a detail that should be exposed in the usage as many will justify props.x communicates this.

✅ DO mark state type properties as read-only accordingly to avoid direct mutations.

interface ComponentState {
  userIds: number[];
  name: string;
  owner: User;
}

interface User {
  name;
}

export const Component: FunctionComponent<{}> = () => {
  const [state, setState] = useState<ComponentState>();

  // these mutations are valid
  state.userIds = [];
  state.userIds.push(1);
  state.name = '';
  state.owner = { name: '' };
  state.owner.name = '';

  return null;
};

interface ComponentState {
  readonly userIds: ReadonlyArray<number>;
  readonly name: string;
  readonly owner: Readonly<User>;
}

interface User {
  name;
}

export const Component: FunctionComponent<{}> = () => {
  const [state, setState] = useState<ComponentState>();

  // these mutations are not valid
  state.userIds = [];
  state.userIds.push(1);
  state.name = '';
  state.owner = { name: '' };
  state.owner.name = '';

  return null;
};

✅ DO use dynamic properties to make mutation logic simpler.

export const Component: FunctionComponent<{}> = () => {
  const [state, setState] = useState<ComponentState>();

  setState(prevState => ({
    ...prevState,
    [field]: value,
  }));
};

✅ DO ensure that element keys are always stable, predictable, and unique. Unstable keys (like those produced by Math.random() or use of array indexes) will cause many component instances and DOM nodes to be unnecessarily recreated, which can cause performance degradation and lost state in child components. See example.

✅ DO apply care when writing conditional renders on variables, use of falsie numbers will be rendered i.e. value && <Component> given 0 for value will render 0. This however works well with bools, null and undefined.

// BAD
names.length && <Users names={names}>

// ALTERNATIVE
names.length > 0 && <Users names={names}>

// ALTERNATIVE
Boolean(names.length) && <Users names={names}>

✅ DO expose render component props from the wrapper when implementing the render props pattern. Best not to pick and expose as custom props upfront, usually exposing all makes for a flexible and un-opinionated render props wrapper.

✅ DO be mindful of the number of components that have state. If a value is not required for render it can be tracked without using state. Given a parent -> child hierarchy it may be possible to only track state at the parent.

✅ DO use correct import syntax (import ‘x’) if importing only for side effects.

import '@testing-library/jest-dom/extend-expect';

✅ DO collocate things as close as possible to where they are being used.

ℹ Keep components, functions, styles, state, etc. as close as possible to the component where it’s being used. This will not only make your codebase more readable and easier to understand but it will also improve your application performance since it will reduce redundant re-renders on state updates.

✅ DO consider use of [react-use](streamich/react-use: React Hooks — 👍 (github.com)) over creating your own custom hooks where applicable.

✅ DO move any logic that is not React specific and does not need to be reinitialized on rerender out of function component. Keep in mind that the function component body is recalled on each render, calling that logic over and over unnecessarily.

⛔ DO NOT make all fields optional or add coalescing operators on something that is not nullable as a way to go around a state that can initially be null. Instead, make this explicit.

ℹ By making this explicit anyone who updates this code later will know to handle null in the render.

⛔ DO NOT add to state or props data than can be derived/calculated from elsewhere.

⛔ DO NOT break HTML semantics with unnecessary elements like divs that do not carry the semantic meaning of divs for other reasons like a container in which case Fragment would have been better. Abuse of HTML tags can also break a11y and can result in an unnecessarily messy DOM.

⛔ DO NOT handle navigation with click event handling unless this is absolutely desired. The first prize is to make use of HTML semantics that the browsers have support for like hyperlinks/anchor tags. This will preserve expected browser behavior like center mouse click or right click open new tab giving the user more control, mouse icon change other current or future features.

⛔ DO NOT use React fully qualified name for types, i.e. React.Fragment, React.FunctionComponent. This is not necessary and in particular, is not used consistently, i.e. you may not find React.useState or React.useEffect in the same code base that uses the latter.

const [user, setUser] = React.useState<User>({});

//instead
const [user, setUser] = useState<User>({});

⛔ DO NOT work around about state that can null in some cases, be explicit about. It will be easier for anyone using that state to know to handle possible nulls.

// BAD
const [user, setUser] = useState<User>({});

// GOOD
const [user, setUser] = useState<User | null>({});

⛔ DO AVOID customizing 3rd party components via CSS. Instead, the customization should be done via props where applicable. Such customizations are not upgrade-friendly.

⛔ DO AVOID uncontrolled components, that is those that access the DOM directly i.e. refs unless absolutely necessary.

⛔ DO AVOID the urge to replace JSX with call-to-custom functions that return JSX, this should be done sparingly and with care. If this is desired then the custom function should be a component.

⚠ Note that splitting component renders with functions can be good but largely an anti-pattern that can hide the fact that a component has gotten too large or has multiple concerns. This approach should be highly avoided in favor of a declarative code. Certain React-specific heuristics are not applied when you use render functions, i.e. if you have a map and call a function React cannot and will not warn you about the need for a key and you can get away with passing unstable values like indexes to use as a key and again React cannot and will not warn you. Another really nasty thing that happens with such functions is that if you are using react dev tools, they will not show up.

⛔ DO NOT have constants scattered all over a component file. Neatly place all constants at the top of the component.

✅ DO create an order to how things flow in a component and stick to it consistently. For example, you could have the order state, hooks, event handlers, functions, and render body. Applying the same order in all your components makes it quick and easy to navigate around.

Hooks

✅ DO follow the rules of hooks:

Only Call Hooks at the Top Level

🛑 Don’t call Hooks inside loops, conditions, or nested functions.
Only Call Hooks from React Functions

🛑 Don’t call Hooks from regular JavaScript functions. Instead, you can call Hooks from React function components or from custom Hooks

ℹ These rules can and should be enforced with linting.

✅ DO name hooks with use prefix i.e. `useRestoreHook.

✅ DO consider creating custom hooks to encapsulate custom logic that is/can be duplicated in multiple places where applicable i.e. useYourImagination

⛔ DO NOT have hooks scattered all over a component file. Neatly place these at the top of the component.

ℹ When reading through a component you will get the most out of immediately knowing its side effects and what it renders. Seeing hooks at the top and scrolling all the way down to see the render return makes this very easy. When hooks are scattered in between other methods like event handler that becomes harder.

⛔ DO NOT use complex objects as dependencies on the hooks

You can read more on it here

Jest and React Testing Library

✅ DO group related tests under a describe if you have multiple tests in one file.

✅ DO locate the tests very close to the file being tested. For example, you could have the _tests_ folder in each subfolder with something to test.

✅ DO name test files with suffix tests i.e. userService.test.ts, breadCrumbs.test.tsx

✅ DO favor data-driven tests with test.each over duplicated tests.

test.each([
  [new Date(2000, 11, 12), false],
  [new Date(1970, 11, 12), false],
  [new Date(1969, 11, 12), true],
  [new Date(1950, 11, 12), true],
])('isDateBefore_1970 returns relevant boolean', (date: Date, expected: boolean) => {
  expect(isDateBefore_1970(date)).toBe(expected);
});

✅ DO favor naming tests in the format given then should or given when then should. Long test names are perfectly fine and in particular, the ability to use any characters including spaces in jest test names makes this very flexible.

✅ DO make it easy to tell apart the Arrange, Act, and Assert sections of your test and in particular to clearly tell what the system under test (sut) is.

✅ DO NOT have magic values in tests. Simple things like inlining variables to have a name can go a long way in test readability.

// BAD - why is this value of significance?
expect(result.count).toBe(24);

✅ DO use the most specific assertions, these give the most specific and useful failure messages as well. If there is an assertion call for it, avoid the alternative of a more generic assertion with added logic in it.

⛔ DO NOT suppress linting rules, these are in place for a reason. Do consult other developers before suppressing rules, whether for specific code lines, files, or entire workspace

⛔ DO NOT write tests with multiple responsibilities, instead a test should have a single focus. Multiple unrelated assertions are a red flag.

⛔ DO NOT use ReactTestUtils. Use React Testing Library which is designed to enable and encourage writing tests that use your components as the end users would. ReactTestUtils has features like mocking elements, which do not simulate how a user would use the components and encourages very bad and brittle testing patterns.

GOTCHAS

Jest

✨ When setting up a mock with jest.mock, ideally you may want to set up one of the properties based on a variable so that you can use the same value in your assertions. Jest will however fail the test complaining that mocks cannot access outside variables. This is in place to avoid dirty mocks, however, you can bypass this by simply naming the variables with the mock suffix i.e. mockUser

const mockUnknown = '#9E9E9E';

jest.mock('../../redux/store', () => ({
  getState: (): RecursivePartial<RootState> => ({
    app: {
      statusColors: {
        unknown: mockUnknown,
      },
    },
  }),
}));

Form validation

✅ DO make use of yup schemas for validation and mutations related to forms.

✅ DO move the yup schema outside a component unless it requires access to something in a component, like a state (which can be avoided with a different design). This avoids a new schema being created on each render. Given the inner workings of yup on each mutation the schema is cloned, so an object schema composed of multiple property schemas is cloned for each, not ideal.

// from this
const formSchema = object<User>({
  name: string().trim().required(customMessage()),
});

// to this
const formSchema = object<User>({
  name: string()
    .trim()
    .required(() => customMessage()),
});

⛔ DO not use the shape unless it is desired, ideally for merging multiple schemas together. This affects the Typescript type coercion and will fail to pick up mismatches in the schema to the target type.

// from this
const formSchema = object<User>().shape({
    name: string()),
});

// to this
const formSchema = object<User>({
    name: string()),
});

Debugging all jest test states

Thu, 01 Jul 2021 10:00:00 GMT

Photo by Nina Mercado on Unsplash

Debugging Jest tests in Visual Studio Code can greatly enhance your JavaScript testing experience. While I am an avid user of the vscode-jest extension, which is incredibly lightweight and encompasses all the primary features needed for efficient testing, there’s one aspect I felt was missing: the ability to debug all tests, regardless of their state.

Fortunately, customizing VS Code to enable debugging for all Jest tests, including those that pass, fail, or are in an unknown state, is quite straightforward. You simply need to modify the settings.json file in the .vscode directory of your project. By adding or updating this configuration file with the following snippet, you unlock the ability to debug every test

{
  "jest.debugCodeLens.showWhenTestStateIn": ["fail", "pass", "unknown"]
}

With this simple addition, you can now trigger the debugger for tests in any state. This flexibility is particularly useful for thoroughly examining test executions, ensuring that your JavaScript code behaves as expected under various scenarios. It’s a small tweak, but it significantly improves the debugging capabilities within VS Code, making the process of testing with Jest more efficient and comprehensive. Sweet!

Killing node processes

Wed, 30 Jun 2021 10:00:00 GMT

Photo by Gunnar Ridderström on Unsplash

Visual Studio has this annoying tendency to leave zombie node processes after you have run a node app from it. This can be quite expensive and leave your machine very laggy as they accumulate from multiple runs. You may also run into this situation for different reason.

There are two ways to kill these zombie processes

Kill all node processes running
Target specific node processes

To kill all node processes run the command

taskkill /f /im node.exe

This has proven to be a bit problematic for me as I sometime have node processes that I want to keep alive such as those from running VS Code and extensions or other apps. I therefore prefer to target specific node apps based on the command that is used to spawn them. For instance if using react-app-rewired start to run the app I would target that by running the following script.

Get-WmiObject Win32_Process | Where-Object {$_.CommandLine -like '*react-app-rewired*start*' -and $_.Name -eq 'node.exe' } | ForEach-Object {$_.terminate(0)}

VS Code workspace recommended extensions

Wed, 30 Jun 2021 10:00:00 GMT

Photo by Barn Images on Unsplash

Visual Studio Code has support for specifying extension recommendations as part of you workspace. Developers can then see these in the extensions tab and decide what to install.

Here is how to set it up

This creates a file named extensions.json under .vscode in your workspace root which looks something like this.

{
  "recommendations": ["editorconfig.editorconfig"]
}

Generalize .net test dependencies

Tue, 29 Jun 2021 10:00:00 GMT

Photo by Shubham Dhage on Unsplash

Generalize .net test dependencies with Directory.Build.props

In the .NET ecosystem, Directory.Build.props is a powerful feature that allows developers to apply common settings across multiple project files. It acts as a central place to define properties and item groups that are automatically imported into every project in the directory tree. This means you can specify a single version of a NuGet package or a set of common properties in one file, and these settings will propagate to all relevant projects. This approach not only streamlines the configuration process but also ensures consistency and reduces the likelihood of errors in large solutions.

The Directory.Build.props file offers a lot of interesting features for targeted projects, solution folders, or the entire solution. Let’s focus on how we can leverage this to generalize test dependencies and any required configuration in one place.

Test project dependencies tend to be mostly the same across multiple projects and annoyingly can be quite easy to point to different versions and require consolidating. I use the Directory.Build.props at the tests folder to target all test projects, in my case by the suffix .Tests in the name, and apply the generic dependencies in one place.

Utilizing Directory.Build.props offers several key benefits:

Reduced Redundancy: By centralizing common configurations, it eliminates the need to duplicate settings across multiple project files.
Ease of Updates: Updating a single Directory.Build.props file propagates changes to all projects, simplifying maintenance and version upgrades.
Consistency Across Projects: Ensures uniformity in settings and dependencies, making the codebase more cohesive and manageable.”

Example

<Project>
    <PropertyGroup Condition="$(MSBuildProjectName.EndsWith('.Tests'))">
        <IsPackable>False</IsPackable>
    </PropertyGroup>

    <ItemGroup Condition="$(MSBuildProjectName.EndsWith('.Tests'))">
        <PackageReference Include="Autofac.Extras.Moq" Version="6.1.0" />
        <PackageReference Include="FluentAssertions" Version="6.7.0" />
        <PackageReference Include="FluentAssertions.Analyzers" Version="0.17.2" />
        <PackageReference Include="XUnitToFluentAssertionsAnalyzer" Version="1.0.2">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
        </PackageReference>
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
        <PackageReference Include="xunit" Version="2.4.1" />
        <PackageReference Include="xunit.runner.console" Version="2.4.1">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
        </PackageReference>

        <PackageReference Include="xunit.runner.visualstudio" Version="2.4.5">
            <PrivateAssets>all</PrivateAssets>
            <IncludeAssets>runtime; build; native; contentfiles; analyzers</IncludeAssets>
        </PackageReference>

        <PackageReference Include="AutoFixture" Version="4.17.0" />

        <PackageReference Include="coverlet.collector" Version="3.1.2">
            <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
            <PrivateAssets>all</PrivateAssets>
        </PackageReference>
    </ItemGroup>

    <ItemGroup Condition="$(MSBuildProjectName.EndsWith('.Tests'))">
        <ProjectReference Include="..\Sample.Tests.Common\Sample.Tests.Common.csproj" />
    </ItemGroup>
</Project>

In this example, we have all test projects marked as not being packable, setup of required package references and all tests have a project reference to a common test project that could contain shared things like fixtures, builder, etc. We no longer need to have these in each project and can control what goes in all the tests in one place.

While this was specifically about tests, there are many other use cases. For example, we could set things like copyright and assembly info in one place by targeting all projects, indicating the C# language version at the solution level, etc. Cases like these are often the same for all projects but over time people forget to update some projects or it just becomes tedious to keep up.

Pitfalls

One common pitfall is inadvertently affecting projects with unwanted configurations. To avoid this, use conditional properties in your Directory.Build.props file to selectively apply settings. Another issue is the misplacement of the file; ensure it’s correctly located at the solution root or appropriate directory level. Also, be mindful of the order of properties and their overriding behavior, as this can lead to unexpected configurations.

Conclusion

In conclusion, Directory.Build.props is a highly effective tool in the .NET developer’s arsenal for managing project dependencies and configurations. By centralizing common settings, it not only simplifies project management but also enhances the maintainability and consistency of your codebase. As demonstrated, its benefits extend beyond test projects, offering a streamlined approach to handle various aspects of .NET development. Adopting Directory.Build.props can significantly improve the efficiency and reliability of managing large and complex .NET solutions.

Make linting great again

Tue, 29 Jun 2021 10:00:00 GMT

Photo by Bruno Martins on Unsplash

I am a firm believer in the role of linting in keeping code bases from decaying and enforcing standards and consistency. Particularly, I feel linting tools really shine as code bases age. For JavaScript based projects, I find eslint to be pretty powerful but acknowledge that one thing that can keep developers from using it often is that it can be slow in running if not setup properly, especially with regards with caching and is hooked up to commits with something like husky, and if linting violations have been ignored for a while one can drown in a lot errors and warnings.

My common eslint usecase is running it against files I have just added/updated and quite rarely do I need to run it on the whole solution until I am ready to open a pull request. If I do however I always make sure caching is on by using the --cache flag that only runs linting on file changed since last run.

Setup the --cache flags as part of a lint command in your package.json for ease of use

However how do I get to run eslint only on files I just updated? The answer is simple, I use git and one easy way to narrow down these files is to have them staged. Enter lint-staged!

lint-staged

Lint staged is a tool that runs linters against git files that are staged to stop the 💩 from slipping into your code. Notice I said linters, not eslint, so this can be used with various linting tools. Install it with npm i lint-staged --save-dev.

To set it up for the first time run npx mrm lint-staged. I prefer to use the .linstagedrc file for configuration and if you are using eslint you can set it up as

{
  "*": "eslint --cache"
}

I typically run my linting and manually go over any warning/errors. If you want to auto-fix what can be auto-fixed you can use the --fix flag.

Once setup you can give it a go with npx lint-staged. Don’t forget off course that you need staged files for this to work. You can go a step further and set it up to auto-run with hooks by using tools such as husky.

By default lint-staged will only show error in the run summary, to see warnings as well, which is super useful run with the --verbose flag.

Now that you can’t complain about being overwhelmed by lint violations others introduced or slow runs, what is your excuse not to take advantage or linting with eslint 😀?

Control Home Assistant with an Apple Watch

Thu, 04 Mar 2021 10:00:00 GMT

Photo by Luke Chesser on Unsplash

It’s pretty nifty and also surprisingly useful to be able to control your smart home from your wrist. What you control depends on your use cases but I get to show you how to control it and maybe inspire you with how I use it. We will work based on the following use cases

Get Dustbin Bieber busy
Set lounge lights to a relaxed mode
Set lounge lights to max brightness
Turn off all the lights

Dustbin Bieber is our trusty Roborock S6 Pure robot vacuum.

First, you will need to set up actions from the Home Assistant iOS app. Open the app, navigate to settings -> Actions, and there you will be able to add actions. You will need to specify a name for your action, which will be used as a tag, and a text description, which will be displayed on the watch, and optionally you can specify an icon and a color for both the text and icon.

You should then see something like the following when opening the Home Assistant app on your Apple Watch.

Executing Actions

So now we have actions showing up on our Apple Watch, but they do not do anything, well not much at least. Clicking on these actions will currently trigger an event in Home Assistant but because that event has no listener and action attached to it that is all it does. I prefer to use Node Red to execute custom actions in Home Assistant and that is how I have set up my actions.

I have set up a flow as shown in the screenshots below to execute the following.

Capture all ios events
Branch on the event name
On the clean action trigger the vacuum to start a clean session
On the relax action toggle the lounge scene between a cool dim mode and a warm dim mode
On the bright action set the lights in the lounge to full brightness and reset the color
On the all-off action switch off all the lights

To capture all ios events I use the events: all nodes with the event type filtering on ios.actions_fired.

To branch on the event name, I use a switch node with an exit for each action name as defined in the Home Assistant app.

To trigger the cleaning I use the xiaomi_miio service specific to Roborock and specify the rooms to clean as segments.

To set the light scene first I use a switch to determine the current scene by checking a global variable I control, lounge.scene, then toggle the scene by calling the Home Assistant light.turn_on service with a call service node for the desired light/light group. I set the color_name and brightness_pct as desired. After setting each scene I then set the lounge.scene value accordingly between relax.warm and relax.cool.

To set the full brightness I use the Home Assistant light.turn_on service with a call service node for the desired light/light group. I set the color_name to white brightness_pct to 100.

To turn off all the lights I use the Home Assistant light.turn_off service with a call service node for the desired light/light group.

I use the Conbee II Zigbee stick and have deCONZ setup. There I have defined light groups, for instance, I have multiple lights in the lounge set under one group so that from automation I only call the group name and not individual light names. Light groups can also be set up in Home Assistant.

While I prefer Node Red, this entire setup is achievable using Home Assistant automations and capturing the events from there and creating actions.

Test Builder Pattern

Sat, 30 Jan 2021 10:00:00 GMT

Photo by Alan Rodriguez on Unsplash

While working with DTOs and entities, particularly those that are used throughout your domain and boundaries, you will find that they are required in a multitude of tests. A natural approach is to call the constructor of each when required and hydrate them with required setup. While this is very easy and straightforward there are a couple of challenges. There is always a caveat.

To illustrate these challenges let us introduce an example of a simple domain object. Let’s take a dumbed down conceptual take at a financial trade position, that has an identifier and monetary values for fixed and float legs.

Investopedia defines a position as follows. A trade position is the amount of a security, commodity or currency which is owned by an individual, dealer, institution, or other fiscal entity.

The TradePosition properties will be as follows:

A unique identifier of type Guid
A floating leg notional of custom type PositionValue
A fixed leg notional of type PositionValue

In turn the TradeValue properties will be as follows:

A numeric value of type double
A currency iso code of type string

    public record TradePosition(
        Guid Identifier,
        TradeValue FixedLegNotional,
        TradeValue FloatLegNotional);

    public record TradeValue(
        double Value,
        string CurrencyIsoCode);

Now what could go wrong with the simple approach? Well the following:

As your solution grows and the trade position is used in hundreds of tests you may update the domain by say adding new properties that affect the ctor. In this case you need to update a multitude of tests manually.
Over time you may change the meaning of certain things in the domain. For instance if we had maturity in the trade position and furthermore in the future we may want the aspects that determine maturity to change. Any tests that were created with a ‘mature’ trade position will also need to updated with this knowledge. This breaks a lot of principles, ideally the state and validity of an object should be self contained and if the outside has to understand inner workings to attain a given state that is a fail, and yes even for tests.
In many cases you may simply want a trade position for the purposes of the test without a need for it to be in a specific state, but rather just to be valid. In this case you find code copy pasted all over the code base creating a bigger maintenance problem over time.

Investopedia defines maturity as follows. The maturity is the date on which the principal amount of a note, draft, acceptance bond or other debt instrument becomes due.

So, how do I solve this?

Let’s start with another very tempting pattern that I have seen often. The following is a very simple attempt to address this problem.

    public class MediocreTradePositionBuilder
    {
        public static TradePosition CreateTradePosition(
            Guid identifier,
            TradeValue fixedLegNotional,
            TradeValue floatLegNotional
        )
        {
            ...
        }
    }

This does not really solve all the problems, or at least does so by introducing new ones. As our domain evolves and trade position/trade value grow then the method parameters here become messy. This particular case takes in TradeValue for the leg notionals, which means the test has to also build these and know about their inner workings, not great. One way around that is to accept the raw values further causing the parameters to blow up.

    public class MediocreTradePositionBuilder
    {
        public static TradePosition CreateTradePosition(
            Guid identifier,
            double fixedLegNotionalValue,
            string fixedLegNotionalCurrencyIsoCode,
            double floatLegNotionalValue,
            string floatLegNotionalCurrencyIsoCode
        )
        {
            ...
        }
    }

The biggest problems with this pattern however is that it does not communicate intention or evolve well. How do I build a trade position that is mature? The answer often then lies in the test, which is not great as this is an auxiliary concern and not the intent of the test. As you can have multiple scenarios also how does this work with this pattern? One way is to add more parameters to control this, gosh 🤦. Or maybe create multiple of these methods for each scenario. With this pattern I often find this code will still be copy pasted to add the flexibility, so naturally I am not promoting this one.

Finally the tests that use this pattern or none at all tend to be very long and messy to read. By looking at the code at a glance it is not possible to understand what the arrange stage is doing, worse still it makes it hard to distinguish clearly the arrange from the act.

Get to it already, show me the way

Let’s get straight to it and look at a different pattern in code.

    public class TradePositionBuilder : Builder<TradePosition>
    {
        public TradePositionBuilder WithIdentifier(Guid identifier)
        {
            Set(x => x.Identifier, identifier);
            return this;
        }

        public TradePositionBuilder WithFloatLegNotional(PositionValue positionValue)
        {
            Set(x => x.FloatLegNotional, positionValue);
            return this;
        }

        public TradePositionBuilder WithFixedLegNotional(PositionValue positionValue)
        {
            Set(x => x.FixedLegNotional, positionValue);
            return this;
        }

        public TradePositionBuilder ThatIsValid()
        {
            Set(x => x.Identifier, Guid.NewGuid());
            Set(x => x.FixedLegNotional, Some.PositionValue.ThatIsValid());
            Set(x => x.FloatLegNotional, Some.PositionValue.ThatIsValid());
            return this;
        }

        protected override TradePosition Build() => new TradePosition(
            Get(x => x.Identifier),
            Get(x => x.FixedLegNotional),
            Get(x => x.FixedLegNotional));
    }

    public class PositionValueBuilder : Builder<PositionValue>
    {

        public PositionValueBuilder WithCurrencyIsoCode(string currencyIsoCode)
        {
            Set(x => x.CurrencyIsoCode, currencyIsoCode);
            return this;
        }

        public PositionValueBuilder  WithValue(double value)
        {
            Set(x => x.Value, value);
            return this;
        }

        public PositionValueBuilder ThatIsValid()
        {
            Set(x => x.CurrencyIsoCode, "EUR");
            Set(x => x.Value, 1_500_000);
            return this;
        }

        protected override PositionValue Build()
          => new PositionValue(Get(x => x.Value), Get(x => x.CurrencyIsoCode));
    }

OK so why is this better? I’m glad you asked, here we go.

Communication of intent. The methods we see here are very clear on what we are building and while we kept this simple, they can start to cater for scenario like a mature trade position with say ‘ThatIsMatured’ or an invalid trade with ‘ThatIsInvalid’. This makes it easy in tests especially in scenarios where the trade position is not primary to the test but still required for the scenario.
Fluent. Who does not love fluent code, this one makes this even easier to use and very natural to read. If there is anything you should strive for it is readable tests. Recall that when we ditched explicit documentation in code, we made an oath to write self documenting code, one of which is through tests, so they better be easy to read and understand.
Less code. So if the auxiliary act of creating objects for our tests is not key to the tests why should that mess make the test hard to read? I would rather see var tradePosition = A.TradePosition.ThatIsMature() than see all the code that entails this. It is quite rare from a ‘reading through test code’ perspective that I even want to see that at all.
Ease of refactoring. This approach isolates the actual creation of something to one place and one place only much like a factory. So now as your domain evolves and you change the meaning of things, ctors change, etc among many change and as far as your tests are concerned this change only needs to be done in one place.
Clean and simple.** In particular this caters very well for scenarios were you want a valid object without need to control the actual values. Now you do not need to have this copy pasted all over the place.

Let’s see some usage examples:

    public class Sample
    {
        [Fact]
        public void Test()
        {
            // caters for the common scenario, you just need a valid trade position

            TradePosition tradePosition1 = A.TradePosition
                .ThatIsValid();

            // caters for scenario were you need to be in control of all properties
            TradePosition tradePsition2 = A.TradePosition
                .WithIdentifier(Guid.NewGuid())
                .WithFixedLegNotional(
                    // notice the chained builder here, very powerful!
                    Some.PositionValue
                        .WithValue(1_500_00)
                        .WithCurrencyIsoCode("EUR"))
                .WithFloatLegNotional(
                    Some.PositionValue
                        .WithValue(2_480_00)
                        .WithCurrencyIsoCode("EUR"));

            // caters for scenario where you care for single property
            // in your test but the rest also should be valid
            TradePosition tradePosition3 = A.TradePosition
                .WithIdentifier(Guid.NewGuid());
        }
    }

All things considered we can certainly say that this is both simple and powerful. Creating the builders is easy and the pattern fosters for clean code. Using the builders is also very intuitive and the fluent pattern further makes this a pleasure to use.

Show me the guts

Let’s have a look at what is going on behind the scenes to power this and to also understand some of the calls in the builder examples.

The first one is something that will not need to be changed often, if at all. You would have noticed the two builders above have a generic builder base class. The base class is as follows:

    public abstract class Builder<T>
    {
        private readonly Dictionary<string, object> _properties =
            new();

        protected abstract T Build();

        public static implicit operator T(Builder<T> builder)
        {
            return builder.Build();
        }

        /// <summary>
        /// Temporarily hold property value
        /// </summary>
        /// <typeparam name="TProp">Property type</typeparam>
        /// <param name="action">Property expression</param>
        /// <param name="value">Value to hold</param>
        public void Set<TProp>(Expression<Func<T, TProp>> action, TProp value)
        {
            var expression = (MemberExpression)action.Body;
            var name = expression.Member.Name;

            _properties[name] = value;
        }

        /// <summary>
        /// Get previously set value
        /// </summary>
        /// <typeparam name="TProp">Property type</typeparam>
        /// <param name="action">Property expression</param>
        /// <returns></returns>
        public TProp Get<TProp>(Expression<Func<T, TProp>> action)
        {
            var expression = (MemberExpression)action.Body;
            var name = expression.Member.Name;

            var exists = _properties.TryGetValue(name, out var value);

            return exists ? value as dynamic : default(TProp);
        }
    }

This base class has an abstract Build method returning the generic type. What this means is that if you have TradePositionsBuilder: Builder\<TradePosition\>, the implementation must have a build method returning a trade position. This makes sense to be enforced as that is the core purpose of the builder. More interestingly however you will notice that there is an implicit operator overload from the builder itself to the generic type that calls this build method. This allows us to use the builder and have it implicitly cast to the target type in the end, making for succinct code. An alternative would have been to explicitly call build.

Whilst this pattern is simple, effort is made here clearly to make this as powerful out of the box as possible. As this is used in the practice take care to ensure patterns are observed especially the pain points of making builders and using them and address them as you go along.

You would have taken notice of the Get and Set methods that work with a backing dictionary. This is in place to further make creating your own builders easy. Remember that as someone uses your builder they call the fluent methods one at a time before the implicit call to build. That means the values for each property need to be saved somewhere until build is called. If you are working with mutable types like DTOs this is easy as you can create the object and set the properties for each call, however this cannot be for immutable types like domain objects and as you can guess I am promoting immutability wherever possible, so this has to be supported out of the box for my pattern. The builder can call set providing the property and value and call get providing property to get back the previously stored value. Have another look at the two builders above to see this more clearly. Behind the scenes Expressions are used to make this approach clean, easy and tied to whatever type the builder is returning via the generic use. From that the property name is used to store the values in the dictionary and to get them back all in an efficient and type safe manner.

If you were paying close attention you would have noticed the readability added by the use of the A and Some to give results like A.TradePosition and Some.PositionValue that conform to natural language. While optional this can be the cherry on the top to make the calls natural to read.

    public class A
    {
        public static TradePositionBuilder TradePosition => new TradePositionBuilder();
    }

    public class Some
    {
        public static PositionValueBuilder PositionValue => new PositionValueBuilder();
    }

Notice the use of => and not =. This is intentional and care must be taken to do it this way. You want to ensure that each call to this property makes a new builder. This isolation for tests is essential especially considering that the builders have state.

This pattern is very simple. However given that 1. this involves ‘only tests’ and 2. the problem does not seem that complex and pressing, this tends to be highly neglected. The consequences are however dire and do not discriminate because this is only tests. The amount of time lost to go around the challenges of not handling this problem properly can be great.

That’s all folks!

Track API breaking changes in .net

Fri, 29 Jan 2021 00:00:00 GMT

Photo by Jackson Simmer on Unsplash

If you have ever maintained a public API, in particular a large one, you may know just how easy it is to accidentally make breaking changes and roll them out. I have found some interesting ways in the past to avoid this, but I won’t bore you with those as I found a better way. Microsoft rolled out a Roslyn Analyzer, PublicApiAnalyzers, that solves just this problem.

The Analyzer works via two text files, PublicAPI.Unshipped.txt and PublicAPI.Shipped.txt. The analyzer once setup gives the option via Visual Studio to track certain entities as being part of the public API. This will be done by serializing as text, a representation of that entity in the PublicAPI.Unshipped.txt. How you use the PublicAPI.Shipped.txt is entirely up to you. Once you have made changes to the API you will see a warning about something not being in the public API, ideally you would choose the option to update the API details. Now each time you see a change in your pull requests that results in this file being changed it is good reflection point to check if all the API changes are safe or intended and as we will cover later, you can also fail the build on some of the changes.

Example

You will need to install the nuget package Microsoft.CodeAnalysis.PublicApiAnalyzers and manually add the two text files we discussed to your project. Ensure that the text files are added as AdditionalFiles in, this is important for the analyzer to work. Your project file should look something like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <AdditionalFiles Remove="PublicAPI.Shipped.txt" />
    <AdditionalFiles Remove="PublicAPI.Unshipped.txt" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.CodeAnalysis.PublicApiAnalyzers" Version="3.3.2">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
  </ItemGroup>

</Project>

Let’s say you had the following as part of your public API

namespace PublicApiSample
{
    using System;
    using System.Collections.Generic;

    public delegate void TagsAppender(List<string> tagBuilder);

    public class Metrics
    {
        public void Timer(string key, TimeSpan elapsed, TagsAppender tags = null)
            => throw new NotImplementedException();

        public IDisposable Timer(string key, TagsAppender tags = null)
            => throw new NotImplementedException();
    }
}

You will now notice a warning on the various types indicating that they are not part of the public API.

You can proceed to add everything in this document to the public API.

The PublicAPI.Unshipped.txt file should now have the following.

PublicApiSample.Metrics
PublicApiSample.Metrics.Metrics() -> void
PublicApiSample.Metrics.Timer(string key, PublicApiSample.TagsAppender tags = null) -> System.IDisposable
PublicApiSample.Metrics.Timer(string key, System.TimeSpan elapsed, PublicApiSample.TagsAppender tags = null) -> void
PublicApiSample.TagsAppender

Now if you for instance blindly add a new constructor that replaces the default constructor it would be breaking the public API. You will have the warning again to add this to the public API and doing so would raise a question in the pull request.

If again you delete the first of the two methods in the Metrics class and build the project you will have the following warning in the build output: Symbol 'PublicApiSample.Metrics.Timer(string key, System.TimeSpan elapsed, PublicApiSample.TagsAppender tags = null) -> void' is part of the declared API, but is either not public or could not be found

In addition to the IDE features, all the change scenario have warnings emitted on build. I would recommend determining which of those warnings are of interest and treating them as error instead.

The following warnings are available:

RS0016: Add public types and members to the declared API
RS0017: Remove deleted types and members from the declared API
RS0022: Constructor make noninheritable base class inheritable
RS0024: The contents of the public API files are invalid
RS0025: Do not duplicate symbols in public API files
RS0026: Do not add multiple public overloads with optional parameters
RS0027: Public API with optional parameter(s) should have the most parameters amongst its public overloads
RS0036: Annotate nullability of public types and members in the declared API
RS0037: Enable tracking of nullability of reference types in the declared API
RS0041: Public members should not use oblivious types
RS0048: Missing shipped or unshipped public API file

See the documentation of these warnings here.

Making your builds fail on these warnings is easy. You could for example treat all the warnings as errors by adding the following to the csproj file.

<WarningsAsErrors>$(WarningsAsErrors);RS0016;RS0017;RS0022;RS0024;RS0025;RS0026;RS0027</WarningsAsErrors>

Dotnet Tools

Tue, 01 Dec 2020 00:00:00 GMT

Photo by Cesar Carlevarino Aragon on Unsplash

Dotnet tools are not just add-ons; they are essential components that significantly enhance our development workflow in .NET environments. They streamline complex tasks, automate repetitive processes, and ensure consistency across our projects. In this post, I’ll share some of my favorite tools that have become indispensable in my .NET development journey. From managing dependencies to ensuring consistent coding styles, these tools are vital for any serious .NET developer looking to optimize their workflow and maintain high-quality code.

Managing references

I like to keep my projects clean and tend to prune transitive dependencies. Doing it manually, however, can be tedious. There is a tool for that. Snitch is a powerful tool for pruning unnecessary transitive dependencies in your .NET projects. It helps in identifying packages that are no longer needed, thus keeping your project clean and efficient. Especially useful in large projects, Snitch simplifies what would otherwise be a daunting task of manually sifting through dependencies.

[snitch] (https://github.com/spectresystems/snitch) dotnet tool install -g snitch

In a recent large-scale project, I faced the challenge of managing an overwhelming number of dependencies. Snitch was a game-changer, identifying several unused packages that were bloating our project. By removing these, we not only streamlined our application but also reduced build times significantly.

Visualizing code coverage

I use coverlet and often need to convert the output into a human-readable format, HTML in my case. There is a tool for that. Report Generator converts coverage reports generated by Coverlet into a human-readable format like HTML. This tool is vital for visualizing which parts of your code are being tested, helping you to identify areas that need more thorough testing. It’s an essential tool for maintaining high-quality code and ensuring that your tests are as comprehensive as possible.

[report generator] (https://github.com/danielpalme/ReportGenerator) dotnet tool install -g dotnet-reportgenerator-globaltool

Enforcing .editorconfig

I honestly get frustrated when formatting is inconsistent, particularly when it makes me start to read through code slower. Well, how do you enforce that nice editor config you already worked on as a team? There is a tool for that. Dotnet-Format is a lifesaver when it comes to enforcing coding styles and formats. This tool automatically applies the rules specified in your .editorconfig file across your project, ensuring consistent formatting. It’s especially helpful in collaborative environments, eliminating the frustration of inconsistent coding styles among team members.

[dotnet format] (https://github.com/dotnet/format) dotnet tool install -g dotnet-format

Update packages

I like dependabot but there are times when I need to do this locally as well. In particular, if say there are 10 packages to update, I would like to update 1 at a time and on each update run a dotnet clean, build and test. If all is good then move to the next. There is a tool for that.

[nukeeper] (https://github.com/NuKeeperDotNet/NuKeeper) dotnet tool install nukeeper --global

For my particular use case I use the nukeeper update option. There are however more options including the ability to open pull requests for a project or entire org (only supported in GigHub)

Conclusion

In conclusion, the right set of .NET tools can make an immense difference in your development experience. They bring efficiency, consistency, and quality to your work. Whether it’s managing dependencies, ensuring code coverage, or maintaining a consistent coding style, these tools are key to a smooth and effective .NET development process. I encourage every .NET developer to explore these tools and integrate them into their workflow to experience the benefits firsthand.

Unit Testing with pytest

Sat, 04 Apr 2020 00:00:00 GMT

Photo by Hitesh Choudhary on Unsplash

Unit testing is one of the most valuable tools we have as developers to ensure that we write working code. I have tried out both the inbuilt unittest and the third party pytest and no doubt have found pytest to shine. At a glance the two appear to be equal and unittest for me personally has a better assertions syntax. However when you start to do some more advanced things like fixtures and mocking then you really see pytest shine and when it comes to control of running tests you will have way more at your disposal with pytest.

This article is not meant to compare the two but instead will be focused on highlighting some key features of pytest.

Sample test

Let’s look at two very simple tests

from typing import NoReturn

import pytest


def to_str(x: int) -> str:
    return str(x)


def test_to_str() -> None:
    assert to_str(3) == '3'


def raises() -> NoReturn:
    raise SystemExit(1)


def test_raises()  -> None:
    with pytest.raises(SystemExit):
        raises()

For tests to be discovered they have to have the naming convention test_*

PRO TIP: If you want to continually run your tests in Pycharm as you make changes you can set the current test run to be run on a delay when you stop coding by toggling this with the auto-test icon (refresh icon). The top bar of the test runner has a settings menu item (cog wheel icon) and there you can configure the delay for values between 1 and 10 seconds.

I have pytest set as the default test runner in Pycharm, see screenshot below for setting. Additionally you can run this from the console with the command pytest -q test_file.py

The tests above are simple functions, alternatively they could have been grouped under a class:

To my delight each test run by pytest in a class gets it’s own instance, which is great for isolation. Are you paying attention NUnit?

from typing import NoReturn

import pytest


def to_str(x: int) -> str:
    return str(x)


def raises() -> NoReturn:
    raise SystemExit(1)


class TestSamples:
    def test_to_str(self) -> None:
        assert to_str(3) == '3'

    def test_raises(self) -> None:
        with pytest.raises(SystemExit):
            raises()

    def test_needsfiles(self, tmpdir):
        print(tmpdir)
        assert 0

Test that creates temp files

If you are writing a test that creates temp files or any other scenario that requires a temp folder, pytest has your back, no need to jump through hoops. This can be achieved by using a parameter named tmpdir that will be fulfilled by a fixture to inject a LocalPath to your test.

from py._path.local import LocalPath

def test_create_temp_files(tmpdir: LocalPath):
    assert 0

JUnitXML

You can configure your test runs to emit the JUnit XML report that can be consumed by many Continuous Integration tools like Jenkins, providing a visual report on. This can be achieved by triggering the test run with the junitxml flag and providing output path pytest —junitxml=path

If you require a human readable report you can also emit plain text by triggering with the resultlog flag and providing an output path pytest —resultlog=path

Assert that an exception is raised

Asserting if an exception has been raised is straightforward with context managers

def test_raises() -> None:
    with pytest.raises(SystemExit):
        raises()


def test_raises_access_exception() -> None:
    with pytest.raises(SystemExit) as exc_info:
        raises()
    print(exc_info.typename)

Test logging

It is very common to want to test if you code is logging out as expected, this can be achieved easily with the caplog built in fixture.

import logging
from _pytest.logging import LogCaptureFixture

def get_logger(name: str) -> logging.Logger:
    logger = logging.getLogger(name)
    logger.setLevel("DEBUG")
    return logger

def log()-> None:
    logger = get_logger(__name__)
    logger.info('sample log')


def test_logging(caplog: LogCaptureFixture):
    log()
    print(caplog.messages)
    print(caplog.text)
    print(caplog.text)
    print(caplog.records)
    print(caplog.record_tuples)
    caplog.clear()

The same can be achieved with capsys for writes to sys.stdout and sys.stderr

def test_sys_output(capsys: CaptureFixture):
    print('hello there')
    captured = capsys.readouterr()
    print(captured)

Mocking

This is the big one. To have complete test coverage you will no doubt have to test code that has external dependencies that you do not desire in your test or that you cannot control at all. In that case one way, is to mock the specific dependencies. The mocks can target both attributes and behavior. This is surprisingly very easy to achieve with pytest using the MonkeyPatch fixture.

This fixture has support for mocking various things such as method calls, properties, dictionaries and environment variables. Here are a few examples:

import functools
import os
from pathlib import Path

import pytest
import requests

from _pytest.monkeypatch import MonkeyPatch


def get_path():
    return Path.home()


# Mock method return value
def test_get_path(monkeypatch: MonkeyPatch):
    def mock_return():
        return Path("/abc")

    monkeypatch.setattr(Path, "home", mock_return)

    x = get_path()
    assert x == Path("/abc")


def get_json(url):
    response = requests.get(url)
    return response.json()


class MockResponse:

    @staticmethod
    def json():
        return {'Name': 'Chloe'}


# Mock method return object
def test_get_json(monkeypatch: MonkeyPatch):
    def mock_get(*_args, **_kwargs):
        return MockResponse()

    monkeypatch.setattr(requests, 'get', mock_get)

    result = get_json('https://fake')
    assert result['Name'] == 'Chloe'

# Scoping of mock with context manager
# Useful as mocking certain framework aspects may break pytest
def test_patch_scoped(monkeypatch):
    with monkeypatch.context() as m:
        m.setattr(functools, "partial", 10)
        assert functools.partial == 10


def get_environment_variables():
    username = os.getenv("USER")

    if username is None:
        raise OSError("USER environment is not set.")

    return username.lower()

# Mock environment variables
def test_upper_to_lower(monkeypatch: MonkeyPatch):
    monkeypatch.setenv("USER", "TestingUser")
    assert get_environment_variables() == "testinguser"


def test_raise_exception(monkeypatch: MonkeyPatch):
    monkeypatch.delenv("USER", raising=False)

    with pytest.raises(OSError):
        _ = get_environment_variables()

That is all I will cover in this post. Take a look at the pytest website for documentation and more examples here.

Pycharm hidden gems

Wed, 01 Apr 2020 00:00:00 GMT

Photo by Artturi Jalli on Unsplash

Here are a few interesting features of Pycharm that are otherwise not commonly known or used.

Live templates

Live templates make it easy to add commonly used or otherwise annoying boiler plate code. I will not cover the existing live templates as you can have a look for yourself (File -> Settings [ctrl + alt + s] -> Editor -> Live Templates)

I do have a few custom ones to share that may be of interest to you.

# add a new sanic route
@app.get('/$ROUTE$')
async def $ROUTE$(_: Request) -> HTTPResponse:
    return response.json(['a', 'b'])

# import logger in a file
import logging
logger = logging.getLogger(__name__)

# create a data class
@dataclass
class $NAME$:
    test: str

# create a json data class
@dataclass_json(letter_case=LetterCase.CAMEL)
@dataclass
class $NAME$:
    test: str

Post fix autocompletion

Post fix autocompletion can make writing certain Python constructs faster. For example if you want the following

if user.authorized:
    current_user = user

Instead of typing if user.authorized , you could type user.authorized.if and hit tab resulting in the if statement block and cursor on the next line.

The other supported postfix template are:

a.ifn
if a is None:

a.ifnn
if a is not None:

a.len
len(a)

abs(1).main

if __name__ == '__main__':
    abs(1)

a.not
not a

a.print
print(a)

a.return
return a

a.while
while a:

To my disappointment Pycharm currently does not support the adding new post fix template targetting python, but somehow does for SQL.

File Templates

File templates are a sort of specification for the default contents of new files you create. Adding your own for the commonly bootstrapped files types can really increase your creativity.

Adding/Editing is easy (File -> Settings [ctrl + alt + s] -> Editor -> File and Code Templates)

Here are a few examples for context.

Bootstrap a new new unit test file

import unittest

class #[[$Title$]]#Tests(unittest.TestCase):
    def test_x_given_when_then(self):
        self.assertEqual(True, False)

if __name__ == '__main__':
    unittest.main()

Bootstrap a new sanic api

import logging

from sanic import Sanic, response
from sanic.request import Request
from sanic.response import HTTPResponse
from sanic_openapi import doc, swagger_blueprint
from sanic_openapi.doc import summary

logger = logging.getLogger(__name__)
app = Sanic(__name__)
app.logger = logger

app.blueprint(swagger_blueprint)
app.config['API_SCHEMES'] = ['http', 'https']
app.config['API_TITLE'] = APP_NAME
app.config['API_DESCRIPTION'] = ''


@summary('')
@doc.produces(doc.List(doc.String))
@app.get('/test')
async def test(_: Request) -> HTTPResponse:
    return response.json(['a', 'b'])


if __name__ == '__main__':
    app.run(host=HOST, port=PORT, threaded=True)


def api() -> Sanic:
    return app