How to create “template” MarkDown files for generating GitHub pages

Recently I needed to write a script that could auto-generate sample MarkDown files that would be published as GitHub Pages. The idea is that someone could either run my script or run the GitHub Action that calls my script to generate a stub of a new WhatTheHack. The stub would contain the correct format, generate the requested number of challenges, generate the MarkDown navigation links & give the author of a new WhatTheHack the template to follow.

Generating the template by embedding the template text directly in the bash script was easy. The hard part was that we wanted to have both human-readable template files (so authors could see examples), but also use these example template files to generate the stub (machine-readable).

The solution is to use both GitHub MarkDown comments and regexes to do the find and replace.

I started with the template file that serves as the basis of the stub. This template has sample text for describing what should be in each section.

# What The Hack - xxx-IoTHackOfTheCentury

## Introduction
The IoT Hack of the Century will take you on a whirlwind tour in the world of IoT and how it is being used in the modern world of mineral extraction in exotic locations like the Arctic and the wilds of South Africa.

## Learning Objectives
In this hack you will be solving the common business problem that companies in the mineral extraction industry face and how IoT solutions from Azure are brought to bare

1. Provision an IoT Hub
2. Set up an IoT Edge device
3. Bring Azure Sphere to your solution for scale and resiliency 

## Challenges
1. Challenge 01: **[Description of challenge](Student/Challenge-01.md)**
	 - Description of challenge
1. Challenge 02: **[Description of challenge](Student/Challenge-02.md)**
	 - Description of challenge
1. Challenge 03: **[Description of challenge](Student/Challenge-03.md)**
	 - Description of challenge

I need to put in placeholders that will get replaced when the template is turned into a stub. Easy enough to use the bash variables as placeholders in the template.

# What The Hack - ${wthDirectoryName}

...

## Challenges
${challengesSection}

I can use cat to read in the file, store the text in a variable, use eval to do the string interpolation and write out the file.

local -r pathToFile=$1
local -r pathToWriteTo=$2


local -r wthDirectoryName="xxx-IoTHackOfTheCentury"
local -r challengesSection=$(GenerateChallengesSection $numberOfChallenges "Student" "Challenge")

local initialText=$(cat $pathToFile)

local template=$(eval "cat <<EOF
$initialText
EOF
" 2> /dev/null)

cat > "$pathToWriteTo" <<< $template

This substitutes the wthDirectoryName & challengesSection placeholders in the template file with the values of those variables when the stubs are written out.

This works well, but we also wanted to include sample text in the templates so if someone wanted to see what would get generated, it is available. Therefore, I needed both human-readable text & machine-readable text. The template file needed to include the human-readable text if someone looked at the template file directly. It also needed to have this “sample” text removed if someone ran the script to generate the stubs.

Enter GitHub MarkDown comments & regex.

First, I needed to decide on a format for the template files. Here is the “Challenges” section.

## Challenges
<!-- REPLACE_ME ${challengesSection} (remove this from your MD files if you are writing them manually, this is for the automation script) REPLACE_ME -->

<!-- REMOVE_ME (this section will be removed by the automation script) -->
1. Challenge 01: **[Description of challenge](Student/Challenge-01.md)**
	 - Description of challenge
1. Challenge 02: **[Description of challenge](Student/Challenge-02.md)**
	 - Description of challenge
1. Challenge 03: **[Description of challenge](Student/Challenge-03.md)**
	 - Description of challenge
<!-- REMOVE_ME (this section will be removed by the automation script) -->

The REPLACE_ME line will get cleaned up to be the following by a preprocessing step. This text is there so if a human reads the template file, they will understand that this is for automation and should be removed if someone is copying the template file manually.

## Challenges
${challengesSection}

The REMOVE_ME section contains the sample text someone would look at to see what the “table-of-contents” (TOC) should be if writing the files manually. The automation script will generate a TOC like this. It should be completely removed in the generated stub files.

The first regex is pretty simple since it is only 1 line and can be done with sed. It will get rid of the “REPLACE_ME” comment before and after the substitution variable.

sed -e 's/<!-- REPLACE_ME \(\${.*}\) .* REPLACE_ME -->/\1/gm')

The 2nd is a little harder because it is a multi-line regex substitution (the TOC spans multiple lines). I tried to use sed for this too, but it was easier to switch to perl. It will get rid of everything including the “REMOVE_ME” tag.

perl -0777 -pe "s/<!-- REMOVE_ME .* -->(?:.*)*<!-- REMOVE_ME .* -->//gms")

The entire regex substitution looks like this.

local initialText=$(cat $pathToFile)

local -r returnText=$(echo "$initialText" | sed -e 's/<!-- REPLACE_ME \(\${.*}\) .* REPLACE_ME -->/\1/gm' | perl -0777 -pe "s/<!-- REMOVE_ME .* -->(?:.*)*<!-- REMOVE_ME .* -->//gms")

The entire code would look like the following.

local -r pathToFile=$1
local -r pathToWriteTo=$2

local -r wthDirectoryName="xxx-IoTHackOfTheCentury"
local -r challengesSection=$(GenerateChallengesSection $numberOfChallenges "Student" "Challenge")

local initialText=$(cat $pathToFile)

local -r returnText=$(echo "$initialText" | sed -e 's/<!-- REPLACE_ME \(\${.*}\) .* REPLACE_ME -->/\1/gm' | perl -0777 -pe "s/<!-- REMOVE_ME .* -->(?:.*)*<!-- REMOVE_ME .* -->//gms")

local template=$(eval "cat <<EOF
$returnText
EOF
" 2> /dev/null)

cat > "$pathToWriteTo" <<< $template