The Ops Community ⚙️: Andrew Owen

Getting started with GraphQL

Andrew Owen — Sun, 06 Apr 2025 12:51:58 +0000

GraphQL is an API query and manipulation language. Created by Facebook in 2012, it was open-sourced in 2015. In 2018 it moved to the GraphQL Foundation and introduced a schema definition language (SDL). It seems to be replacing REST as the standard way to expose public APIs. With REST, you have to define the inputs and outputs for each endpoint. Whereas with GraphQL, there's only one endpoint and you define a schema. The user sends only the required data to get what they want back. And unlike SQL, you're not limited to a single data source.

This year, I've had to get rapidly up-to-speed with GraphQL. I thought I'd be starting from nothing, but I'd forgotten that TinaCMS (the headless content management system that I use with this site) uses it. One of the first problems I had to solve was how to generate static documentation. My limited research led me to two possible solutions: SpectaQL, developed from the earlier DociQL, and Magidoc. The latter has built-in search, so that made my choice for me. I use Hugo as my static site generator, so the first thing I had to do was start the local version of TinaCMS from my site's Git repository:

npx tinacms dev - c "hugo server -D -p 1313"

With the local server running, you can access GraphiQL at http://localhost:1313/admin/#/graphql. GraphiQL is a reference implementation of the GraphQL API playground. If it's too basic for you, there's a commercial alternative called Apollo. The TinaCMS implementation gives you three options (selected from the icons on the left):

Docs: API docs in a tree structure.
History: Previous queries.
Queries: Query builder.

After you've taken a look at the Docs section in GraphQL, you'll understand why I wanted to generate a static site. With Magidoc it's easy. I added this configuration file to my Git repository:

export default {
  introspection: {
    type: 'url',
    url: 'http://localhost:4001/graphql',
  },
  website: {
    template: 'carbon-multi-page',
      customStyles: ['/static/css/custom.css'],
  },
}

Then to generate the documentation, enter: pnpm add --global @magidoc/cli\@latest && magidoc generate (you'll need pnpm installed). By default, this creates a static site in a folder called docs. However, you can't just open the index.html file. You'll need to launch the server with magidoc preview and then follow the link. You may want to add the docs folder to your .gitignore file. But in production, you can deploy using any web server. The output is based on IBM's Carbon Design System.

Now you've got some static searchable documentation, you can start exploring your schema. GraphQL schema can be written in any programming language that implements the type system. With TinaCMS that means either TypeScript or JavaScript. Here's a snippet of my Tina config.js file:

    schema: {
      collections: [
        {
          name: "blog",
          format: "md",
          label: "Blog",
          path: "content/blog",
          defaultItem: () => {
            return {
              draft: true,
            }
          },
          fields: [
            {
              name: "draft",
              type: "boolean",
              label: "Draft",
              required: true,
            },
            {
              name: "title",
              type: "string",
              label: "Title",
              isTitle: true,
              required: true,
            },
            {
              name: "date",
              type: "datetime",
              label: "Date",
            },
            {
              name: "description",
              type: "string",
              label: "Description",
            },
            {
              name: "image",
              type: "image",
              label: "Image",
            },
            {
              name: "image_license",
              type: "string",
              label: "Image License",
            },
            {
              name: 'tags',
              type: 'string',
              label: 'Tags',
              list: true,
            },
            {
              name: 'body',
              type: 'rich-text',
              isBody: true,
              label: "Body",
              templates: [
                {
                  name: 'shortcode',
                  label: 'shortcode',

This creates a schema type called Blog with the fields:

draft: Boolean!
title: String!
date: String
description: String
image: String
image_license: String
tags: [String]
body: JSON
id: ID!
_sys: SystemInfo!
_values: JSON!

Fields can be scalar (Boolean, Float, Integer, String and so on) or complex (containing other data). A trailing exclamation mark ( ! ) means the field is required (non-nullable). Square brackets mean an array. If the type is JSON, then a set of sub-fields is defined. You can also use previously defined types to create relationships. For example, if your blog has multiple contributors you could have a field called author with the type User!.

Queries

Because there's only one endpoint, you have to tell it what you want. For example, to get the list of collections you'd use:

{
  collections {
    name
  }
}

The collections filed is the root field. Everything else is the payload. Because the payload only contains name the query returns a list of all the collections:

{
  "data": {
    "collections": [
      {"name": "blog"},
      {"name": "recipe"}
    ]
  }
}

You get exactly the amount of information you ask for in the payload. You can also pass arguments. For example, if you wanted to get the title, date and draft status of this article you could use:

{
  blog(relativePath: "getting-started-with-graphql.md") {
    title
    date
    draft
  }
}

However, typically you would use a variable in the query, like this:

query blog($relativePath: String!) {
  blog(relativePath: $relativePath) {
    title
    date
    draft
  }
}

Variables have a prefix ( $ ). The type (String in this example) is defined in the query. The variable is passed in the blog parameters.

Mutations

Query is the equivalent of GET in REST or read in traditional data terminology. For create, update and delete there's mutation. The syntax is the same as for queries. You might have noticed the ID type earlier. If you specify it when creating a record, the GraphQL server will give it a unique identifier. You can try out this example from the TinaCMS documenation (sticking with the Napoleon Dynamite references):

mutation {
  updatePost(relativePath: "voteForPedro.json", params: {title: "Vote For Napolean Instead", category: "politics", author: "content/authors/napolean.json"}) {
    title
    category
    author {
      ... on Author {
        id
      }
    }
  }
}

Subscriptions

I've written previously about event-driven architectures. Subscriptions are a way to get data from the GraphQL server every time an subscribed event takes place. Not all GraphQL servers are configured to support websocket subscriptions, and that's true of my TinaCMS instance. But if it was, this is what a request to be notified when a new blog article is created would look like:

subscription {
  newBlog {
    title
  }
}

Custom scalars

Five months after I originally wrote this article, I ran into an issue with Magidoc. The doc build was failing with missing definitions for custom scalars. At first, I thought this was a bug, but it turns out that the bug was that the previous version would build the docs even if the definitions were missing. You don't need to provide definitions for the predefined scalars (String, Int, Float, Boolean and ID). But if you have any custom scalars in your schema you must provide an example for each in JSON format in the Magidoc config file. These definitions go under options in the website section:

  website: {
    template: 'carbon-multi-page',
      customStyles: ['/static/css/custom.css'],
    options: {
      queryGenerationFactories: {
        accountID: "01234567-890a-bcde-f012-34567890abcd",
      }
    }
  }

Conclusion

It's beyond the scope of this article to do more than cover the basics. For more information, the GraphQL documenation is a good starting point. TinaCMS is a good application to start experimenting with its GraphQL API. For the adventurous, Kingdom Orjiewuru wrote the first part of an article on building a simple document manager with GraphQL.

Getting started with Bitbucket Pipelines

Andrew Owen — Sun, 06 Apr 2025 12:41:31 +0000

I'm a big fan of GitHub Actions. But if you're working for an enterprise software company, there's a fair chance you're using Atlassian's Bitbucket Cloud (along with Confluence and Jira). If so, then you can use Pipelines to build continuous integration and deployment workflows. If you're new to DevOps and CI/CD, I have a TL:DR for you.

In Bitbucket, go to your repository and select Pipelines.
Click Create your first pipeline to scroll down to the template section.
Select the Starter pipeline. This will create a file called bitbucket-pipelines.yml.

Previously, I wrote a GitHub Action to unpack a zip archive. So let's recreate that:

image: atlassian/default-image:3

pipelines:
  default:
    - step:
        name: 'extract a zip file'
        condition:
          changesets:
            includePaths:
              - "*.zip"
        script:
          - rm -r uploads/extracted
          - filename=$(basename -s .zip *.zip)
          - unzip *.zip
          - rm *.zip
          - mv $filename temp
          - mv temp/out/* .
          - rm -r temp
          - git add .
          - git commit -m "unzip"
          - git push origin main

When you're done, click Commit File.

Bitbucket Pipelines runs your builds in Docker containers. So first you need to choose an image. The default is atlassian/default-image:latest. Atlassian recommends using this until you get your pipeline working and then finding a specific image. So in this instance, we're using Ubuntu 20.04 LTS.

default tells the script to run when a commit is pushed to any branch.
parallel enables you to run steps simultaneously, but it's not necessary here.
step is an individual task.
name is the display name of the step.
condition limits the circumstances under which the script runs.
changesets specifies changes to use as a condition.
includePath sets the file path that will match the condition.
script is the Linux CLI input.

The script is virtually identical to the GitHub Action I wrote, so for a more detailed explanation, you can read that.

Image: original by Sigmund.

Adding languages to a Hugo site

Andrew Owen — Thu, 16 Mar 2023 08:44:48 +0000

I'm a long-term advocate for localization, but this site has been monolingual for over a year now. It's past time I started following my own advice. So last weekend I finally got around to localizing the site for French.

As with most localization tasks, I'm retrofitting it to an existing project. Fortunately, I chose Hugo as my static site generator, and it has built-in internationalization support using go-i18n. On the downside, I chose a theme that doesn't fully support localization, which meant there was some work involved. But not as much as I'd expected. So let me take you through the steps involved.

First, I updated my config.toml file to tell Hugo that my site is now multilingual:

defaultContentLanguageInSubdir = false
DefaultContentLanguage = "en-us"
[languages]
  [languages.en-us]
    title = "Andrew Owen | Writer | Designer"
    languageName = "English"
    weight = 1
  [languages.fr]
    title = "Andrew Owen | Écrivain | Concepteur"
    languageName = "Français"
    weight = 2

Setting defaultContentLanguageInSubdir leaves the language out of the URL for the default language, in my case US English. Any settings you leave out from the language definitions will fall back to the default.

My 404.md page is already stored in a folder called en. So adding a French version was just a case of creating a new folder called fr, copying the file across and changing the text.

The next step was to set up some a set of translatable phrases for the generated content. This is as simple as creating an i18n folder in the root of the Hugo repository and adding a YAML or TOML file for each language. In my case en-us.yaml and fr.yaml. These phrases are included using the shortcode {{T "phrase" | formatting}}. If you want to include HTML such as <br> tags, then use safeHTML for the formatting value. Each value should have an ID and a definition in each language file. For example:

- id: January
  translation: "janvier"

In several places in the site, titles were derived from data contained in YAML files. In each of these cases, I replaced the data-derived version with a shortcode to the translated version instead. I also had to modify some of the links to go to the correct place when viewing the site in French. In most cases, that just meant using the .Site.Language.Lang value.

Because I'm only supporting two languages, I wanted to have a simple toggle. The nav bar was starting to get rather full, so I removed the Home link, because the logo already takes you there. I had to change the RSS link to make it work in French, so I also replaced it with the icon. I figure anyone who still cares about RSS should recognize it:

<li class="nav-item">
  {{ if eq .Site.Language.Lang "en-us" }}
  <a class="nav-link" href="/blog/index.xml"><i class='fa fa-rss'></i></a>
  {{ else }}
  <a class="nav-link" href="/fr/blog/index.xml"><i class='fa fa-rss'></i></a>
  {{ end }}
</li>

The last thing to do was to translate the individual articles. Hugo gives you two ways of doing this. You can set up a folder for each language, and if you have three or more languages, you should do this. But by default, Hugo will treat anything ending in .md as the default language, and you can add other languages by adding the language code to the extension, for example: .fr.md. If you're using folders, then you add the folder details to the language definitions in your TOML file. For example:

[languages]
  [languages.en]
    contentDir = "content/english"

While I was editing the site in VScode, I also took the opportunity to reorganize the images. They were getting a bit tricky to navigate with TinaCMS, so I moved the blog images into folders by year. I also did some cleanup on the top nav bar. The last thing to do was change the article date string from {{ .PublishDate.Format “2 January 2006” }} to {{.Date.Day}} {{i18n .Date.Month}} {{.Date.Year}} after setting up a set of translatable date strings in the YAML files in the i18n folder.

Of course, just when I thought I'd caught everything, I noticed that tag links were broken. I fixed this by making the URL relative so that it would pick up the current language. I'm pleased to say that I didn't have to do anything at all to localize search.

With the technical piece done, all that's left to do is the translation. There are ways to automate the process, but there's still no substitute for a human reviewer. I'm using a combination of DeepL, LanguageTool and my own modest language ability. To begin with, I'm only translating the core content. But eventually I intend to have a fully translated site. If you are a native French user, and you spot any egregious mistakes, please drop me an email.

Setting up a website with GitHub, Hugo, Netlify and TinaCMS

Andrew Owen — Thu, 02 Mar 2023 09:23:42 +0000

My first blog post of 2022 was on setting up a free personal website with GitHub, Hugo, Netlify and Forestry. But Forestry is due to be discontinued at the end of this month. So this is a rework of that post using Forestry's successor TinaCMS. If you need to migrate, a tool and guide are available. Or you can do it the hard way and follow my post from earlier this year.

I spent most of last year as a developer advocate, and I felt that my personal website should be a more modern solution than WordPress. At a previous job, I'd built a developer portal with Hugo and spoken to people at Netlify and Forestry. I also already had a GitHub account. What made that a particularly attractive solution was that, aside from the domain name fees, it doesn't cost anything. Hugo is open source. And GitHub, Netlify and TinaCMS all have a free tier for personal websites.

Next, I needed to find a portfolio style Hugo theme that would integrate with all these services. After some searching, I settled on using Kross, a free MIT-licensed theme from Themefisher. It's built on Bootstrap, HTML5 and CSS3. I recommend finding a theme hosted on GitHub with the option to deploy to Netlify. You can fork the repository (create a copy in your own repository) and deploy it automatically. Then you just need to add TinaCMS support.

TinaCMS has come a long way since it came out of beta in November 2022. As more users adopt it, it's become more mature. And that's also reflected in the documentation. I was expecting to have to write a guide to setting up Hugo on Tina, but it's already been done.

Modify your theme

You'll have the site up and running very quickly, but you'll want to change the placeholder text and images. I found the easiest way to replace the images was directly through the GitHub web interface. When replacing the illustration SVGs, I had to manually edit the files in VS Code to set the correct display size. For the text, save yourself the pain of YAML induced build errors and do it all in Tina Cloud.

The contact form seems to have a dependency on a non-free service, so I've switched it off for now and provided a mailto link instead. There also seems to be a bug in the Portfolio section where, with tags enabled, some content wasn't rendering, so for now I've removed the tags.

I wasn't completely happy with the typography, but originally the entire Kross theme is inherited as a submodule, and I was only overriding the illustrations. That way, if the theme got updated, my site would automatically get updated too. However, there were changes I needed to make and after about a week I replaced the submodule with a local copy of the theme.

With most of the static content now in place, you can concentrate on writing your blog in Tina Cloud. If you need to make developer changes to the site, you can do it locally in TinaCMS, which for a personal site means you can do without staging and deploy to your main repository in GitHub.

I had an issue with the SSL/TLS certificate. I resolved it by making andrewowen.net the primary domain and *.andrewowen.net the redirect domain in the Netlify settings.

One tool I find invaluable when working in web applications like Tina Cloud is LanguageTool. It's a multilingual grammar, style and spell checker. The one caveat is that it doesn't seem to check text on TinaCMS within blockquotes. So if you use it, make sure you do the checking before applying the blockquotes style.

Check your styles

The other thing I learned post-launch is that I should have checked to see how the theme renders all the styles. It was doing some odd things with emphasis and code snippets, so that required some changes to the theme. Fortunately, I only had to edit the style.css file in the theme.

If you're using the Kross theme, I would suggest making the following changes:

ol {
list-style-type: decimal;
margin: 0
}

    ul {
      list-style-type: disc;
      margin: 0
    }

    .content strong {
        font-weight: 700;
        color: #4c4c4c;
        font-size: 15px;
        line-height: 1.8;
        font-family: Roboto, sans
    }

Otherwise, numbers and bullets are missing from lists and the bold (strong) style uses the heading font. I also decided to replace all the fonts with IBM Plex, a free alternative to Helvetica. After some user feedback, I also set the base font size to 17 pixels.

I made a change to layouts/index.html to increase the size of the social media icons:

<!-- social icon -->
<ul class="list-unstyled ml-5 mt-3 position-relative zindex-1">
{{ range .Site.Params.social }}
<li style="font-size:32px" class="mb-3"><a class="text-white" href="{{.URL | safeURL }}"><i class="{{.icon}}"></i></a></li>
{{ end }}
</ul>
<!-- /social icon -->

I also removed the email / phone number / address content from the footer. I would expect most people to contact me using one of the social media links on the front page. I've made the icons bigger and added Font Awesome support. Syntax highlighting is supported automatically in Hugo with Chroma, but you have to enable it. You no-longer need to use a shortcode, and TinaCMS enables you to select the tag when you enter the Markdown to start a code block.

After I got the site up and running, I made some tweaks to the theme:

Changing the heading font sizes.
Removing H3 headings from the blog entries.
Reducing the top background on the home page.
Removing the All button from the portfolio (tagging didn't work out of the box).
Removing the address footer.
Making all the hard-coded paths relative.
Adding a fav icon.

RSS support

Hugo has built-in RSS support. Just add /index.xml at whatever level you want the feed to be generated from. For example: https://andrewowen.net/blog/index.xml. However, it won't work if you have baseurl="/" in your config.toml file. You need to set it to your actual base URL.

Site-wide search

There are lots of options for adding search, but I think Victoria Drake's solution using Lunr is the best for a personal site: https://victoria.dev/blog/add-search-to-hugo-static-sites-with-lunr/. The only extra thing I had to do beyond the instructions was add the search form to the nav bar and style the results page to match the rest of the site.

Analytics

It's good to know who your audience is and which blog subjects attract the most interest. I recommend adding Google Analytics. After you've signed up for an account, all you need to do is provide your tracking ID in your config file and add a short code to the head.html partial. Or you can add it as a snippet to Netlify (I've found this to be the more reliable approach).

Custom 404 page

The last thing I added was a custom 404 page. After that, you're all set.

Image: Original by Toa Heftiba. I used one of Toa's images that came with the Kross theme on the original post, so I wanted to highlight her work again. The previous image wasn't exactly a forest. This isn't exactly an alpaca.

Migrating a Hugo site from Forestry to Tina

Andrew Owen — Thu, 05 Jan 2023 08:26:21 +0000

I launched the current version of my website a year ago. Having become a developer advocate in 2021, I didn't think a WordPress site that hadn't been updated in a decade would cut it any more. I wanted to do something a bit more modern. At my previous company, I'd built a developer portal on Hugo. The company ended up hosting the site itself, but I'd had discussions with Netlify and Forestry at the time. And I'd been using GitHub for my big open source projects for a long time. I picked a free Hugo starter theme from Themefisher that had built-in support for Netlify and Forestry. I spent a weekend on it: setting up the site structure, customizing the theme and adding content. I didn't have all the features at first (search, tags and RSS came later), but it was a huge step up from my old site.

According to Jamstack, Vercel's Next.js with its React templates has overtaken Hugo in popularity. And I'm not surprised. By most measures, JavaScript has never been out of the top 10 programming languages over the last two decades. And of 347 static site generators listed, 130 are written in JavaScript, 51 are written in Python and 26 are written in PHP. Hugo, and 16 others, are written in Go. But Hugo claims to be the fastest (you can check out those claims with PageSpeed) so I'm sticking with it. However, since 2019 the team behind Forestry have been developing the next iteration called TinaCMS. And on November 8, 2022 it came out of beta. Forestry is scheduled to be discontinued in late March 2023. Existing users will be offered a migration path to TinaCMS, a next generation headless CMS from the creators of Forestry. There are plans to share the migration tool in mid-January, but I decided to go ahead and do a manual migration.

I decided to take the opportunity to do some clean up and add a Hugo shortcode for audio (thanks to John Arrroyo for information on how to do that). I also finally got around to adding a custom 404 page. I created a new empty GitHub repository and connected it to a new staging site on Netlify. I checked out a local copy and brought in the content from the old version of the site. You can run TinaCMS locally, so after installation I was able to make changes before pushing to staging. I removed the Forestry config (although it wouldn't have done any harm to leave it in place). But I'll assume you want to simply add TinaCMS support to your existing repo.

Create the project

Register for a Tina account and log in.
From the Dashboard, navigate to Projects and click New Project.
Click Import Your Site, then click Authenticate with GitHub. If you're not already logged in to GitHub, do it now.
Select the repository for your Hugo site and enter the site URLs (your live site's URL and localhost with the port you want to use when you're doing local editing).
Click Create Project.

Set up your site schema

You'll need npm. If it's not already installed:

On macOS with Homebrew: brew install npm.
On Ubuntu: sudo apt install npm.
On Windows with Scoop: scoop install npm.

You also need hugo. You can install it the same way you installed npm.

Check out a local copy of your repo from GitHub.
In the root of the repo folder: npx @tinacms/cli@latest init.
When prompted to choose your package manager, select NPM.
Choose if you want to use Typescript.
When prompted for the public assets' storage folder name, enter static.
Start TinaCMS: npx tinacms dev -c "hugo server -D -p 3003".
Navigate to the admin page: http://localhost:3003/admin.

Because I used VScode, I created a tasks.json file in the .vscode folder to automate deploying TinaCMS locally:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "start TinaCMS",
      "type": "shell",
      "command": "npx tinacms dev -c \"hugo server -D\"",
      "group": {
        "kind": "build",
        "isDefault": true
      },
    },
  ]
}

By default, TinaCMS expects to find images in a media folder. Edit your TinaCMS config file to point to /static/images/ or wherever you keep your images. For example:

    media: {
      tina: {
        mediaRoot: "images",
        publicFolder: "static",
      },

Model your content

Before you can edit your content, you need to model it, based on the metadata you're using in the headers of your markdown files. In my case I'm using date, description, draft status, image, a tags list and a title. Besides the metadata, you also need to define the body text. My schema looks like this:

schema: {
      collections: [
        {
          name: "blog",
          format: "md",
          label: "Blog",
          path: "content/blog",
          defaultItem: () => {
            return {
              draft: true,
            }
          },
          fields: [
            {
              name: "draft",
              type: "boolean",
              label: "Draft",
              required: true,
            },

I have a single collection called Blog that corresponds to the folder where my posts go (content/blog). The draft field is a Boolean that determines if the post is displayed. You can set a default value for new posts in the defaultItem list so that new posts are all created as drafts.

            {
              name: "title",
              type: "string",
              label: "Title",
              isTitle: true,
              required: true,
            },
            {
              name: "date",
              type: "datetime",
              label: "Date",
            },
            {
              name: "description",
              type: "string",
              label: "Description",
            },
            {
              name: "image",
              type: "image",
              label: "Image",
            },

The title, date, description, and image are all fairly self-explanatory. You can set the required value to true to prevent saving a post that is missing a required field.

            {
              name: 'tags',
              type: 'string',
              label: 'Tags',
              list: true,
            },
            {
              name: 'body',
              type: 'rich-text',
              isBody: true,
              label: "Body",
              templates: [
                {
                  name: 'shortcode',
                  label: 'shortcode',
                  match: {
                    start: '{{',
                    end: '}}',
                  },
                  fields: [
                    {
                      // Be sure to call this field `text`
                      name: 'text',
                      label: 'Text',
                      type: 'string',
                      required: true,
                      isTitle: true,
                      ui: {
                        component: 'textarea',
                      },},],},],},],},],},

The tags are a set of text strings. For items like this, set the list value to true. Setting the type to rich-text enables the GUI editor for the body text. To be able to include Hugo shortcodes, you need to include the above template for them. In practice, I found the need to include the opening and closing angle brackets in the start and end items so that no space would be inserted between the curly brackets and the angle bracket (because a space kills the audio shortcode).

Fix your Markdown

This part is a headache, but until there's a Forestry to Tina migration tool, there's no getting around it. The big problem I encountered was that all my Markdown front matter was in TOML and, at the time of writing, TinaCMS only supports YAML. I used search and replace in VScode. But there is a better way.

Enable Tina Cloud in TinaCMS

Log into Tina Cloud.
Navigate to Overview and get a copy of your clientID.
Navigate to Tokens and click New Token.
Give the token a name. For example: Production Content Token.
Enter the Git branches the token has access to. For example: main. Then click Create Token.
Navigate to Tokens and get a copy of the token you just created.
Add your clientID and token to your config:

  export default defineConfig({
    branch,
    clientId: "",   // Get this from tina.io
    token: "",      // Get this from tina.io

In your netlify.toml file, add TinaCMS to your build command:

[build]
publish = "public"
command = "yarn tinacms build && hugo"

You can set this directly in your build settings on Netlify, but whatever is in the file will override whatever is on Netlify.

Now you can push your changes to GitHub. After Netlify deploys your build, you'll be able to work on your site in Tina Cloud.

Sync your media

Before you start editing. Go to Media Manager, click Sync and then click Sync Media. This copies media assets from the images folder in your designated branch (typically main) in your git repository to Tina Cloud's asset service. Now you can use those assets in your site with Tina Cloud.

Afterthought

I wrote this before TinaCMS published its Forestry migration tool and guide. I am indebted to Forestry CEO and co-founder Scott Gallant for sharing a draft of that guide with me when I got stuck. He was also super helpful on the TinaCMS Discord. Fun fact: Tina is named after the llama in “Napoleon Dynamite”.

Bulk updating documents with XSLT

Andrew Owen — Mon, 19 Dec 2022 19:47:23 +0000

XSLT (Extensible Stylesheet Language Transformations) is a language for transforming XML documents into other documents. I've mentioned it before in my post on creating release notes from Jira.

Now that every document under the sun is either stored in XML or can easily be converted to XML, XSLT provides a great way to perform batch processing on those documents.

To use XSLT, you need an XSLT processor. If you stick to version 1.0 of the specification, there are lots of choices. But if you want support for the latest version (3.0 at time of writing) your choices are RaptorXML (integrated with Altova's XMLSpy XML editor), or Saxon. Because my preferred XML editor is XMLmind's XML Editor (XXE), I use Saxon. Saxon and XXE both have free editions for personal use and open source projects.

As an aside, I've been using XXE for over a decade. If you have to write docs in an XML format like DocBook or DITA, it's the only editor I would recommend. It has a WYSIWYG interface, keyboard shortcuts for everything and you rarely have to deal with tags.

When you use the processor to apply your XSLT declarations to a file, it creates a new output, leaving the original files unchanged. Although it's a special purpose language, XSLT is Turing-complete, so you can do any kind of computation you might need with it.

Here's an example I created to convert <b> and <span class="BodyWord"> tags to <span class="Emphasis"> tags in MadCap Flare. To use it, save it as emphasis.xsl in the path with your files and then from the command line, enter:\
saxon -it:main -xsl:emphasis.xsl.

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:strip-space elements="*"/>

<xsl:template match="b">
<xsl:element name="span">
<xsl:attribute name="class">Emphasis</xsl:attribute>
<xsl:value-of select="@*|node()"/>
</xsl:element>
</xsl:template>

<xsl:template match="@class\[.='BodyWord'\]">
<xsl:attribute name="class">Emphasis</xsl:attribute>
</xsl:template>

<xsl:template match="@*|node()">
<xsl:copy><xsl:apply-templates select="@*|node()"/></xsl:copy>
</xsl:template>

<xsl:template name="main">
<xsl:for-each select="collection('.?select=*.htm;recurse=yes')">
<xsl:result-document href="output/{tokenize(document-uri(.))}">
<xsl:apply-templates select="."/>
</xsl:result-document>
</xsl:for-each>
</xsl:template>

</xsl:transform>

And here's a slightly more complex example that converts the contents of headings (<h1>) to sentence case in MadCap Flare. To use it, save it as sentence.xsl in the path with your files and then from the command line, enter:\
saxon -it:main -xsl:sentence.xsl.

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:transform xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="2.0">
<xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes" />
<xsl:strip-space elements="*"/>

<xsl:template match="h1">
<xsl:element name="h1">
<xsl:value-of select="substring(upper-case(.),1,1)"/>
<xsl:value-of select="substring(lower-case(.),2)"/>
</xsl:element>
</xsl:template>

<xsl:template match="@*|node()">
<xsl:copy><xsl:apply-templates select="@*|node()"/></xsl:copy>
</xsl:template>

<xsl:template name="main">
<xsl:for-each select="collection('.?select=*.htm;recurse=yes')">
<xsl:result-document href="output/{tokenize(document-uri(.))}">
<xsl:apply-templates select="."/>
</xsl:result-document>
</xsl:for-each>
</xsl:template>

</xsl:transform>

You can learn more about XSLT at w3schools.com.

Converting images with Image Magick

Andrew Owen — Mon, 19 Dec 2022 19:46:29 +0000

Photoshop is old. Really old. Well, in computer terms anyway. As of writing, it's currently on version 24.0. It was originally developed for the Mac in 1987 by Thomas Knoll, a PhD student at the University of Michigan, and his brother John, who worked for Industrial Light & Magic. Photoshop is a raster graphic (pixel image) editor. And because of its age, it has features for rendering 24-bit images (16 million colors) down to limited palettes, (which were a common feature of computers in the 1980s), that are missing from, or hard to use in, other image manipulation programs.

Another piece of software dating from 1987 is Image Magick. It was created by John Cristy at DuPont to convert 24-bit images to 8-bit images (256 colors). In 1990 Adobe became the publisher of Photoshop, and DuPont assigned the copyright in its tool to ImageMagick Studios LLC, a non-profit organization “dedicated to making software imaging solutions freely available”. So, if you can't afford over $200 a year for a Creative Cloud subscription just to convert 24-bit images to low color images, Image Magick is the obvious solution.

Some hardware that definitely doesn't date from 1987, but is designed as if it did, is the Chloe 280SE (the retro computer project that I will talk endlessly about if given the opportunity). The project is still in development, and at present its highest resolution graphics mode is incredibly complicated. It has a non-linear bitmap of 256×192 pixels. This is overlaid with a non-linear attribute map of 32×192 cells (each 8×1 pixels in size) that determine the foreground and background color. Each 8 pixel cell is rendered by paring a bitmap byte with an attribute byte. The attribute byte is divided into three parts. The highest two bits select one of four color look up tables (CLUTs). The middle three bits set one of 8 background colors, and the lowest three bits set one of 8 foreground colors. Background and foreground colors are defined independently. This gives 16 colors per CLUT, for a maximum of 64 colors on screen. However, it is not possible to combine colors from different CLUTs. The colors are chosen from a fixed palette of 256 colors based on half the 9-bit RGB palette (3-bits are used for red and green, but only two for blue). These are stored in GRB (MSB) order. A complete image consists of a 6K bitmap, 6K attribute map and 64 byte palette.

I said it was complicated. Fortunately, Edward Cree wrote an image conversion tool for Linux (scrplus) and Klaus Jahn made a Windows version (Image2ULAplus). These can both be used on a Mac with a suitable virtualization solution. However, the results can vary widely. This is the pre-conversion process if you have access to any version of Photoshop.

Scale the image you want to convert to 256×192 pixels.
Pattern dither the image to a uniform 128 colors.

Typically, Floyd-Steinberg or diffusion dither will give better results than pattern dither when reducing color depth. However, pattern dither seems to be the best method to reduce the visibility of the attribute cells. The palette has 8 levels for red and green, but only 4 for blue. Reducing the range to a uniform 128 colors translates to giving each color 5 levels in the input image. I don't know why, but that seems to give the best conversion results.

The conversion process after opening the input image in Image2ULAplus is:

Switch off:
- Reduce colors to 8-bit.
- Use dithering
- Create HAM256 screen
- Stretch to SCREEN$ size.
Switch on:
- Calculate palette
- CLUTs: 0, 1, 2, 3
- Create Timex screen
Set maximum colors to 64.
Click Render.
Save the image in .SCR format.

I've got a licensed copy of Photoshop CS6 on a 2010 MacBook Air. But I don't really want to have to get that out of storage every time I need to convert an image. So here's how to use Image Magick instead. First, you have to install it (unless it's pre-installed in your Linux distro): On Ubuntu, you'd use sudo apt install imagemagick and invoke it with the convert command. On macOS, the easiest way to install it is using Homebrew with brew install imagemagick. On Windows, you can download an executable.

With experimentation, I've found that the best setting for conversion to the Chloe 280SE is: magick input.jpg -resize 256x192 -ordered-dither o8x8,5,5,4 output.gif. This scales the image to 256×192 and then performs an ordered (pattern) dither on the image using an 8×8 pattern (which works well with 8×1 attributes). It applies a uniform palette with 5 levels of red and green, and 4 levels of blue (128 uniform colors in Photoshop gives approximately 5 levels for each color). The results vary, but for the test image that I've used in all my tutorials, Image Magick actually does a better job than Photoshop.

Now that's just one use case. Back in my days as a tech writer, I mainly used it for making .PNG images as small as possible for inclusion in online help files. And that's barely scratching the surface of what's possible with Image Magick. Fortunately, there's good documentation and an active community.

Fostering security awareness

Andrew Owen — Mon, 19 Dec 2022 19:44:25 +0000

Today's post is based on a presentation I gave at a security conference in the 2010s. It's a bit longer than what I would normally share, but I think it's still relevant, possibly more so than when it was originally written.

Introduction

This article is aimed at a non-technical audience. The aim is to raise awareness of the threats that often get overlooked when hardening software and hardware. In practice, you can only ever mitigate against security threats. For example, Bryan Dye once told the Wall Street Journal that Symantec, then the biggest antivirus vendor, was getting out of the antivirus business because the software stopped at most around 45 per cent of viruses. He said the money was no longer in “protect”, but instead in “detect and respond”.

Or consider the compromising of RSA’s SecurID. The theory at the time was that a nation state was trying to get access to secrets at a military aerospace vendor, but was blocked by the vendor’s use of SecurID. So instead they sent targeted email to RSA employees, which enabled them to breach the SecurID security and get what they were really after. RSA took a lot of criticism for how it responded to this attack. As an aside, that’s why it’s important to train your staff to recognize phishing emails.

Say you don’t have a disaster recovery plan. When the worst happens, even if you changed all your passwords, you know you’ll have to do it again after all the services you use have got their new keys in place. And you’ll still wonder if anyone managed to leave any snooping software on those services while the keys were compromised. The heartbleed OpenSSL vulnerability is an example of the worst happening without a disaster recovery plan.

As IT professionals, when we talk about security we’re mostly talking about confidentiality, integrity, and availability (CIA) of data. We don’t want confidential data leaving the organization, so we enforce a trusted device policy to ensure all BYO devices have their data encrypted and can be remotely wiped. We block the use of file sharing applications like DropBox that can lead to confidential data being stored in the public cloud. And we provide users with alternatives that keep the data within the corporate network, because users really like DropBox.

We lock down all the USB ports, because corporate spies have started sending out free mice with hidden malware to employees (I’m not making this up). And we use access controls to ensure people only have access to the information they need to do their job. We look after data integrity by making regular backups, and we do periodic restores to make sure those backups are working. And we make sure the data is available by doing system maintenance while the west coast of America is asleep. Ok, so outside of California your mileage may vary. So assuming you’ve done everything you should to secure your software and hardware, what have you missed? Well, I’ll get to that later.

Case study: the payment card industry

I’ve been interested in security since the late 1980s when I got my copy of Hugo Cornwall’s Hacker’s Handbook, where I discovered the existence of the Internet, or ARPAnet as it was known back then. Prior to joining the security business, I worked for a retail software company, where I discovered all sorts of frightening things about how card payments are processed. For instance, did you know that when chip and PIN payment was originally introduced in the UK that there was no encryption between the mobile radio units and the base stations? Thankfully, that’s now been resolved.

Or did you know that all the card payment transactions in high street stores used to be stored and sent unencrypted to the banks? Now the reason for this was that, as I’m sure you can imagine, there are a very large numbers of transactions throughout the day’s trading. Traditionally, these were sent to the bank at the end of the day for overnight processing. You’ll be glad to know that these were sent over a dedicated line rather than the public Internet.

But even so, they were sitting on the host system without any encryption. And the reason for that was that the overhead added by decrypting each transaction. Because they would all have to be individually encrypted and decrypted to work with the batch processing system at the banks. It would have added just enough delay to ensure that eventually the system wouldn’t be able to keep up with the number of transactions. Payments would be going into the queue faster than they could be processed.

Now, you may have heard of PCI DSS (the Payment Card Industry Data Security Standard). And what, among other things, that standard says, is that organizations have to restrict who has access to the folder with the card payments in it. And so already we’ve gone beyond the software and hardware, and we’ve got a security policy, the PCI DSS, and that policy is based at least in part on trust. If you want to read more about trust, I recommend Bruce Schneier’s book Liars & Outliers.

What I want to get across here, is that software and hardware are just part of the security solution. All retailers in the UK are supposed to be audited for compliance with PCI DSS. But according to Financial Fraud Action UK, card fraud losses in the UK for 2013 totaled £450.4 million. Now that sounds bad, but it to put it another way it’s equal to 7.4 pence for every £100 spent. And the things we have to consider here are the risk and, the cost of mitigating that risk.

The payment card industry wants to keep fraud down, but if putting in place a solution that eliminates fraud costs more than the cost of the fraud itself, then it will look for a cheaper solution. So actually, even before you secure the box, you really need a security policy. Because if there’s nothing of value in the box, then you don’t really need it to be that secure. But if what’s in the box is the most valuable thing you have, then you really need to be able to deal with a situation where all of your security measures failed.

The importance of a security policy

So, although that was a bit of a roundabout way to get to my point, what I’m advocating is that organizations need a security policy. And vendors of security solutions, need to help their customers to think about security in this way. So what makes a good security policy? Well first of all you need to have someone with the responsibility for the policy, the chief security officer (CSO). And one of their most important responsibilities is to keep the policy under review, because the environment is changing all the time, and a static policy can’t address that.

So how do you come up with a good security policy? Well, there are various things you need to take into account. But primarily it’s about working out the risk: How likely is it that someone will walk out of this facility with all this government data on a USB pen drive? And the cost: what will be the effect if this confidential information about everyone we’re spying on gets into the public domain?

So for each risk, you work out the associated cost, and then you come up with a solution proportionate to the risk. Let’s go back to the early days of hacking. I’m not sure anyone ever calculated the risk of hackers going dumpster diving for telephone engineer manuals. But I’m reasonably confident that the cost of shredding all those manuals set against the risk of someone typing the whole thing into a computer and uploading it to a bulletin board system was fairly high. Now this is in the days before cheap scanners, good optical character recognition and widespread access to the Internet, which is why everyone now securely disposes of confidential documents, don’t they?

Now, in the Snowden case there were a couple of things that surprised me. First, that the NSA wasn’t using mandatory access control (MAC). Or in other words, they weren’t using a trusted computing solution. They were using the same operating systems as the rest of us. I think partly that can be explained by the fact that it’s expensive to get support for trusted operating systems, because almost no-one besides governments use them. And often the applications that governments want to run aren’t available on those platforms, so the cost of using them may exceed their benefit in mitigating risk. But the other thing that surprised me is the practice of password sharing.

And that brings me to the main vulnerability you face if your hardware and software are secure. Your users. Kevin Mitnick, I’m assuming you’ve heard of him, if not look him up. He asserts, and I don’t disagree with him, that humans are the weakest link in security. In fact, I recommend his book The Art of Deception if you want to know exactly how predictable and easy to manipulate people are.

So let’s look at the password sharing issue. If you put up a big enough road block for your users to getting work done, they will find a detour around it. Is it easier to tell someone your password than jump through hoops to get that one file they need? Many companies have a policy that passwords need to contain at least eight alphanumeric characters, both upper and lower case letters, at least one number, and at least one special character. It also can’t be one of the previous three passwords. So what do users do? They pick dictionary words with substitutions.

And then users have to change their password every six months, or quarterly if it’s an administrative password. This leads to one of two things. They write the passwords down. Or they repeatedly change their password until they cycle back to their original password. It’s pretty easy to get a valid corporate username. They’re in all of our email addresses. If you can actually get on to a corporate site and physically connect to the network, you can just keep trying to connect until you brute force the password.

So how do you get on site? Well, this touches on the other main vulnerability, physical security. Many companies use employee badges for building access, and various areas are restricted to specific groups of employees. They have a policy of not holding the door open for people employees don’t recognize. Unfortunately, it is in most people’s nature to be helpful. If I smile at someone as they go through a door, and I’m dressed appropriately, they’re less likely to question if they should have just let me follow them.

Mitnick’s book is full of these kinds of social engineering techniques. But actually the easiest way to get on site at some companies is to sign up for a training course. You might have read in the news earlier this year about the gang of crooks who stole £1.25 million by going into bank branches and attaching KVM (that’s keyboard/video/mouse) switches. Reports haven’t detailed how they got into the building, but it’s safe to assume it was low tech, and they didn’t break in.

So you need to educate staff about threats. Phishing email, social engineering, not picking up USB pen drives that you find lying around and connecting them to your corporate PC. I'm not even going to cover BYOD. That’s “Bring Your Own Device”, although some have called it “Bring Your Own Disaster” because of the additional risks and management headaches it entails. I will say that the mitigation is to require BYO devices to meet a minimum level of protection: a secure password, encrypted storage, the ability to do a remote wipe. But basically, the message is that it’s all very well having a security policy, but it isn’t much use if your staff don’t know about it.

Once you’ve got a policy in place, then you need to stress test it. This is where the “red team” comes in. This can be an internal group, or an externally hired group, whose job is to attempt to penetrate your security, for instance by leaving USB pen drives lying around or sending test phishing emails. Penetration testing needs to be conducted on a regular basis, the frequency of which will depend on the risk and cost analysis, and the security policy updated following the findings.

But let’s come back to physical security. In the aftermath of hurricane Sandy, it seems fairly obvious to state that if you’re doing offsite backup to multiple data centers that at the very least you don’t want them co-located in the same flood plain. Of course since then everyone has looked at where their critical services are and ensured sufficient redundancy to deal with a major disaster. Haven’t they?

Assuming you’ve got the location sorted out, and you’re outside the 500-year flood plain, you’re going to want to consider alternate power sources, given the increasing demands being placed on the power grid. And when you’ve got your failover power supply in place, it helps to test that it actually works. Your backups are only as good as your ability to recover from those backups, so it’s important to perform regular testing to make sure that’s the case. Physical access can be controlled by physical barriers, locks, guards, but it can also be monitored by video cameras. Servers get hot, so you need to consider fire suppression systems. Ideally, ones that will leave the data in a recoverable state.

Summary

I’ve barely scratched the surface, but hopefully I’ve given you some things to think about. So to sum up: you want a security policy that is under continual review and covers:

Human nature
Disaster recovery
Physical location
Penetration testing
Social engineering

And really the most important thing is to raise security awareness.

Managing packages on macOS with Homebrew

Andrew Owen — Mon, 19 Dec 2022 19:43:09 +0000

If you're a Linux user, or you read my post on Scoop, you'll be familiar with package managers. They aim to simplify installing, upgrading, configuring and removing software. A key feature is dependency management: if a package requires software that isn't already installed, the package manager can install it.

Apple's solution to managing installed software on macOS, and its other operating systems, is the App Store application. Typically, end-user applications don't have dependencies. However, command line tools often do, particularly if they originated on Linux. Another issue is that macOS typically ships with older versions of common tools, if it includes them at all.

The most popular third-party package manager on macOS used to be MacPorts, and it's still a valid choice. But it has been replaced in popularity by Homebrew, which is based on Git and Ruby.

If you haven't already installed the Xcode command line tools, you might want to do that first. From the command line, enter: xcode-select --install.

To install Homebrew, open the Terminal and enter /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)".

To update the package list, enter brew update. To upgrade all packages, enter brew upgrade. Now you're ready to add whatever packages you like.

Homebrew includes an extension called Cask. You install command line apps with brew install <app name>. For graphical apps use brew casks install <app name>.

Having been using an M1 (ARM CPU) Mac for over a year now, I'm pleased to report that I haven't run into any problems with Homebrew.

Managing packages on Windows with Scoop

Andrew Owen — Mon, 19 Dec 2022 19:41:48 +0000

If you have even a passing familiarity with Linux, you're probably aware of the concept of package management. The goal is to simplify the installing, upgrading, configuring and removing software. One of the key features is dependency management: if your software depends on some other software that isn't already installed, the package manager will give you the option to install it.

Microsoft provides a Windows Package Manager, known as winget. However, at the time of writing it's only been around for a couple of years and I got a hash error when I tried to use it to install Google Chrome.

A popular third-party option is Chocolatey, but it depends on NuGet which is a package manager in its own right. I prefer to use Scoop. The latest install instructions are on the website, but I'll go through how I installed it on Windows 11.

Open PowerShell.
[Optional] Enter Set-ExecutionPolicy RemoteSigned -Scope CurrentUser. This enables a remote script to run the first time.
Enter Invoke-WebRequest get.scoop.sh | Invoke-Expression. This installs Scoop.
Enter scoop bucket add extras. The extras bucket includes software that doesn't match the main criteria for inclusion.
Enter scoop bucket add nonportable. The nonportable bucket includes software that must be installed to a specific location.
Enter scoop install git. Even if you're not using Git, Scoop uses it to perform updates.
Enter scoop update. This will update the buckets you added.

Now you're ready to add whatever packages you like. Here are some examples:

Docker: scoop install docker
GitHub Desktop: scoop install github
Google Chrome: scoop install googlechrome
Helm: scoop install helm
Notepad++: scoop install notepadplusplus
SQL Server Management Studio: scoop install sql-server-management-studio-np
Visual Studio Code: scoop install vscode

You can verify the software has been correctly installed by viewing the C:\Users\<user>\scoop\apps folder.

To update all your installed packages to the latest version, enter scoop update all. When I was creating a VM for developer training with numerous prerequisite software packages, I found it useful to put this command in a startup script.

Having recently revisited the VM setup for Azure Labs, I've discovered a few gotchas. Here's how I got it working.

Create an Azure Labs template VM from the Windows 10 N Enterprise with Visual Studio 2019 Community image in the marketplace.
Fire up the VM, then from the standard shell enter: runas /trustlevel:0x20000 powershell.exe. The Scoop install script won't run with admin privileges. The ExecutionPolicy directive doesn't work when the shell is run as administrator. And the VM template doesn't give you the option to start a shell without admin privileges.
From PoweShell, enter: iwr get.scoop.sh -useb | iex. The -useb directive is required with Windows N (something to do with Internet Explorer).
Close both shells and then open PoweShell and enter scoop bucket add main.
Enter scoop update.

You can now carry on as normal. You don't need to install Git, because it's included with the VS 2019 Community image.

Image: Original by Ian Dooley.

Using GitHub Actions to automatically unpack a zip archive

Andrew Owen — Mon, 19 Dec 2022 19:39:05 +0000

I was recently working with some software that could push a zip archive of content to a Git repository. However, what I really wanted was for the contents of the archive to be pushed to the repository. So I created a GitHub Action to do that for me. I've covered the format of GitHub Actions before. But to recap:

name is what gets displayed in the actions list.
on sets the triggers:
- push and pull trigger the script on push and pull requests. If you don't specify branches, they default to main.
- paths specifies the paths to match. In this case, the script will trigger when a zip file is pushed.
- worklflow_dispatch enables you to manually trigger the script from the actions list.
jobs defines one or more named tasks, in this example unzip.
runs-on specifies the VM environment. If you can use Ubuntu, it's the cheapest option with hosted runners.
steps can be used to invoke actions such as checkout (which fetches a copy of the repository to the VM) and to execute shell commands with name: run.
run with a pipe character ( | ) executes a multi-line script. Without it, a single line is executed.

And here's the action I created:

name: extract a zip file

    on:
      push:
        paths:
        - '**.zip'
      workflow_dispatch:

    jobs:
      unzip:
        runs-on: ubuntu-latest

        steps:
          - uses: actions/checkout@v2

          - name:
            run: |
              rm -r css
              rm -r en
              rm -r fonts
              rm -r js
              filename=$(basename -s .zip *.zip)
              unzip *.zip
              rm *.zip
              mv $filename temp
              mv temp/out/* .
              rm -r temp
              git config user.name github-actions
              git config user.email github-actions@github.com
              git add .
              git commit -m "unzip"
              git push origin main

The script is written for the Linux command line. Let's break it down.

rm -r css

    rm -r en
    rm -r fonts
    rm -r js

The idea here is that the contents of the zip file should replace what is already in that particular branch of the repository. You might want to call the branch uploads. The checkout action has already been run, but it's a good idea to clear out any known folders. The -r tag makes the action recursive.

filename=$(basename -s .zip *.zip)

    unzip *.zip
    rm .zip
    mv $filename temp

This script assumes that we don't know the name of the zip file, but that there is only one file. It will determine the name, unzip the file to the root, remove the zip file and rename the folder containing the zip to temp.

mv temp/out/ .

    rm -r temp

In this example, the contents of the zip file are two folders deep (in the out folder). This moves the contents from the nested folder to the root, and then removes the temp folder and its contents (the empty out folder). The dot (.) represents the current working directory (where the repo was checked out on the VM).

git config user.name github-actions

    git config user.email github-actions@github.com
    git add .
    git commit -m "unzip"
    git push origin main

This part of the script pushes the changes back to the repository.

Image: Detail from The Unarchiver zip file icon. I looked for an appropriate unzip image with a Creative Commons license, but the results were not safe for work.

Manage the contents of a Readme.io docs site from your own Git repository

Andrew Owen — Mon, 22 Aug 2022 14:49:46 +0000

ReadMe.io is a popular user docs site. It has a Markdown editor, theme builder and Swagger / OpenAPI file import. It's fast and responsive, and it looks nice. But the last time I checked, all your content goes in a bucket that you don't have direct access to.

One upshot of this is that if you want to make changes across documents, such as changing a product name, you have to edit documents individually. It would be much nicer if you could directly access the repository in VS Code and make site-wide changes.

Even though this isn't supported, there are a couple of features in Readme.io that can enable you to take control of your data. First, you can export your content. Second, Readme.io has an API. There are some tasks that you'll still have to perform in the web interface. But for the most part, you can keep your data in a Git repository, make local changes, and use the API to push your changes to your live site.

ReadMe.io APIs

When I was getting started, I found it useful to use Postman to query the Readme.io site. You can save this collection as readme.postman_collection.json:

{
"info": {
"_postman_id": "<!--insert your postman ID here-->",
"name": "readme.io",
"schema": "[https://schema.getpostman.com/json/collection/v2.1.0/collection.json](https://schema.getpostman.com/json/collection/v2.1.0/collection.json "https://schema.getpostman.com/json/collection/v2.1.0/collection.json")"
},
"item": \[
{
"name": "Get doc",
"request": {
"method": "GET",
"header": \[\],
"url": {
"raw": "[https://dash.readme.com/api/v1/docs/](https://dash.readme.com/api/v1/docs/ "https://dash.readme.com/api/v1/docs/"){{slug}}",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs",
"{{slug}}"
\]
}
},
"response": \[\]
},
{
"name": "Get category ID",
"request": {
"method": "GET",
"header": \[\],
"url": {
"raw": "[https://dash.readme.com/api/v1/docs/get-category-id](https://dash.readme.com/api/v1/docs/get-category-id "https://dash.readme.com/api/v1/docs/get-category-id")",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs",
"get-category-id"
\]
}
},
"response": \[\]
},
{
"name": "Update doc",
"request": {
"method": "PUT",
"header": \[\],
"body": {
"mode": "raw",
"raw": "{\\n \\"title\\": \\"{{title}}\\",\\n \\"excerpt\\": \\"{{excerpt}}\\",\\n \\"category\\": \\"{{category}}\\",\\n \\"hidden\\": {{hidden}},\\n \\"body\\": \\"{{body}}\\"\\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "[https://dash.readme.com/api/v1/docs/](https://dash.readme.com/api/v1/docs/ "https://dash.readme.com/api/v1/docs/"){{slug}}",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs",
"{{slug}}"
\]
}
},
"response": \[\]
},
{
"name": "Delete doc",
"request": {
"method": "DELETE",
"header": \[\],
"url": {
"raw": "[https://dash.readme.com/api/v1/docs/](https://dash.readme.com/api/v1/docs/ "https://dash.readme.com/api/v1/docs/"){{slug}}",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs",
"{{slug}}"
\]
}
},
"response": \[\]
},
{
"name": "Create doc",
"request": {
"method": "POST",
"header": \[\],
"body": {
"mode": "raw",
"raw": "{\\n \\"title\\": \\"{{title}}\\",\\n \\"excerpt\\": \\"{{excerpt}}\\",\\n \\"category\\": \\"{{category}}\\",\\n \\"hidden\\": true,\\n \\"body\\": \\"{{body}}\\"\\n}",
"options": {
"raw": {
"language": "json"
}
}
},
"url": {
"raw": "[https://dash.readme.com/api/v1/docs](https://dash.readme.com/api/v1/docs "https://dash.readme.com/api/v1/docs")",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs"
\]
}
},
"response": \[\]
},
{
"name": "Search docs",
"request": {
"method": "POST",
"header": \[\],
"url": {
"raw": "[https://dash.readme.com/api/v1/docs/search?search=sphinx](https://dash.readme.com/api/v1/docs/search?search=sphinx "https://dash.readme.com/api/v1/docs/search?search=sphinx")",
"protocol": "https",
"host": \[
"dash",
"readme",
"com"
\],
"path": \[
"api",
"v1",
"docs",
"search"
\],
"query": \[
{
"key": "search",
"value": "sphinx"
}
\]
}
},
"response": \[\]
}
\],
"auth": {
"type": "basic",
"basic": \[
{
"key": "username",
"value": "{{apiKey}}",
"type": "string"
}
\]
},
"event": \[
{
"listen": "prerequest",
"script": {
"type": "text/javascript",
"exec": \[
""
\]
}
},
{
"listen": "test",
"script": {
"type": "text/javascript",
"exec": \[
""
\]
}
}
\],
"variable": \[
{
"key": "category\\n",
"value": "<!--get the category ID from the web interface-->"
},
{
"key": "slug",
"value": "sandbox"
},
{
"key": "title",
"value": "Sandbox"
},
{
"key": "hidden",
"value": "true"
},
{
"key": "body",
"value": "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."
}
\]
}

Import the collection to Postman and use Basic Auth with the API key as the username and an empty password.

Create doc

This endpoint requires these attributes:

title – the name as you want it displayed on the site.
category – UUID
hidden – false
body – JSON string

The hidden attribute is always set to true. The final publishing stage after reviewing the content as it will appear is to set this to false.

Unless I missed something, when I originally came up with this solution, it was only possible to create categories on Readme.io. But there is an API for that. I don't currently have an active Readme.io subscription to test, so I'll leave it to you to add those endpoints to the Postman collection.

Articles can be posted under existing articles. In this case, you should do a GET on the parent article to get the category ID.

Convert Markdown to JSON string with JQ

Readme.io stores articles in Markdown format (with a YAML header). But the API requires the data in JSON format. Fortunately, there's a command line tool call JQ that can encapsulate the Markdown in a JSON string. This will form the body attribute. From the CLI, enter: jq -R -s . < filename.md > filename.txt.

Pushing changes in your repository to the live site

I wrote a Bash script to automatically updates all the .md files in a repository using the PUT method. It's a bit of a hack. It requires the $AUTH token to be defined, and it will only work on files where the header is exactly eight lines.

---
title: "article title"
category: "category UUID"
excerpt: "article description"
hidden: true
createdAt: "2022-05-19T00:00:00.000Z"
updatedAt: "2022-05-19T10:00:00.001Z"
---

By default, articles download from Readme.io include slug metadata but no category metadata. Since the slug is derived from the filename, you need to replace the slug metadata with the category metadata and give it the appropriate UUID for the category heading it appears under.

Note: If you don't include an excerpt definition, you must ensure the body text starts on line 9 of the file.

Here's the Bash script. Save it as md2json.sh in the v1.0 folder of your exported site.

export AUTH="<!--your AUTH token-->"
for subdir in *; do
test -d "$subdir" || continue
echo $subdir
cd "$subdir"
for f in _.md; do
export SLUG=${f%%._}
sed '1,8d' $f > "$SLUG.tmp"
sed '6,$d' $f > "$SLUG.yml"
jq -R -s . < "$SLUG.tmp" > "$SLUG.bdy"
rm "$SLUG.tmp"
printf "body: " >> "$SLUG.yml"
cat "$SLUG.bdy" >> "$SLUG.yml"
yq < "$SLUG.yml" > "$SLUG.json"
rm "$SLUG.bdy"
rm "$SLUG.yml"
curl -X PUT -H 'Content-Type: application/json'   
\-H "Authorization: Basic $AUTH"   
\-d "$(<$SLUG.json)"   
https://dash.readme.com/api/v1/docs/$SLUG
rm "$SLUG.json"
done
cd ..
done

Managing images

The last part of the puzzle is managing images. At the time of writing, I don't know a way to upload images using the API. My suggestion is that you keep a hidden article on the website and add images to it using the web interface. Readme.io will rename the file and assign it an ID. You can then extract this information from the article. I would then add a copy of the image with this modified filename to your repository (in case you ever want to migrate to a different docs solution). This will also enable you to preview your content locally before pushing it to the live site.