A static site generator using Pandoc

Why a static site?

I’ve been running a static site for a long time now. The obvious advantages are:

• Easy – one just needs to write the content in HTML, and push it to the server.
• Fast – static sites are much faster to serve, and easier for intermediaries to cache.
• Secure – there is no dynamic content running on the web-server, therefore limiting the attack surface significantly.

This was all working fine until one fine day my – admittedly unbalanced, so my fault really! – RAID-1 array died due to one of the (smaller) disks giving up. Because it was unbalanced (different sized disks) it took down the whole server.

Lesson learned! And an opportunity presented itself to rebuild the server in a better (hopefully!) way.

This was an opportunity to address a long-standing wish: the problem with my current setup was the “easy” bit - it wasn’t really either quick or easy writing HTML. Ok, there are loads of reasonably good HTML editors that make life slightly easier. However, the GitHub style markdown rendered transparently to HTML is very appealing – just type quickly in markdown, perhaps run a generator process, produce HTML, publish it, and you’re done.

So that started me down the path of a markdown content based static site.

How does it work

I looked around a bit – there are a fair few static site generators like Pelican or Jekyll or Hugo… You get the gist! They’re all good - but I wanted something simpler.

Enter Pandoc. Pandoc is a general purpose, cross-document format conversion tool. It supports a vast array of input formats, and is able to convert to a vast array of output formats. This is great - because if I happen to have some documents in Libre Office format, I can convert even those into markdown, and put it on the site!

And of course, it’s really very quick to put together a post (like this one!) in markdown – much simpler and faster than doing the same thing in HTML! Almost like a WIKI!

🛈 A note on the choice of markdown and Pandoc

The combination of Pandoc and markdown delivers a very powerful and flexible solution for document creation and rendering in various output formats, including PDF and HTML. It’s really very easy to write up a post in markdown, and using Pandoc, it is possible to generate quite a few output formats – the templates Pandoc comes with, gives you a lot of flexibility in output formatting and layout: from GitHub styled HTML, to elaborate PDFs for academic reports.

For example, you could use latex directly embedded in the content, to write complex mathematical equations: $$\begin{equation} \lim_{x \to 0} \frac{d^2y}{dx^2} \end{equation}$$ Or $$\begin{equation} \int_{a}^\infty \frac{x^2}{dx} \end{equation}$$ When rendered with the --mathml option, most browsers will display the correct mathematical format in HTML. For PDF generation the standard latex mechanism should work – you should not need the --mathml option.

An excellent latex cheat-sheet can be found here.

This makes it ideal for usage in a light-weight web-site generator from markdown input, as well as easily reusable for more complex PDF layouts.

Underpinnings

Having decided on Pandoc, it was time to read up on it and learn how to get the basic conversion of markdown to HTML working. This article turned out to be an excellent, step-by-step guide to get a basic conversion working. It also has a GitHub site with templates for the demo.

And of course, there is the Pandoc manual. Plus the Pandoc GitHub with lots of useful stuff, including templates and problems reported by other people – like issue-3778 – that become useful references.

Armed with all this information, I set out to get a generator working.

The generator project

Pandoc must be invoked per input file, with appropriate options, to get nice-looking output. The first part of nice-looking HTML is a style-sheet. For this, I simply followed this guide and used Pico-css – the default style is more to my tastes than the lime theme used in the demo. So that is what I went with.

One page doesn’t a site make! So I needed something to run in a loop over all files in a source directory.

Although the demo site linked above has a loop written in power-shell, I don’t use it! And I didn’t want to convert it to bash.

I decide to write a simple driver program in Java, that handled options I cared about for this project. This class lives in my private git repository.

Now I was ready to start my web-site project.

I decide to go with maven since I’m quite comfortable with it, and have seen similar Pandoc generator projects automated with make.

The maven project structure iis simple:

• markdown files live in the src/site directory
• css styles live in the src/css directory
• Header/footer HTML inclusion templates live under the src/templates directory.

The driver bundle is uncompressed under the driver directory, which creates a driver/utils-1.0 directory, under which the jar that contains the main driver class lives.

We now have everything we need to run the driver and generate our site!

Putting it all together

The maven project descriptor just needs two stages setup:

• validate – this stage copies resources like images and style-sheets into the directory structure of the actual web-site, preparing it for the copy to the web-server. The resources plugin performs these copies.
• exec – the exec plugin then provides two executions:
  • A generate execution that generates the web-site.
  • A rsync execution that pushes up changed files to the server staging directory from where it can be picked up by the web-server.

Generating the web-site now just requires the following maven command:

mvn clean validate exec:exec@generate -f pom.xml

And promoting the web-site to the server can be done by running:

mvn exec:exec@rsync -f pom.xml

Or, it can also be done in one step with the following command:

mvn clean validate exec:exec@generate exec:exec@rsync -f pom.xml

Getting the show on the road

At this point, we have:

• Generated the web-site.
• Uploaded it to a staging directory on our server.

Now, all that remains to be done is a sync step into the web-server root, so that the web-server can pick up changes.

Currently, I simply run rsync manually to push the changed files into the web-server root:

sudo rsync -av /staging/web-site/html/ /web-root/html/

Obviously – not automated, so not a great solution.

Systemd provides a built-in mechanism to watch a directory for changes. We will use this mechanism to watch the staging directory and run the rsync into the web-server root.