Behind the scenes: an in-depth look

Greetings everyone.

I have thought for a long of making a post like this, where I would talk about how the book is made, the tools(and the shameless hacks) that are behind every page.

Let's take a deep dive into the madness that is my workspace!

OS and Editor

This is more of an introduction than anything else, but we need to start from somewhere, don't we?

My main daily driver (both in my day job and in writing this book) is Linux, the distribution is Artix Linux (a variation of ArchLinux that gets rid of SystemD) using a completely customized desktop environment based on tools taken from LXDE, XFCE and the OpenBox Window Manager.

My main editor is NeoVim (also known as NVim), an aggressive refactoring of the Vim editor, filled with custom hotkeys, custom function and plugins.

Base Toolkit

The book is written using Pandoc Markdown, a variation on the Markdown language that is used in the Pandoc application. Pandoc has been extensively used as a markup conversion tool, but its markdown has shown to be readable and having enough features to write a book. Plus the pure-text format works well with my favourite version control system: Git.

To output the PDF format I use XeLaTeX, mainly for its advanced unicode support: it is part of the TeXLive package. To create the PDF version of the ebook, Pandoc will convert the Markdown files into a single LaTeX file, which in turn will be converted to PDF by XeLaTeX.

Instead of using the standard Pandoc LaTeX template, I went through a painstaking process of editing and adapting to have a somewhat better (and more book-looking) experience.

Pandoc allows me to get an EPub output too, but there is a small problem: the output from Pandoc is using MathML, which is not supported by all E-Readers (including mine), thus a new tool enters the toolchain: GladTeX.

GladTeX's job is to take all the math blocks in the book and transform them into SVG images, preventing possible rendering errors and incompatibilities.

So to have an EPub output Pandoc must output a JSON file (containing the Abstract Syntax Tree) that is then fed into GladTex, which will generate the SVG images and edit the AST accordingly. The edited AST (still in JSON format) will be then passed through Pandoc again, which will take care of the creation of an EPub file.

The whole toolkit is glued together by GNU Make: I make use of a MakeFile to make the various formats and editions (I'm making overuse of the "make" verb on purpose). This makes everything as easy as typing "make" to create all the available formats and editions.

Filters

At the heart of some of the additional features of this ebook lie a bunch of Lua filters for Pandoc: these add some features that are not natively present in Pandoc (but may be native to LaTeX) and extends Pandoc Markdown.

• boxes.lua - This script takes care of converting a text representation of a box into a proper "tip", "pitfall", "trivia" or "note" box:

• epigraphs.lua - This one takes care of quote at the beginning of a chapter, either by building an HTML div for the EPub format or by building an epigraph environment for the PDF output.
• placeholder.lua - This one is dedicated to building the sentence that comes out when a chapter is not considered completed yet, this way I can just write "{{placeholder}}" and the sentence will pop out automatically.
• pagebreak.lua - This script is dedicated to forcing a page break, usually used at the beginning of a chapter, to make it start on a new page.
• symbols.lua - A script that replaces some placeholders with some symbols, for instance "{{donprod}}" (freemium product or accepts donations) gets turned into a bold and orange "[D]", without having to set the color and font weight every time.
• dynamic_code.lua - Probably the most complex script in the folder. This one takes care of switching out code listings according to the edition that is being built: this way I can keep only one text and have as many coding languages as I want. The sources are taken from the dynamic_listings folder.
• metavars.lua and filter_helper.lua - These scripts work together to orchestrate the activity of all the other scripts, this way I don't need to edit the Makefile every time I add a new filter (although this hasn't happened in a very long time).

Scripts

There are only two optional scripts that I use to gauge the status and building the ebooks.

The first (and oldest) is "listing_status.py" a Python3 script that compares the listings in the Pseudocode Edition with all other editions and outputs a graphical summary of the status of all editions, plus a list of missing listings for each one of them:

The other one is "build.sh" which I called the "Mis-Guided Building System", it's an all-in-one system that does some additional checks (using listing_status.py, for instance) to check if all editions can be built without errors:

Other tools

The following tools are not absolutely necessary to get an output from the book build system, but they are used in creating the resources, charts, drawings and diagrams that are in the book.

• GNU Dia - A diagram drawing toolkit, it is pretty old and it's being slowly phased out of this project in favor of more up-to-date alternatives;
• Inkscape - A great open-source vector graphics editor, used to draw and edit some of the SVG images;
• Gimp - Used for manipulating raster images, even if vector formats are preferred sometimes PNG images are needed;
• GraphViz - Used to create simple diagrams from text definitions, this is used to update all resources in the *.dot format and to make SVG images from them;
• PlantUML - Used to create UML diagrams from text definitions, this is used to update all resources in the *.puml format and to create SVG images from them;

Templates

In order to be able to output PDF and EPub formats, Pandoc needs to start from a template. The templates are in written in two (and a half) main languages: LaTeX (for PDF) and xHTML + CSS (for EPub).

Both are a mix of the original markup language and some placeholders (usually marked by dollar signs) that are interpreted by Pandoc.

I won't get deeper than this, because it would get really boring really quickly (provided so far things haven't been boring).

Conclusion

This marks the end of this long form post (looks like I took "long form" way too literally!).

If you want to take a look at the source code of the Book, feel free to check out its repositories:

• On GitHub at: https://github.com/2DGD-F0TH/2DGD_F0TH
• On GitLab at: https://gitlab.com/Penaz/2dgd_f0th

Thank you all for reading and for your support, stay safe and take care!

Penaz