5 Pull it Together

5.1 Automation with make

Because we will probably run commands like this a lot, it’s convenient to automate them a little bit, and to add some extra bells and whistles to accommodate things we will routinely add to files, such as author information and other metadata, together with the ability to process bibliographic information and cross-references. These are handled by pandoc by turning on various switches in the command, and by ensuring a couple of external “filters” are present to manage the bibliographies and cross-references. Rather than type long commands out repeatedly, we will automate the process. This kind of automation is especially useful when our final output file might have a number of prerequisites before it can be properly produced, and we would like the computer to be a little bit smart about what needs to be re-processed and under what conditions. That way, for example, if a Figure has not changed we will not re-run the (possibly time-consuming) R script to create it again, unless we have to.

We manage this process using a tool called make. Inside our project folder we put a plain-text Makefile that contains some rules governing how to produce a target file that might have a number of prerequisites. In this case, a PDF or HTML file is the target, and the various figures and data tables are the prerequisites—if the code that produces the prerequisites changes, the final document will change too. Make starts from the final document and works backwards along the chain of prerequisites, re-compiling or re-creating them as needed. It’s a powerful tool. For a good basic introduction, take a look at Karl Broman’s “Minimal Make”. (Incidentally, Karl Broman has a number of tutorials and guides providing accurate and concise tours of many of the tools and topics described here, including getting started with reproducible research, using git and GitHub, and working with knitr.)

Following Karl Broman’s example, let’s imagine that you have a paper, paper.md written in Markdown, that incorporates references to a figure, fig1.pdf generated by an R script, fig1.r. You could of course have an .Rmd file that produces the output, but there are situations where that isn’t ideal. The end-point or target is the full article in PDF form. When the text of the paper changes in paper.md, then paper.pdf will have to be re-created. In the same way, when we change the content of fig1.r then fig1.pdf will need to be updated, and thus also paper.pdf will need to be re-created as well. Using make we can take care of the process.

Here is what a basic Makefile for our example would look like:

## Read as "mypaper.pdf depends on mypaper.md and fig1.pdf"
mypaper.pdf: mypaper.md fig1.pdf 
    pandoc mypaper.md -o mypaper.pdf

## Read as "fig1.pdf depends on fig1.r"
fig1.pdf: fig1.r
    R CMD BATCH fig1.r

The big gotcha for Makefiles is that for no good reason they use the tab key rather than spaces to indent the commands associated with rules. If you use spaces, make will not work. With the Makefile in Listing 4, typing make at the command line will have make check the state of the final target (makefile.pdf) and all its dependencies. If the target isn’t present, make will create it in accordance with the rules specified. If the target is present, make will check to see if any of its prerequisites have changed since it was last created. If so, it will recreate the file. The chain of prerequisites propagates backwards, so that if you change fig1.r, then make will notice and re-run it to create fig1.pdf before running the commands in the rule to create mypaper.pdf. You can also choose to just invoke single rules from the makefile, e.g. by typing make fig1.pdf instead of make at the command line. This will evaluate just the fig1.pdf rule and any associated prerequisites.

For a simple example like this, make is mostly a minor convenience, saving you the trouble of typing a sequence of commands over and over to create your paper. However, it becomes very useful once you have projects with many documents and dependencies—for example, a dissertation consisting of separate chapters, each of which contains figures and tables, which in turn depend on various R scripts to set up and clean data. In those cases, make becomes a very powerful and helpful way of ensuring your final output really is up to date.

To deal with more complex projects and chains of prerequisites, make can make use of a number of variables to save you from (for example) typing out the name of every figure-x.pdf in your directory.

The Makefile in the sample md-starter project will convert any markdown files in the working directory to HTML, .tex, .pdf, or .docx files as requested. Typing make html at the command line will produce .html files from any .md files in the directory, for example. The PDF output (from make pdf) will look like this article, more or less. The different pieces of the Makefile define a few variables that specify the relationship between the different file types. In essence the rules say, for example, that all the PDF files in the directory depend on changes to an .md file with the same name; that the same is true of the HTML files in the directory, and so on. Then the show the pandoc commands that generate the output files from the markdown input. The Makefile itself is shown in Listing Listing 5 makes use of a few variables as shorthand, as well as special variables like $@ and $<, which mean “the name of the current target” and “the name of the current prerequisite”, respectively.

## What extension (e.g. md, markdown, mdown) is being used 
## for markdown files MEXT = md

## Expands to a list of all markdown files in the working directory
SRC = $(wildcard *.$(MEXT))

## Location of Pandoc support files.
PREFIX = /Users/kjhealy/.pandoc

## Location of your working bibliography file
BIB = /Users/kjhealy/Documents/bibs/socbib-pandoc.bib

## CSL stylesheet (located in the csl folder of the PREFIX directory).
CSL = apsa

## x.pdf depends on x.md, x.html depends on x.md, etc
PDFS=$(SRC:.md=.pdf)
HTML=$(SRC:.md=.html)
TEX=$(SRC:.md=.tex)
DOCX=$(SRC:.md=.docx)

## Rules -- make all, make pdf, make html. The `clean` rule is below.
all:    $(PDFS) $(HTML) $(TEX) $(DOCX)
pdf:    clean $(PDFS)
html:   clean $(HTML)
tex:    clean $(TEX)
docx:   clean $(DOCX)

## The commands associated with each rule. The first one is run with 
## make html. 
%.html: %.md
    pandoc -r markdown+simple_tables+table_captions+yaml_metadata_block \ 
    -w html -S --template=$(PREFIX)/templates/html.template \
    --css=$(PREFIX)/marked/kultiad-serif.css --filter pandoc-crossref \     
    --filter pandoc-citeproc --csl=$(PREFIX)/csl/$(CSL).csl \ 
    --bibliography=$(BIB) -o $@ $<

## Same goes for the other file types. 
## Watch out for the TAB before 'pandoc'
%.tex:  %.md
    pandoc -r markdown+simple_tables+table_captions+yaml_metadata_block \ 
    --listings -w latex -s -S --latex-engine=pdflatex \ 
    --template=$(PREFIX)/templates/latex.template \ 
    --filter pandoc-crossref --filter pandoc-citeproc \ 
    --csl=$(PREFIX)/csl/ajps.csl --filter pandoc-citeproc-preamble \ 
    --bibliography=$(BIB) -o $@ $<

%.pdf:  %.md
    pandoc -r markdown+simple_tables+table_captions+yaml_metadata_block \ 
    --listings -s -S --latex-engine=pdflatex \ 
    --template=$(PREFIX)/templates/latex.template \ 
    --filter pandoc-crossref --filter pandoc-citeproc \ 
    --csl=$(PREFIX)/csl/$(CSL).csl --filter pandoc-citeproc-preamble \ 
    --bibliography=$(BIB) -o $@ $<

%.docx: %.md
    pandoc -r markdown+simple_tables+table_captions+yaml_metadata_block \ 
    -s -S --filter pandoc-crossref --csl=$(PREFIX)/csl/$(CSL).csl \ 
    --bibliography=$(BIB) -o $@ $<

clean:
    rm -f *.html *.pdf *.tex *.aux *.log *.docx

.PHONY: clean

Note that the pandoc commands are interpreted single lines of text, not several lines separated by the return key. But you can use the \ symbol to tell make to continue to the next line without a break. With this Makefile, typing make pdf would take all the .md files in the directory one at a time and run the pandoc command to turn each one into a PDF, using the APSR reference style, my latex template, and a .bib file called socbib-pandoc.bib.

You shouldn’t use this Makefile blindly. Take the time to learn how make works and how it can help your project. The official manual is pretty clear. Make’s backward-looking chain of prerequisites can make it tricky to write rules for complex projects. When writing or inspecting a Makefile and its specific rules, it can be helpful to use the --dry-run switch, as in make --dry-run. This will print out the sequence of commands make would run, but without actually executing them. You can try this with the Makefile in Listing 5 in a directory with at least one .md file in it. For example, look at the commands produced by make pdf --dry-run or make docx --dry-run or make clean --dry-run.

The particular steps needed for many projects may be quite simple, and not require the use of any variables or other frills. If you find yourself repeatedly running the same sequence of commands to assemble a document (e.g. cleaning data; running preliminary code; producing figures; assembling a final document) then make can do a lot to automate the process. For further examples of Makefiles doing things relevant to data analysis, see Lincoln Mullen’s discussion of the things he uses make to manage.

The examples directory includes a sample .Rmd file. The code chunks in the file provide examples of how to generate tables and figures in the document. In particular they show some useful options that can be passed to knitr. Consult the knitr project page for extensive documentation and many more examples. To produce output from the article-knitr.Rmd file, you could of course launch R in the working directory, load knitr, and process the file. This produces the article-knitr.md file, together with some graphics in the figures/ folder (and some working files in the cache/ folder). We set things up in the .Rmd file so that knitr produces both PNG and PDF versions of whatever figures are generated by R. That prepares the way for easy conversion to HTML and LaTeX. Once the article-knitr.md file is produced, HTML, .tex, and PDF versions of it can be produced as before, by typing make at the command line. But of course there’s no reason make can’t automate that first step, too. The rmd-starter project has a sample Makefile that begins with the .Rmd files in the directory and produces the outputs from there.