Integrating Lilypond with more complex LaTeX documents

In a previous post, I described my trials and tribulations in getting the Lilypond music-engraving package to work with TeXShop, a Mac-based front-end for the LaTeX typesetting system. The example used to illustrate the technique was a rather simple document, with no included external images or files.

As I've continued my rediscovery of LaTeX (having first been exposed to it as an undergraduate), I've come to realize that the rather straightforward techniques I had described previously break down when dealing with more complex documents.  This first came to light when I attempted to incorporate Lilypond content into a LaTeX document built with the Tufte-LaTeX classes, a collection of layout classes created to render LaTeX documents using the design principals of Edward Tufte, author of The Visual Display of Quantitative Information, and one of the worlds foremost experts on the presentation of information. A self-documenting example of the output of the Tufte-LaTeX book class can be seen here.

To really understand what went awry when trying to incorporate Lilypond content into a Tufte-style document, it is necessary to take a closer look at the Lilypond-LaTeX.engine configuration file that I came up with (these engine files merely being Unix shell scripts which are invoked by the TeXShop GUI front-end when it comes time to build and render the document):

#!/bin/bash
export PATH=$PATH:/Applications/LilyPond.app/Contents/Resources/bin
# Delete existing output (PDF) file
rm "$1".pdf
# Create directory to put all temporary stuff
mkdir -p "$1_r"-out/
# Invoke L-B
lilypond-book --output="$1_r"-out --pdf "$1"
cd "$1_r"-out
pdflatex "$1"
# Copy output (PDF) file to parent directory
mv *.pdf ..
# Delete temporary stuff
rm -rf *
cd ..
# Delete (empty) temp directory
rmdir "$1_r"-out
# Display output (PDF) file
open "$1".pdf

Note that a temporary output directory is created to which the output of lilypond-book is directed. Why? When lilypond-book is invoked, it reads in the specified .tex file, processes any lilypond commands it finds, and uses them to create small graphics files containing the rendered music notation, then rewrites the .tex file, replacing the lilypond commands with \input commands pointing to small .tex files which, in turn, use \includegraphics commands to load the images generated by lilypond. Pdflatex is then invoked on the newly rewritten version of the main .tex file, resulting in the generation of the output PDF.  If a separate subdirectory were not used for the output of lilypond-book, the script would fail when it tries to overwrite the original version of the primary .tex file with the updated version. By creating the output in a subdirectory, the original .tex file is preserved.

The problem with this approach in a more complex document, such as the case of something built with the Tufte-LaTeX document classes, is if \include or \input commands (or related commands) come into play, not to mention ancillary files such as .toc (table-of-contents) files.  In the specific case of the Tufte-LaTeX example, external graphics are loaded into the document via the \includegraphics command, the path to which is determined by the \graphicspath command. (In retrospect, a simple solution to this specific problem would have been to simply edit the argument of the \graphicspath command to specify an explicit rather than relative path, but that isn't exactly what I came up with, nor would it solve the issue of the .toc file.) When lily-pond book spits out a new copy of the .tex file, then internally invokes LaTeX to process it (even before the explicit invocation of pdflatex found in the engine file shown above), LaTeX tries and fails to find the external graphics called by the \includegraphics command, and the overall process fails.

My first successful workaround to this issue was to simply make use of a pre-created output directory (rather than one created on the fly as in the engine file above) and make sure that this output directory contained copies of the graphics subdirectory and other ancillary files.  However, despite the fact that it works, such a workaround is too cumbersome to be practical since the ancillary files would have to be constantly synchronized to the output directory.

After further tinkering (mainly in the command line environment, bypassing TeXShop entirely), I came up with a better way: renaming the primary .tex file to have a .lytex extension. Lilypond-book will accept a .lytex file as input, and the output can be written to the same directory with the standard .tex extension, thus negating the overwrite issue. The included graphics are found, and all is well.

Well, that worked specifically for the Tufte-LaTeX class. But what about for even more complex documents. For example, when creating a book with LaTeX, it is common to maintain a minimal primary file which then uses \include commands to bring in the individual chapters, which are contained in separate .tex files. For example, suppose we have a main file called master.tex with the following contents:

\documentclass[11pt]{book}
\usepackage{graphicx}
\begin{document}
\author{Me, Myself, & I}
\title{My Magnum Opus}
\frontmatter
\maketitle
\tableofcontents
\mainmatter
\include{chapter1}
\include{chapter2}
\include{chapter3}
\end{documents}

Furthermore, let us suppose that the three chapters consist of the following files, called chapter1.tex, chapter2.tex, and chapter3.tex respectively:

\chapter{It Begins}
This is the content for the first chapter....

---

\chapter{The Middle}

Now the plot thickens....

This time, with music!

In Western music, the primary reference tone is the pitch A above middle C, which is defined by ISO standards as having a fundamental frequency of 440Hz. In twelve-tone equal temperament, the frequencies of all other tones within that octave are related to this pitch according to the following formula:

\begin {displaymath}
F(n) = 440 \times 2^{ \frac{n}{12} }
\end {displaymath}

Note that this "twelth roots of two" scheme does not provide pure harmonics of the sort obtained with Pythagorean Tuning or Just Temperament, but it does avoid the "wolf tones" found in those older tuning systems in the transition from one octave to another, thus allowing music to span multiple octaves or to be transposed into different keys.

To find the frequency of the note in the next octave up, simply double the frequency. To go down an octave, divide the frequency by two.

With frequency intervals thus defined, we can begin dividing the continuous acoustic spectrum into discretized structures known as scales. Here is a one octave chromatic scale:

\begin[quote,fragment,staffsize=16]{lilypond}
\set Score.timing = ##f
\override Staff.TimeSignature #'transparent = ##t
\relative c' {
c4 cis d dis e f fis g gis a ais b c
}
\end{lilypond}

---

\chapter{Finale}
The thrilling climax....

Note that Chapter 2 includes the same lilypond code that I used in my first article. For the reasons explained earlier, this will cause lilypond-book to fail.  However, if we change the filename of that file to chapter2.lytex, and run lilypond-book against it with the command "lilypond-book --pdf chapter2.lytex", a proper chapter2.tex file will be generated (albeit with a harmless warning about "\begin{document}" not being found), and we can then process master.tex with pdflatex normally, including multiple runs so that the table of contents will be properly generated.

So what about including lilypond code in the main file?  Well that is an entirely different kettle of fish, and has thus far proven to be problematic in my experiments. For the time being, it appears that stuffing the lilypond code into include files might be the best way to go.