Your Promotion to Package Manager Manager

Let's talk about Homebrew. No, not the beer-making technique that left you befuddled by the appearance of approximately two thousand quirky-named, identically-tasting IPAs at the liquor store. I mean the popular package manager for macOS.

Homebrew is a beloved tool of digital preservation educators and practitioners alike, and it's little wonder why. While I will continue to bang the drum for Linux operating systems and the Windows Subsystem for Linux, macOS remains a popular choice for digipres workstations given the general familiarity of Macs and the ability to easily and natively run much handy Bash/command line software. And Homebrew provides macOS with a major piece that it's "missing" out-of-the-box: a command line-based package manager.

A "package manager" or package management system allows a user to install, uninstall, upgrade and configure the software on their computer in a consistent and centralized manner. The official Mac App Store is a package manager, for instance - rather than needing to trawl the internet to find, download, and run individual installers for every single application you'd like to try, the App Store (in theory) puts everything in one place. Homebrew is frequently described as an "App Store" for CLI programs, and the comparison is pretty apt.¹ Setting up Homebrew is pretty much step one for any Mac-based digipres machine: once it's in place, it's only a matter of minutes to ffmpeg, mediainfo and mediaconch, vrecord, exiftool, imagemagick, rsync, youtube-dl, hashdeep, ddrescue, and much more.

Homebrew is far from the only command line package manager out there - part of why it's called the "missing" package manager for macOS is because CLI package managers are included in Linux operating systems by default. Debian-based systems like Ubuntu have the Advanced Packaging Tool (APT);² Red Hat-based systems have YUM or DNF; Arch systems have Pacman, SUSE systems have zypper, etc etc etc (and Homebrew can be used on Linux as well, incidentally). Since 2017, native Windows users can even get in on the action with Chocolatey.

Homebrew isn't even the first attempt to bring command-line package management to MacOS, following in the footsteps of Fink and MacPorts (both of which are still around, though I would say with less robust or user-friendly communities around them). And that's not even getting into the crazy number of package or "dependency managers" that do basically the same work for developers looking to add modules in particular programming language environments: pip (Python), RubyGems (Ruby), npm and yarn (NodeJS/JavaScript), Maven (Java)...

All of these projects have more or less the same goal and advantages: automatically do all the work it takes to install a piece of software and make it available to the user. In many cases that means, for example, typing $ brew install mediainfo so that the mediainfo command-line application is then available by calling $ mediainfo example.mov , where it wasn't available before.

However, as with so many lovely, user-friendly innovations, there is an element here of ceding control for convenience. And while most of the time it is hopefully unnecessary to peek under the Homebrew hood, a couple of times in the past year I've seen situations come up where it helps to have some clarity about what all, exactly, Homebrew is doing when one hits enter on a "brew install".

So today I'm going to dive a bit deeper on the ins and outs of package management. I'll use Homebrew for several concrete examples because of its popularity and ubiquity in digipres training, so keep in mind that some of Homebrew's (beer-obsessed) vocabulary may be unique; but the concepts described here are basically applicable across the board, no matter what package manager you're using.

Return to the Source

First off, I would like to clarify a few basic computing concepts: namely, the different types of code and executable programs that can be installed with a package manager.

There is a key difference in programming between source code and machine code. Source code is "human-readable" - open up a source code file and you will be greeted with plain text, written and formatted according to the conventions of a particular programming language (Python, Bash, JavaScript, Ruby, HTML, etc). These sorts of files make up the majority of what you see on popular source code hosting and version control platforms like GitHub, GitLab, Bitbucket, SourceForge. They facilitate the work of programmers and developers: broadly speaking, people read and write source code.

Machine code, as the name implies...is for machines. It is a set of instructions that can be directly executed by a computer's processor. Machine code is numerical - there are no words to read, but the CPU interprets and understands a given string of numbers, or even just ones and zeroes, as a series of actions for it to perform. Open up a machine code file in a text editor, and you will just get a string of absolute wingding gobbledygook as the editor tries (and fails) to interpret and display this numerical series as text. (Open it up in a hex editor instead, and you might get a different result, but that's another day's topic)

In order to get from source code (human-readable) to machine code (machine-readable) you usually have to compile the source code. The purpose of a compiler is to take your source code text and translate it down into ones and zeroes for your computer to actually do something with. Very broadly speaking, every operating system has its own compiler to make executable machine code for the particular OS and CPU that you are using right now. The same source code file (written, let's say, in Python) is going to create different machine code whether the machine code is intended and compiled to run on MacOS or Windows.

I'm probably making this sound more complicated than it is: we see the results when you, for instance, try to open and run a ".exe" file intended for Windows on macOS³ Windows .exes are machine code: again, try opening one up in a plain text editor like Notepad or Atom or Sublime Text and see the results.

Because executable machine code is numerical, and at its very core just a very long string of ones and zeroes, executable machine code files, like Windows .exe files, are sometimes just called "binaries".

This can get *really* confusing because technically a "binary" is just "any file not written and meant to be displayed as plain text" - .mp3s, .movs, .pdfs, .jpgs, all of them and many more are also "binary" files. But in the particular context of package management and installing applications, the term "binary" is very frequently used interchangeably and synonymously with "version", e.g. "a macOS binary", "a Windows binary", etc. I sort of wish I could've avoided this altogether but it will absolutely come up when troubleshooting or searching support forums with package management questions, so here we are.

So if you want to create software that works on different operating systems and processors, someone usually has to compile it from source code first into executable binaries that match the desired operating systems and processors. Again, you've seen this out in the digital world: you're trying to download a piece of software from a website and there are two different links, one "For Windows" that downloads a .exe, and one "for MacOS" that gives you a .app or .pkg or such. And because compilation is essentially an act of translation, there are correspondingly things that can get lost in that process. We'll see a direct example of this later in a case study.

A last aside: you may have encountered scripts and be thinking, "hey, but I can run (execute) a .py Python script or a .sh Bash script, which are source code files, without compiling!" Well, you got me! Just as I mentioned above that not all binaries are executable, not all executables are binaries. Scripting uses a process called interpreting *instead* of compiling. This usually allows scripts to be a little more portable than binaries across operating systems, but the basic idea - that there is a layer of translation necessary between the source code file and the computer - is the same.

Define Your "Install"

So what all exactly does source code and machine code and binaries have to do with package management and installing software?

Let's say you are a web archivist running macOS and you installed two different pieces of software at the same time: youtube-dl, which is a handy command line tool for downloading media from YouTube and other hosting sources, and Webrecorder Player, a wonderful desktop tool for viewing and inspecting web archives (WARCs), no internet connection required.

You "installed" both programs. But youtube-dl is not in your "Applications" folder next to Webrecorder Player. And no matter how many variations on $ webrecorder you type into your Terminal, Webrecorder Player does not launch on your desktop.

If that's the case... what does "install" mean?

"Installing" an application is actually a highly contextual process. The end-goal is always the same: you want to take a program and make it usable. But how you're supposed to use a program....depends on the program! And thus the steps actually necessary to complete an installation also depend on the program, user, and goals involved. When broken down into smaller, discrete actions - downloading source code, compiling the code, creating a binary, moving that binary to a particular folder, changing some operating system settings so that you can execute that binary - the installation process can get quite variable and customizable.

So finally, we return to package managers. As I said up top, package managers take care of many of the nitty-gritty details of installing programs so that as far as you, the user, are concerned, "install" just means "click this button" or "type one brief command." They are meant to cover a large majority of use cases with minimal effort - but potentially to the detriment of edge cases or tinkering an installation to exactly how a user needs it.

To return specifically to Homebrew, every package or program that you can install with Homebrew has a corresponding formula. You can browse all of them here. Homebrew's formulae are all hosted on GitHub, and every single one of them is just a brief script (written in Ruby) which defines instructions to answer the question:"what does 'install' mean for this particular program?" When you type $ brew install mediainfo, Homebrew searches in the Homebrew/homebrew-core repository in GitHub, finds the mediainfo.rb formula, and then follows whatever steps that formula tells it to do.

To get a better idea of what these instructions actually look like and how Homebrew interprets and performs them, let's look at a concrete example.

Case Study: vrecord's Homebrew Formula

This is the whole Homebrew formula for vrecord, the open source video digitization program, developed and maintained by the Association of Moving Image Archivists' Open Source Committee - let's take a look and break it down by section:

class Vrecord < Formula  desc "Capturing a video signal and turning it into a digital file"  homepage "https://github.com/amiaopensource/vrecord"  url "https://github.com/amiaopensource/vrecord/archive/v2020-07-01.tar.gz"  version "2020-07-01"  sha256 "983264ca6a69b78b4487a7479ab5a4db04cbc425f865ec2cb15844e72af4f4ac"  head "https://github.com/amiaopensource/vrecord.git"  depends_on "amiaopensource/amiaos/deckcontrol"  depends_on "amiaopensource/amiaos/ffmpegdecklink"  depends_on "amiaopensource/amiaos/gtkdialog"  depends_on "cowsay"  on_macos do    depends_on "bash"    depends_on "gnuplot"    depends_on "mediaconch"    depends_on "mkvtoolnix"    depends_on "mpv"    depends_on "qcli"    depends_on "xmlstarlet"  end  on_linux do    def caveats      <<~EOS        ** IMPORTANT FOR LINUX INSTALL **        Additional install steps are necessary for a fully functioning Vrecord        install on Linux. This includes using the standard package manager to        install gnuplot, xmlstarlet, mkvtoolnix and mediaconch. Additionally,        it often is necessary to remove the Homebrew installed version of SDL2        to prevent conflicts. For more information please see:        https://github.com/amiaopensource/vrecord/blob/master/Resources/Documentation/linux_installation.md      EOS    end  end  def install    bin.install "vrecord"    bin.install "vtest"    prefix.install "Resources/audio_mode.gif"    prefix.install "Resources/qcview.lua"    prefix.install "Resources/vrecord_policy_ffv1.xml"    prefix.install "Resources/vrecord_policy_uncompressed.xml"    prefix.install "Resources/vrecord_logo.png"    prefix.install "Resources/vrecord_logo_playback.png"    prefix.install "Resources/vrecord_logo_audio.png"    prefix.install "Resources/vrecord_logo_edit.png"    prefix.install "Resources/vrecord_logo_help.png"    prefix.install "Resources/vrecord_logo_documentation.png"    man1.install "vrecord.1"    man1.install "vtest.1"  end  test do    system "#{bin}/vrecord", "-h"  endend

Metadata

class Vrecord < Formula  desc "Capturing a video signal and turning it into a digital file"  homepage "https://github.com/amiaopensource/vrecord"  url "https://github.com/amiaopensource/vrecord/archive/v2020-07-01.tar.gz"  version "2020-07-01"  sha256 "983264ca6a69b78b4487a7479ab5a4db04cbc425f865ec2cb15844e72af4f4ac"  head "https://github.com/amiaopensource/vrecord.git"

• class Vrecord < Formula - This line is required to start every formula - Homebrew needs to know that the script you've pointed it to is indeed a formula! (The first letter of the program/formula name has to be capitalized to conform with Ruby syntax, regardless of how you normally write the name)
• desc - A brief description of the application and its purpose. Not strictly required, but helpful - this will often match the "About" project description on the application's GitHub/GitLab page, if the code is hosted there.
• homepage - A project site where users can go for more information about the program. It's mandatory to include a homepage if you want to include your formula in Homebrew's core list of packages.
• url - This is required - it directs Homebrew to where it should download the program's code (in this case, and in the case of many Homebrew formulae, the source code is contained in a tarball, a format that takes all the source code files and packages them together into one, like a .zip file). If a program has multiple versions or releases, this is the field that specifies *which* version Homebrew will try to install.
• version - This is metadata that helps Homebrew keep track of which version of a program you have installed (particularly helpful if you have multiple versions of a program/formula installed on the same computer). It's not required to have this field, and if the source code URL above comes from GitHub, Homebrew can usually pull this version info from the file name automatically - but it doesn't hurt to specify manually.
• sha256 - This is the checksum for the source code tarball from the URL above. Once the tarball has downloaded to your computer, Homebrew will automatically check that it matches this checksum here - basically a security feature to make sure that what Homebrew downloads is indeed the code/program that you wanted. This is required.
• head - "Head" specifies a cutting-edge version of the program - if it's specified, it means early adopters can try out the absolute newest changes and revisions to the program by its developers by running $ brew install --HEAD <formula> instead of just $ brew install <formula>. It's basically a way to signal to users that some new options or features may be available for them to try out but they are not yet considered stable. It's not required if the formula/program author only wants Homebrew users to install "guaranteed", stable releases of their software.

Dependencies

depends_on "amiaopensource/amiaos/deckcontrol"  depends_on "amiaopensource/amiaos/ffmpegdecklink"  depends_on "amiaopensource/amiaos/gtkdialog"  depends_on "cowsay"  on_macos do    depends_on "bash"    depends_on "gnuplot"    depends_on "mediaconch"    depends_on "mkvtoolnix"    depends_on "mpv"    depends_on "qcli"    depends_on "xmlstarlet"  end  on_linux do    def caveats      <<~EOS        ** IMPORTANT FOR LINUX INSTALL **        Additional install steps are necessary for a fully functioning Vrecord        install on Linux. This includes using the standard package manager to        install gnuplot, xmlstarlet, mkvtoolnix and mediaconch. Additionally,        it often is necessary to remove the Homebrew installed version of SDL2        to prevent conflicts. For more information please see:        https://github.com/amiaopensource/vrecord/blob/master/Resources/Documentation/linux_installation.md      EOS    end  end

In this section of a formula, the program developer needs to specify any and all external dependencies - that is, if there is other code, or other programs, that have to be present and installed on the user's computer before vrecord can be installed and used correctly.

Every depends_on line specifies a dependency, and every dependency listed is...another Homebrew formula. So before Homebrew proceeds to the next section of the vrecord formula (the actual "install" section), it will go to each and everyone of these formula *first* and complete the instructions found there...(including, if those formula specify dependencies, going to their dependent formula - and on and on, down the line, if necessary).

(This means that the amount of time it takes vrecord to install can vary wildly, depending on how many of those depends_on formula are already present on your computer when you start - if you already have installed cowsay or mediaconch before, Homebrew will just skip over this part for those dependencies.)

In this case, the vrecord formula also specifies slightly different behavior depending on whether the Homebrew user is running macOS or Linux. Homebrew works on Linux systems, but, as I mentioned earlier, Linux systems usually have their own baked-in package managers (e.g. apt), and sometimes Homebrew and native Linux package managers don't play nice with each other - so in this case, rather than having Homebrew run the install process for all those dependencies on_linux, the vrecord formula instead defines a caveat, which is just a warning to display to the user. This caveat, obviously, recommends using the native package manager to install certain dependencies instead of Homebrew, to avoid conflicts and errors.

Installation

def install    bin.install "vrecord"    bin.install "vtest"    prefix.install "Resources/audio_mode.gif"    prefix.install "Resources/qcview.lua"    prefix.install "Resources/vrecord_policy_ffv1.xml"    prefix.install "Resources/vrecord_policy_uncompressed.xml"    prefix.install "Resources/vrecord_logo.png"    prefix.install "Resources/vrecord_logo_playback.png"    prefix.install "Resources/vrecord_logo_audio.png"    prefix.install "Resources/vrecord_logo_edit.png"    prefix.install "Resources/vrecord_logo_help.png"    prefix.install "Resources/vrecord_logo_documentation.png"    man1.install "vrecord.1"    man1.install "vtest.1"  end

Finally, the meat of the matter - in this section of the formula, we actually get to what "install" even means in the context of vrecord.

vrecord is a relatively straightforward case because, the source code doesn't need to be compiled. The source code is itself a Bash script, which can be interpreted and run by a macOS or Linux system as-is (or, as-long-as-the-dependencies-have-been-installed). There is no translation down to the ones and zeroes of machine code required. So the installation process here isn't a matter of compiling, it's just a matter of moving the files to where they can be used.

prefix here is a variable - it's a setting that's part of your overall Homebrew configuration, so that any time any Homebrew formula mentions "prefix", the package manager will just sub in the value it has stored there. Specifically, prefix defines a file path, an over-arching directory/folder for all of Homebrew's files to live in.

Usually, prefix is set by default when you first install Homebrew and you never have to mess with it again (the whole point of a package manager being to mess with things as little as possible). You can see yours by running $ brew config and looking for the HOMEBREW_PREFIX line - on MacOS, it's usually something like /usr/local. So, all of the programs that Homebrew downloads, installs, compiles, whatever - they'll go into that /usr/local directory (unless otherwise specified by a formula) - all neatly nested and organized according to the package/formula name and version.

(The nested directory for keeping code, in Homebrew-speak, is called the Cellar. So if you want to find all your Homebrew-installed programs, poke around your /usr/local/Cellar directory.)

So all those lines that start with prefix.install are saying: take this file (from inside the tarball you downloaded) and put them inside the nested folder specified by the prefix variable. In vrecord's case, we're just taking some image and configuration templates (the .XML files) for vrecord's GUI mode and putting them in appropriate locations in the Cellar.

The bin.install options are doing the same thing, but flagging an extra step. These two entries (vrecord and vtest) are the actual scripts that we want to run, so they need to be made executable for you, the user, to run them. The bin.install directive 1) links the specified files to a particular directory - in this case, /usr/local/bin - where your operating system expects to find executable command-line programs, and 2) adjusts their permissions so that you can run these scripts (without needing "sudo" permission.

("bin" in these file paths stands for "binaries" - remember that there is a general but imprecise equation between binaries and executables, so that is why we are putting the vrecord script there - because it is executable, even though it is a source code file, not actually a binary/machine code)

The man1.install directives are again very similar to bin.install - these are manual pages that explain how to run the program (by typing $ man vrecord or $ man vtest). macOS expects to find these files in a certain place, just like how it expects executable binaries to be in /usr/bin or /usr/local/bin. So man1.install copies these files to that location.

Test

test do    system "#{bin}/vrecord", "-h"  endend

Homebrew formula writers can optionally put in a "test" block at the end of the script to see if the installation process proceeded correctly. It's generally out-of-scope of the Homebrew project to make sure that every single feature of every single program works as expected - but it can be a handy check just to make sure at least at the end of all this you got an executable something out of it.

In the case of vrecord, this test block simply directs the user's computer to try running the command $ vrecord -h - if the computer encounters no errors trying to run this command (which just displays vrecord's "help" page), then Homebrew will consider this a successful installation and finish running.

Phew. Let's review - in the end, what did that vrecord Homebrew formula do? It told the computer to:

1. Go to a certain URL and download the file (tarball) it found there.
2. Check for any other external software that vrecord needs to work.
3. Extract vrecord's files out of the tarball and move them to a different folder.
4. Test that the computer can find and execute the program now that it's been moved.

Case Study: Troubleshooting ewfmount

..now what about a slightly more complicated example? One that involves source code compilation? Let's start with a Tweet.

is anyone successfully running libewf with fuse osx on a mac os? If so what build did you use? I can't get libewf (from sleuthkit homebrew) to "see" osx fuse (tried binary from osx fuse website and brewed osxfuse)

— Eddy Colloton (@EddyColloton) August 15, 2019

A bit of context: libewf is a collection of software that allows users to work with EWF-formatted disk images, which is a pretty popular option for preserving hard drives. One of the tools included in libewf is ewfmount, a program that allows you to, well, mount EWF disk images and explore the files on them, just as if they were a physical, external hard drive attached over USB or what have you.

Complicating things, ewfmount won't work with MacOS out-of-the-box. First you have to install "FUSE for MacOS", a piece of software that allows MacOS to work with file systems beyond the ones that Apple cares about (basically just APFS at this point). Otherwise, trying to mount your EWF disk image is going to have the same affect as plugging in an unformatted hard drive.

Eddy had installed these two pieces - libewf/ewfmount and FUSE for MacOS, using Homebrew, but the two pieces of software still weren't "seeing" each other. To find out why, it helped to take a look at the Homebrew formulae - if the installation wasn't working (Eddy didn't have a usable version of the desired program at the end of the process), then one of the steps Homebrew was taking by default had to be incorrect for his context.

Here is the current Homebrew formula for libewf - see if you can spot the problematic line:

class Libewf < Formula  desc "Library for support of the Expert Witness Compression Format"  homepage "https://github.com/libyal/libewf"  # The main libewf repository is currently "experimental".  url "https://github.com/libyal/libewf-legacy/releases/download/20140808/libewf-20140808.tar.gz"  sha256 "dfe29b5f2f1841ff1fe11979780d710a660dbc4727af82ec391f398e6b49e5fd"  license "LGPL-3.0"  bottle do    cellar :any    sha256 "43d8ba6c2441f65080f257a7239fe468be70cb2578ec2106230edd1164e967b6" => :catalina    sha256 "4c5482f8f1c97f9c3f3687bccd9c3628b314699bc26743e641f2ae573bf95eeb" => :mojave    sha256 "cae6fd2f38855fd15f8a50b644d0817181fed055aef85b7793759d7703a833d4" => :high_sierra  end  head do    url "https://github.com/libyal/libewf.git"    depends_on "autoconf" => :build    depends_on "automake" => :build    depends_on "gettext" => :build    depends_on "libtool" => :build  end  depends_on "pkg-config" => :build  depends_on "openssl@1.1"  uses_from_macos "bzip2"  uses_from_macos "zlib"  def install    if build.head?      system "./synclibs.sh"      system "./autogen.sh"    end    args = %W[      --disable-dependency-tracking      --disable-silent-rules      --prefix=#{prefix}      --with-libfuse=no    ]    system "./configure", *args    system "make", "install"  end  test do    assert_match version.to_s, shell_output("#{bin}/ewfinfo -V")  endend

...give up? On Line 40, in the "install" section of the formula, there are several "args" listed, including --with-libfuse=no. This our clue and our culprit!

These "args" are arguments (options) for source code compilation. So by default, the Homebrew formula defines "installation" of libewf/ewfmount to not include a software library called libfuse - which, as the name implies, is a critical component that libewf/ewfmount requires for communicating with FUSE for MacOS. Without it, ewfmount can not mount EWF disk images on MacOS.

Now, you can edit *any* Homebrew formula and how it works, just for you, by running $ brew edit <package>. This will open up a local copy of the formula in a text editor and let you change and edit options, without affecting how this formula behaves for all other Homebrew users.

But, unfortunately, in this case it is still not enough to just run $ brew edit libewf and change line 40 to --with-libfuse=yes. That's because, towards the top of this formula, you'll notice a section that starts with "bottle".

"Bottles" are Homebrew's clever name for binaries. These are pre-compiled versions of the source code for libewf, ready to go for MacOS (with flavors for High Sierra, Mojave, or Catalina, as indicated). If a bottle is specified in a Homebrew formula, that's it - any instructions for how to compile the source code, later on in the formula, will be ignored, because as far as  Homebrew is concerned, you already have a working binary, and it will proceed from there. So editing line 40 will make no difference, because either way Homebrew will by default use a bottle/binary that was (we can assume/infer) already created with --with-libfuse=no.

The solution, in this case, is to both edit line 40 to --with-libfuse=yes AND run $ brew install --build-from-source libewf instead of just $ brew install libewf. This extra flag/option tells Homebrew to ignore the "bottle" section, skip over those pre-compiled binaries, and start from scratch with the source code specified at url. THEN, proceeding down the formula to the "install" section, Homebrew will compile the source according to the options you set, creating a new version/binary with libfuse enabled.

(There is a longer, more rambling and circular version of this solution that I wrote in that Twitter thread - but, it is a little out-of-date, as the Homebrew formula has been edited to adjust the default source code URL since that writing. The basic point - that getting a version of ewfmount that works with FUSE for MacOS using Homebrew requires the two-step process of changing a flag for use during compilation AND telling Homebrew to start from source - still stands)

Wrapping Up

I hope that these examples and explanations help digipres users understand that, while an AMAZING tool and community, Homebrew is not magic! It plays by certain rules and relies on assumptions of what will work best for the most number of users. The vast majority of the time - those assumptions will probably work fine for you!

But if they don't, it's just like any other piece of technology, analog or digital - you'll need to know a bit more about what it's doing in order to effectively troubleshoot, fix, or change it. So I'll leave off with a note that Homebrew's documentation IS ALSO AMAZING! The (open-source and volunteer!!!) contributors have pulled together tons and tons of information about how Homebrew works, guidelines for different ways of using it, templates and automated tools for creating and testing formulae, etc.  I really recommend exploring those pages to learn more about the various tricks up Homebrew's sleeve (there's a lot of built-in neatness you might not even know about!!) and as a diving-off point to learn more about code compilation, operating systems, interpreters, and just how software works, in general.

Am I just saying that because this is the longest blog post I've ever written, and there were about twenty different tangents and other topics that I didn't even explore, and that I probably don't have the time to write about? Maybe! But it's time to put a cork in it.