Sunday, September 21, 2008

Software Packaging on MacOS X

When I first used MacOS X, it was version 10.0.  It was a little rough around the edges.  I remember trying to do some pthreads programming and discovered that pthreads hadn't been completely implemented yet.  Ooops.

I then found another problem and started poking around in the kernel source (the kernel project is called xnu, the installed kernel file is /mach_kernel).  What I found was a huge disappointing combination of BSD code with Mach kernel code.  Yuck.  My attempts to submit patches upstream were met with failure.  But I wasn't the only one.

I upgraded to 10.1 shortly after 10.0 and it improved some things, but not much.  For example, I still couldn't have a UFS filesystem and run iTunes.  iTunes would crap itself if it was running on UFS.  It wanted HFS because of its case-preserving ability (I guess the programmers were lazy).  I hear Apple fixed this later on by making it more difficult or even removing the ability to install to UFS filesystems.  That's one way to fix the problem.

I left OS X after 10.1 and went back to Linux for my workstation.  I should point out that when I moved around between operating systems, it was always for my workstation system.  My servers have always been Linux and most likely always will be.  And Linux is still where I do almost all of my software development.

Enter 2008.  I get frustrated with my company-provided laptop.  It's old and slow and can't do virt under Linux (technologies such as Xen or KVM).  I ask for a new one that can do virt and am told no several times.  So I go buy my own laptop and start using it.  I buy a MacBook and load it 4GB of RAM.

Since it is a personal system, I decide to keep OS X installed on it.  That, and I wanted the ability to edit video and easily work with my camera and OS X is just better at that (JWZ sums it up pretty well here).  But I also use the system during work to check email, IRC, IM, and read the web.  In an effort to do more from it, I start installing the few tools I need to access our source repositories from OS X and tools so I can work with the build system from not-just-Linux.  The result is the horrible time wasting project found here.

[Why don't I use MacPorts or Fink or something else like that?  I did and I hate them.  Both systems want to _own_ your OS X system.  If I go to install, say, GNU sed, don't install a truckload of other crap because it's all interdependent.  I am perfectly fine with the system utilities when they work for me.  Look at my osxpkgs collection and you'll see a very short list of what I had to install to do Fedora Linux development from OS X.  With MacPorts and Fink, I had huge trees of basically every single thing you'd find on a Linux system.  WHY?  I don't need it.]

Development on OS X is certainly a lot better now than it was under the 10.0 days.  Lots of recent tools are available as well as a modern Python.  I like it.  For the things that did not come with OS X that I need to do work, I compiled and installed them to /usr/local.  Here is where we get in to irritating things.

I'm a little more careful with how I manage systems now.  First, I backup every system I care about.  My laptop is most important to me, so I have several backups.  I found that JWZ basically has the same backup philosophy as me.  I say basically the same, because here's where we differ:
  • I only care about my home directory.  When the system's hard disk shits the bed, I buy a new one and then do a reinstall from scratch.  I restore my home directory and then install the software packages I have in my home directory.  I find this to be easier than backing up the entire system.
  • I don't want to wait to backup the entire system, which is currently 138GB and like a kid on Flintstone vitamins...growing.
Still, his page about backups is as perfect as you can get.  Do what he says.  It only takes one disk failure to really piss you off.  You will want backups.  As a Digital Unix developer once told me, "In order to move forward, you have to backup."  (ok, he was old)

For everything I add to the system, I keep the bundles or packages in my home directory under a subdirectory titled "Crap I Use" or something like that.  When I rebuild the system, I just drag the bundles back to Applications or I run the installers again.  For my osxpkgs software, I decided to make packages so reinstalls will be easy (this follows the Unix train of thought where if I can use this hack in at least one place, I might as well spend the time and effort to do it).

Packaging software is one of those annoying developer tasks.  Developers tend to have only a passing interest in packaging software.  It's almost like a task that falls between development and system administration.  I hate system administration.  You want to have something to release, but you don't want to learn all of the details of _any_ system's stupid packaging software.  What the hell is packaging anyway?

When you install software on a Unix system, files all go to the same directory structure.  We don't have a "C:\Program Files" notion on Unix (except for /opt, but that's another story), so we need a way to track what files belong to a specific piece of software so we can remove it later or upgrade it later.  Every reasonable Unix system provides a way to do this.  On RHEL and Fedora and many others, it's RPM.  On Debian systems, it's dpkg.  On Solaris systems, it's the SysV pkg* commands.  On AIX, it's two systems: one is AIX LPP and one is RPM (!).  Each system has a set of commands that lets the user pack up the files that belong to their program in to a single file for delivery to the user.

So what does MacOS X have?  Apple wants developers to create self-contained programs where everything is in a bundle that users can drag to anywhere on their system and run by just double clicking it.  A bundle is a subdirectory that is named NAME.bundle.  Everything the application needs is stuffed in this subdirectory, such as libraries, images, documents, and so on.  This works great for software like Microsoft Office or iTunes or Mozilla.  But for command line Unix-type software, the model fails.  Apple provides another system to track that.

The other system is a bastardization of the half-assed system that was on NEXTSTEP (and OPENSTEP and CamelCaseStEp, please don't argue with me about how it should be written).  Apple provides a tool called packagemaker that can create package bundles (a subdirectory called NAME.pkg) that you can install with the package installer.  They also provide a program called pkgutil which sort of helps you see what's installed and maybe do things with those packages.  Problems I've seen so far under MacOS X 10.5:

  • The --root option to packagemaker is a bit picky.  You have a add a trailing slash on the path you specify, otherwise your staging root directory will end up on the target system.  For example:
    mkdir pkg-root
    # put stuff in ./pkg-root for the package
    packagemaker --root ./pkg-root [options]
    The resulting package will install /pkg-root to your system (dumb!). You have to run it this way:
    packagemaker --root ./pkg-root/
    And then pkg-root is treated like a staging root.
  • To show all installed packages on the system: pkgutil --pkgs
    However, this only shows packages that pkgutil can see. It won't show you any installed packages recorded in /Library/Receipts. It only shows packages with an entry in /Library/Receipts/boms. The locations appear to be mutually exclusive. Why?
  • It is possible to remove an entry from the installed packages database.  All you have to do is:

    pkgutil --forget PACKAGE

    This removes the entry from the database, but not what is actually on the system. Why is this even possible? Also note that this will again only work for what packages pkgutil can see. If you want to forget something in /Library/Receipts, just rm -rf the directory.

  • Want to completely hork your system? pkgutil does offer a way to forget a package and remove the files it owns. You can do this:

    pkgutil --unlink PACKAGE
    pkgutil --forget PACKAGE

    No dependency checking is done, pkgutil just deletes whatever is owned by that package. You do this on the BSD package, for instance, and totally hose the system. Great work, Apple.

  • If you pass the --target option to packagemaker, the resulting flat package it generates won't contain anything from --resources. It just ignores Resources. Great!

  • The different package formats created by packagemaker appear to be recorded differently in /Library/Receipts. A flat package gets a bom file in the /Library/Receipts/boms subdirectory. A package bundle with metadata gets a receipt bundle as /Library/Receipts/NAME.pkg with the bom file in the bundle as

So that's what I've discovered trying to package software on MacOS X.  It's frustrating and annoying.  Packaging is a solved problem.  And uninteresting.  Why can't OS X just use any of the dozens of packaging systems out there that are open source?

1 comment:

Alex Bandit said...

This is a good article and offer some helpful information for me,thank you! web design development