My first book, Docker for Developers, is out now!

My first book, Docker for Developers from Packt Publishing, is available now on Amazon. I’d like to thank my co-authors Mike Schwartz and Andrew K. Dennis, and my employer, Modus Create, who supported this effort.

I wrote the longest part, Section 2, dealing with deploying Docker applications and monitoring them in production. It has taken over 500 hours of effort to write, edit, record screencasts, and promote the book so far. If you are interested in Docker and want to understand some practical paths to deploy your applications, ranging from very simple single host setups to using a Kubernetes cluster in Amazon Web Services, this book can help you get there.

Monitor your backups, and monitor the monitor that monitors your backups

(Edited on 2019-12-30 to reflect the availability of icinga2-watchdog.sh and to fix minor copy problems related to the timeline and headlines, edited 2020-07-16 to fix links and minor copy problems.)

Learning the hard way about data loss, backups, and monitoring

Once upon a time, I had a really nicely tuned Nagios 1.x monitoring system for The Obscure Organization. It was in production for 7-8 years, from 2009 or maybe 2010 to 2016. It ran on a CentOS 5 host running on the Nexcess VPS service.

I had it tuned that it would only alert if something was broken, it would shut up in the middle of the night, and realistically it could go weeks or months between alert notifications. The systems it was monitoring were very stable, also. I want to emphasize that I had it really, really well tuned – better than any other monitoring system I’ve ever worked with.

Trouble in Monitoring Paradise

But then in the early part of 2016 I got email from Nexcess saying they were shutting down that line of business. They asked me several times if I had migrated away from their service and I said “No!”. At the time I was working on an extremely demanding customer engagement at work and putting in 60+ hour weeks and I could not take the time to migrate this system on their schedule.

I heard nothing from them, but they kept sending bills like clockwork that got paid through a PayPal subscription, so I figured all was well and I still had time.

The only production workload running on that server was the Nagios monitoring system. It monitored all of the Obscure hosts, the services that were running on them, and the Bareos / Bacula backup system jobs that reliably backed up all the servers. It also monitored some odds and ends of information systems I cared about: I’d get a text if my printer ran out of paper.

I’ve had either Bacula or its more libre-focused fork Bareos running in production for Obscure since 2005. I had to use it in anger to restore when Obscure suffered a catastrophic hardware failure in 2008 in the RAID controller on our main server, and it worked flawlessly when it counted. I completed a full restore to a borrowed VM in about 6 hours, almost all of it waiting time for the data to spool off the backup disks. In 2016 I had enough disk space attached to hold 3 months of backups (dailies for a week, differential weeklies for a month, and full monthlies for 3 months).

When I needed to restore something trivial due to operator error in the waning months of 2016, I discovered to my horror that:

  1. The backup system had run out of space and had stopped working several months before due to a low disk space condition, and
  2. the Nagios server had all of its backup files deleted or overwritten months before.

I lost the entire configuration of my finely-tuned monitoring system! 😱😭

Nexcess didn’t have any backups of the decommissioned systems, which seemed kind of reckless. Or maybe their backups rotated out of their storage pools by that point too.

Aftermath and Recovery

In January of 2017 I got Nexcess to refund Obscure more than 6 months of service charges, but that doesn’t really make up for losing the entire Nagios configuration.

In February of 2017 I started building a new Icinga2 server on the a new host that is the only one that Obscure currently runs in AWS EC2, with the intent to replace the old Nagios server.

I had to set the new Icinga2 server aside as I cared for my mother in her last months, she was diagnosed with terminal lung cancer in April of 2017 and died on November 28, 2017.

I didn’t get the new Icinga2 server configured to do anything useful until April of 2019, and I got it to be effectively a superset of what I had in the old Nagios monitor by June of 2019.

Conclusion

You can tune your monitoring system too well! If it is normally silent it can lull you into a false sense of security.

This is why I have Uptime Robot configured as a secondary monitoring system for the Icinga system that replaced the old, now-lost Nagios system. I also have Uptime Robot monitor the most critical hosts and services for Obscure, in case the new primary monitoring system fails. The Uptime Robot free tier is pretty capable, it a good enough job where it counts and sends a non-Obscure email address alerts if things go boom.

I also wrote a watchdog script that runs through cron (icinga2-watchdog.sh) that will tickle me (and the Slack #alerts room where these alerts go) once per day. Between Uptime Robot and the watchdog script, it should be hard to ignore a cascading failure that takes out the monitoring system.

SIC TRANSIT GLORIA MUNDI

A Internet site with staying power – and a trip down FTP memory lane

Digging through an ancient archive of some of my earliest surviving code, I found a file called UCSD.txt, containing a directory listing from a File Transfer Protocol (FTP) site from 26 years ago:

$ ls -l UCSD.TXT
-rwx------  1 rbulling  staff  1612 Mar  6  1993 UCSD.TXT
$ cat !$
cat UCSD.TXT
150 Opening data connection for /bin/ls (128.173.18.252,1033) (0 bytes).
total 5690
-r--r--r--  1 102      ftp        510989 Mar 25  1989 CMT.tar.Z
-r--r--r--  1 ftp      ftp        923933 Jan 20  1991 FINALEdemo.exe
-r--r--r--  1 102      ftp           517 Jan 27  1989 K1patched.note
-r--r--r--  1 102      ftp         53594 Jan 27  1989 K1patched.shar
-r--r--r--  1 ftp      ftp        185801 Jan 20  1991 MPPdemo1.exe
-r--r--r--  1 ftp      ftp        122400 Jan 20  1991 MPPdemo2.exe
-r--r--r--  1 ftp      ftp        343994 Jan 20  1991 MPPdemo3.exe
-r--r--r--  1 ftp      ftp        456064 Feb 15  1992 baldem.exe
-r--r--r--  1 ftp      ftp          1301 Feb 15  1992 baldem.txt
-r--r--r--  1 ftp      ftp            68 Sep  4  1991 byear.doc
-r--r--r--  1 ftp      ftp        130846 Sep  3  1991 byear.zip
-r--r--r--  1 ftp      ftp         12399 Jun  9  1992 camsys10.lzh
-r--r--r--  1 ftp      ftp           114 Jun  9  1992 camsys10.note
-r--r--r--  1 ftp      ftp           735 Jul  3  1992 canvas.txt
-r--r--r--  1 ftp      ftp        205023 Jul  3  1992 canvas.zip
-r--r--r--  1 ftp      ftp         22863 Oct 19  1989 copyist.zip
-r--r--r--  1 ftp      ftp         22554 Oct 21  1989 copyist2.zip
-r--r--r--  1 ftp      ftp         14117 Oct  4  1990 d50get.exe
-r--r--r--  1 ftp      ftp         14563 Oct  4  1990 d50put.exe
-r--r--r--  1 ftp      ftp          1715 Oct  8  1990 d50putget.doc
-r--r--r--  1 ftp      ftp           908 Sep  8  1991 eps.README
-r--r--r--  1 ftp      ftp         61913 Sep  8  1991 eps.tar.Z
-r--r--r--  1 ftp      ftp        307200 Dec 31
$ # Now that's some Internet ancient history!

I found the origin FTP site, ftp.ucsd.edu with a quick Google search by looking for baldem.txt, which seemed like a pretty distinct file name! These files are part of the UCSD MIDI archives, useful to electronic musicians and tinkerers:

Astonishingly, the files that were there in 1993 are still there, along with many newer files. Disappointingly, the timestamps currently present are all fixed to a 2013 date. UCSD has hosted a file transfer server of one sort of another since at least 1973 (see RFC 532, which is being mentioned in RFC 959, the definition of the File Transfer Protocol released in 1985..

Back in the day, before NCSA Mosaic and its many cousins made the World Wide Web popular and easy, when you wanted to find software or files, you first had to identify an FTP site that was a likely source for the type of content you were seeking. Some people kept curated lists of FTP servers in documents on other FTP servers, or in Gopher servers, but there was no universal hyperlinking standard in wide use. You might telnet to an Archie server, find out where the file was, and then use FTP from the command line on your computer to retrieve the file.

These days there are still thousands of FTP servers running that allow public (anonymous) access. The Internet Archive also maintains the FTP Boneyard containing archives of long-gone sites. People are doing some interesting security work surrounding FTP to this day.

Note that FTP lacks cryptographic safeguards for data integrity; an attacker with access to a network node between the client and server can view the clear text of FTP transmissions, and may be able to inject malicious content without being detected. More modern approaches that have cryptographic safeguards against both snooping and data corruption include HTTPS, SFTP and SCP.

The convention for authenticating to a public FTP server is to login with the user anonymous and give your email as the password for politeness. To better compare the 1993 experience to our super-easy-modern-times-2019-web-browser-way, please review this fresh session transcript recorded with a Linux ftp client:

$ ftp ftp.ucsd.edu
Connected to mirror.ucsd.edu.
220 Welcome to mirror.ucsd.edu
530 Please login with USER and PASS.
530 Please login with USER and PASS.
KERBEROS_V4 rejected as an authentication type
Name (ftp.ucsd.edu:rbulling): anonymous
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd midi/software/ibmpc
250 Directory successfully changed.
ftp> dir
227 Entering Passive Mode (132,239,1,12,39,28).
150 Here comes the directory listing.
-r--r--r--    1 ftp      ftp          1281 Jan 09  2013 BANKEDIT.TXT
-r--r--r--    1 ftp      ftp        606423 Jan 09  2013 BANKEDIT.ZIP
-r--r--r--    1 ftp      ftp        510989 Jan 09  2013 CMT.tar.Z
-r--r--r--    1 ftp      ftp        923933 Jan 09  2013 FINALEdemo.exe
-r--r--r--    1 ftp      ftp           517 Jan 09  2013 K1patched.note
-r--r--r--    1 ftp      ftp         53594 Jan 09  2013 K1patched.shar
-r--r--r--    1 ftp      ftp        185801 Jan 09  2013 MPPdemo1.exe
-r--r--r--    1 ftp      ftp        122400 Jan 09  2013 MPPdemo2.exe
-r--r--r--    1 ftp      ftp        343994 Jan 09  2013 MPPdemo3.exe
-r--r--r--    1 ftp      ftp        456064 Jan 09  2013 baldem.exe
-r--r--r--    1 ftp      ftp          1301 Jan 09  2013 baldem.txt
-r--r--r--    1 ftp      ftp            68 Jan 09  2013 byear.doc
-r--r--r--    1 ftp      ftp        130846 Jan 09  2013 byear.zip
-r--r--r--    1 ftp      ftp         12399 Jan 09  2013 camsys10.lzh
-r--r--r--    1 ftp      ftp           114 Jan 09  2013 camsys10.note
-r--r--r--    1 ftp      ftp           989 Jan 09  2013 canvas.txt
-r--r--r--    1 ftp      ftp        213493 Jan 09  2013 canvas.zip
-r--r--r--    1 ftp      ftp        212496 Jan 09  2013 canvas30.exe
-r--r--r--    1 ftp      ftp           919 Jan 09  2013 canvas30.txt
-r--r--r--    1 ftp      ftp         22863 Jan 09  2013 copyist.zip
-r--r--r--    1 ftp      ftp         22554 Jan 09  2013 copyist2.zip
-r--r--r--    1 ftp      ftp         14117 Jan 09  2013 d50get.exe
-r--r--r--    1 ftp      ftp         14563 Jan 09  2013 d50put.exe
-r--r--r--    1 ftp      ftp          1715 Jan 09  2013 d50putget.doc
-r--r--r--    1 ftp      ftp           908 Jan 09  2013 eps.README
-r--r--r--    1 ftp      ftp         61913 Jan 09  2013 eps.tar.Z
-r--r--r--    1 ftp      ftp        307200 Jan 09  2013 fmpaka.exe
-r--r--r--    1 ftp      ftp          1644 Jan 09  2013 fmpaka.not
-r--r--r--    1 ftp      ftp         58175 Jan 09  2013 getit30.arc.uue
-r--r--r--    1 ftp      ftp           318 Jan 09  2013 k1lib.not
-r--r--r--    1 ftp      ftp        130944 Jan 09  2013 k1lib20.zip
-r--r--r--    1 ftp      ftp         97664 Jan 09  2013 k1lib3s.zip
-r--r--r--    1 ftp      ftp           131 Jan 09  2013 k1utl.not
-r--r--r--    1 ftp      ftp         22528 Jan 09  2013 k1utl.zip
-r--r--r--    1 ftp      ftp          1591 Jan 09  2013 k1voice.doc
-r--r--r--    1 ftp      ftp         78208 Jan 09  2013 k1voice.zip
-r--r--r--    1 ftp      ftp           486 Jan 09  2013 k4.txt
-r--r--r--    1 ftp      ftp            92 Jan 09  2013 k4sysop.txt
-r--r--r--    1 ftp      ftp         40094 Jan 09  2013 k4v301.zip
-r--r--r--    1 ftp      ftp          1296 Jan 09  2013 k4v311.txt
-r--r--r--    1 ftp      ftp         80798 Jan 09  2013 k4v311.zip
-r--r--r--    1 ftp      ftp         68736 Jan 09  2013 mbdoc.arc
-r--r--r--    1 ftp      ftp         28928 Jan 09  2013 mbexe.arc
-r--r--r--    1 ftp      ftp          2298 Jan 09  2013 mbnot.txt
-r--r--r--    1 ftp      ftp        179072 Jan 09  2013 mbsrc.arc
-r--r--r--    1 ftp      ftp         15008 Jan 09  2013 mdf23.lzh
-r--r--r--    1 ftp      ftp           307 Jan 09  2013 mdf23.lzh.note
-r--r--r--    1 ftp      ftp           400 Jan 09  2013 midi10.txt
-r--r--r--    1 ftp      ftp         40960 Jan 09  2013 midiex.tar
-r--r--r--    1 ftp      ftp         20470 Jan 09  2013 midix13.arc
-r--r--r--    1 ftp      ftp         72595 Jan 09  2013 mmadp36.arc
-r--r--r--    1 ftp      ftp         19456 Jan 09  2013 mpu401c.arc
-r--r--r--    1 ftp      ftp          6988 Jan 09  2013 mpudemo.c
-r--r--r--    1 ftp      ftp         12512 Jan 09  2013 msa.exe
-r--r--r--    1 ftp      ftp          3138 Jan 09  2013 msa.prn
-r--r--r--    1 ftp      ftp         81473 Jan 09  2013 mt32ed.zip
-r--r--r--    1 ftp      ftp         37755 Jan 09  2013 pc-midi.tar.Z
-r--r--r--    1 ftp      ftp        131517 Jan 09  2013 pkz101.exe
-r--r--r--    1 ftp      ftp         48117 Jan 09  2013 play2.zip
-r--r--r--    1 ftp      ftp        194560 Jan 09  2013 playpak.exe
-r--r--r--    1 ftp      ftp          1743 Jan 09  2013 playpak.not
-r--r--r--    1 ftp      ftp         40005 Jan 09  2013 pmuser.arc
-r--r--r--    1 ftp      ftp           754 Jan 09  2013 procdemo.txt
-r--r--r--    1 ftp      ftp        374451 Jan 09  2013 procdemo.zip
-r--r--r--    1 ftp      ftp        181964 Jan 09  2013 prsmdemo.exe
-r--r--r--    1 ftp      ftp           572 Jan 09  2013 prsmdemo.txt
-r--r--r--    1 ftp      ftp           367 Jan 09  2013 pvicnv10.txt
-r--r--r--    1 ftp      ftp        472877 Jan 09  2013 pvicnv10.zip
-r--r--r--    1 ftp      ftp          2889 Jan 09  2013 qseq-10.txt
-r--r--r--    1 ftp      ftp        126764 Jan 09  2013 qseq-10.zip
-r--r--r--    1 ftp      ftp          5341 Jan 09  2013 sampdemo.txt
-r--r--r--    1 ftp      ftp        981274 Jan 09  2013 sampdemo.zip
-r--r--r--    1 ftp      ftp           677 Jan 09  2013 srbanks.txt
-r--r--r--    1 ftp      ftp        157763 Jan 09  2013 srbanks.zip
-r--r--r--    1 ftp      ftp        140186 Jan 09  2013 sysport-bbs.dir
-r--r--r--    1 ftp      ftp        151476 Jan 09  2013 tabdemo.arc
-r--r--r--    1 ftp      ftp           430 Jan 09  2013 tt.note.to.system.admin
-r--r--r--    1 ftp      ftp           204 Jan 09  2013 tt.txt
-r--r--r--    1 ftp      ftp         83931 Jan 09  2013 tt.zip
-r--r--r--    1 ftp      ftp        186896 Jan 09  2013 txturdem.zip
-r--r--r--    1 ftp      ftp            77 Jan 09  2013 vfxlib.txt
-r--r--r--    1 ftp      ftp         69165 Jan 09  2013 vfxlib.zip
-r--r--r--    1 ftp      ftp         98816 Jan 09  2013 watchx.exe
-r--r--r--    1 ftp      ftp          1740 Jan 09  2013 watchx.not
-r--r--r--    1 ftp      ftp           483 Jan 09  2013 wjmr224.readme
-r--r--r--    1 ftp      ftp        263901 Jan 09  2013 wjmr224.zip
226 Directory send OK.
ftp> quit
221 Goodbye.
$ # ttfn

Succesful laptop repair – Dell Inspiron 7352

I managed to repair my Dell Inspiron 7352 laptop computer again today. This time, the power connector attaching the motherboard to the tip-and-ring barrel connector had come a little loose, and I had to tape it in place. We’ve had 2 Inspiron 7000 class laptops and both have had similar problems.

This touchscreen-convertible, middling-powered machine is pretty good for web browsing, accounting, and lightweight programming and system administration tasks, but it has some warts. This is the 4th time the power connector has failed in some way in either this computer or another one our family owns. The symptoms are either that it doesn’t charge at all, or that complains about not recognizing the power adapter, and that it won’t charge the battery.

The first time this happened I called Dell and wasted the usual hour troubleshooting what was obviously a hardware problem. I sent the failing computer to Dell Service for out-of-warranty repair, they charged almost $200 and it took 3 weeks to get back. Ugh!

The second time it happened I was super frustrated about sending it back to Dell again so I let the computer just be broken for months. The tip shroud inside the barrel connector had broken off. Then the other Inspiron we own broke with similar power problems. This really ticked me off, so I opened up the case and found that one of the wire leads to the barrel connector had separated.

None of the jury rigged fixes I tried was stable so I researched the replacement part, which was easy as the assembly had a part number printed on it. I found a replacement part for less than $10 on Amazon. Thanks, Dell!

Replacing this turned out to be easy and fixed the power problem on both computers. I even ordered one spare just in case this happened again.

This week the symptoms returned on one of the computers, so after jiggling the connector as much as I dared without breaking it to no effect, I opened up the chassis. I then observed that the motherboard connector was not seated firmly, re-seated it, and put a tiny rectangle of Gorilla Tape over it to hold it in place. I didn’t have to use my spare connector.

Earlier this year I had also replaced the internal fan and swapped the magnetic hard drive for a solid state drive. The hard drive was actually failing, was throwing SMART warnings and had 4096 bytes in unreadable sectors. I got to learn how to use ddrescue in the process of fixing it. It’s like getting a whole new machine when upgrades such as this work as expected.

I like having multiples of the same machine because it makes it way easier to troubleshoot and repair the systems. Dell gets both the stinkeye for some bad engineering decisions and kudos for having systems that are easy to repair.

The Foo Bird

3 explorers walked down a cliffside path to the edge of a lake with a local guide. The guide warned them that this area was dangerous because it was home to the giant Foo Bird.  if they saw the Foo Bird, they had to remain still or face doom, and whatever they did, they should not run into the lake. The explorers laughed off the story from the guide, whom they thought must be ignorant and superstitious.

As they walked along the shore a shadow passed over them, and they looked up to see the enormous Foo Bird, the size of a full-grown elephant. One of the explorers panicked and started running. The Foo Bird swooped in, let loose a huge torrent of feces, and coated the running explorer, who screamed and yelled “It burns!”

He ran into the lake and when he hit the water he caught fire, screamed a horrible gurgling last gasp, and then sunk beneath the glassy surface, dead.

The two remaining explorers looked at each other and then at the guide, for whom they had some new grudging respect. The Foo Bird soared away above the cliffs.

The next night, as they camped by the lake shore, the second explorer arose to answer nature’s call by moonlight. Suddenly the moon went dark and the rush of feathers filled the air. The second explorer panicked and ran, and the Foo Bird swooped in and let loose a giant gobbet of burning excrement, hitting the second explorer right in the face.

He screamed “It burns!” and ran blindly in terror, stumbled into the lake, hit the water, caught fire, and sank dying beneath the surface.

The next morning the last explorer and the guide broke camp and started to march away from the shore to the cliffs, after saying a few words of remembrance for their departed companions. The sky darkened once more, and though the third explorer tried to stay still, she blinked and the Foo Bird swooped down and covered her in poop. The Foo Bird flew away above the cliffs once more.

She did not run and jump in the lake, having witnessed the horrible death of her companions.

The guide wiped her off and helped her change out of the soiled clothes, and they escaped the lake shore without further incident.

The moral of the story is:

.

.

.

If the Foo shits, wear it

My Silent Generation father taught this shaggy dog story to me around 1980 when I was 8.

Joining the stream of LiveJournal refugees

Like so many other people, I’ve just abandoned LiveJournal due to the increasing entanglement with Russia, the degraded security (they turned off HTTPS for crying out loud!) and the terrible implications of Russian law on speech, particularly political speech and that related to queer topics.

As of today the LiveJournal import plugin built into current (hosted or not) versions WordPress is broken. Furthermore the Internet is littered with fragmentary and incomplete advice about how one might work around this problem. Nevertheless, I persisted 🙂

I managed to export my LiveJournal entries by using ljdump and then importing them into a private, off-the-public-Internet WordPress 2.7.1 instance that still supported the old XML LiveJournal import format. Then I edited the posts and comments in the private instance, and exported them back to WordPress format, then imported them here. I may write more about this and share the tools as soon.

New Game: My #1 Google Creations (without searching for your name)

Here's a Google game I just came up with. The challenge is to find Google searches that return something you've created as the #1 entry without using your name. You get bonus points for short or amusing phrases, or if the results say something interesting about you.

Here are some examples for me: