Don’t “Do” DevOps. Solve Problems.

Don’t “do” devops. Solve problems.

That was the advice I gave at DevOps World | Jenkins World last week in San Francisco.  It deserves more time and words than I could give it then.

The popularity and success of “DevOps” marketing is wonderful in many ways.  For one, explaining my resume is easier than it’s ever been.  But, like agile, the success of the buzzword has created new problems, such as unrealistic expectations among management and confusion about who should be “doing devops.”

Agile and devops concepts/culture/practices didn’t evolve in a vacuum.  They grew out of people solving old problems in new ways or needing solutions to new problems.  This context is critical.  Consider the two statements.

  1. Our management created a DevOps transformation initiative, so we implemented an Infrastructure as Code approach for our simplest test environments to show progress.
  2. We saw a lot of human error manually coordinating test infrastructure for all the components in one of our critical applications, this led to testers often not having a working app to test, so we automated provisioning and deployment across all components in order to eliminate the bottleneck for testing.

One of those is “doing” devops and one of them is solving problems.  One of them is easy, safe and ticks a box in a status report somewhere.  One of them is difficult, potentially disruptive and could cause even more delays until it works, but ultimately eliminates waste and brings real value to the business.

I’m not saying to ignore devops ideas–absolutely learn about them!  Then you can apply those ideas to their best effect in your specific context to solve your problems.

But declarative Infrastructure as Code is better than spaghetti automation!

Maybe.  No one wants spaghetti code, but working is better than elegant. I agree that IaC is a great approach, but if it’s not solving an immediate problem and delivering value, maybe it’s not the right time.

But we can’t afford to add risk to our critical application and the infrastructure that tests it!

Anything that’s important to the business will involve risk.  Why can you afford to be working on things that aren’t important to the business?

Surfacing the difference between #1 and #2 to management and getting agreement at the right levels can be difficult.  I think it’s a worthy cause.

 

Walled Gardens and Working Out Loud

I’ve spent most of my career at companies that hadn’t caught up to the new reality of the internet and opensource, yet.  My employers didn’t have social media policies or guidelines for consuming open source software, let alone policies around contributing back to open source.  I was doing interesting work, but I didn’t have permission to share any of it and wasn’t willing to risk my paycheck.  Instead, I shared things in the walled gardens of my employers: their source control, their wikis and their social media platforms.

Whether or not that was a mistake, it kept me from blogging here about what I was doing at my jobs (and eventually led to not blogging for six years) and contributing to opensource projects.

That’s one of the things that excited me about going to work for CloudBees, the open source software company* behind Jenkins. I’d finally be able to share all stuff I was working on!

After a year-and-a-half, I find I’m still struggling to blog more and contribute to (F)OSS. But why?

I think it’s partly that devops has eaten the software world.  It doesn’t feel (to me, at least) that my work is as interesting anymore–everyone’s blogging about it all the time.  I’m fond of saying that I’ve been walking the devops path since 1998.  Now that there’s a name for it, my resume’s easier to explain, but it’s a lot harder to distinguish what I would write from a thousand other blogs and articles.  I’m a manager now and my role is more about coaching and mentoring (and HR form-filling) than finding new technical approaches to building, delivering, and running software.

Part of the advice I give to teams trying to improve their devops practices, to promote innovation and build generative culture, is to share what they’re working on.  I’m constantly surprised by how resistant companies and teams are to meetups, lunch-and-learns, design reviews, and sprint demos. I guess I’m understanding their troubles better as I try to write about my own work.

While at my last job I attended a talk by John Stepper and was won over by his ideas and corporate social media platform.  I put his book on my to-read list and did my best to share (again, inside the company’s walled garden).  But, I didn’t actually dig into his advice. I missed some of the finer points.

I finally got around to reading John’s book, Working Out Loud.  Right from the start, he spelled out the other thing holding me back: habit.  I’ve never built the habit of sharing my work here.  I’ve never built the habit of contributing on GitHub.  In fact, for years and years I was building the habit of not sharing.

So, I just need to do it. Goodbye to the 6 year hiatus from my blog.  Goodbye to contributing only inside walled gardens (yes, CloudBees has those, too).  Hello to building new habits!

Linux Desktop Rebuild Checklist 2 – Web Dev Tools

Following Linux Desktop Rebuild Checklist 1

Sublime Text

I’m a Sublime Text convert. Official docs are sparse, but there are great unofficial (community) docs for Sublime Text.

It’s quite easy to install on Ubuntu/Kubuntu via webupd8.

# now for Sublime Text 3
sudo add-apt-repository ppa:webupd8team/sublime-text-3
sudo apt-get update
sudo apt-get install sublime-text-installer

# old 2 stuff...
sudo add-apt-repository ppa:webupd8team/sublime-text-2
sudo apt-get install sublime-text

LiveReload

Change a file in your project and LiveReload automatically refreshes the browser via an extension (Chrome, Firefox, Safari, Mobile Safari, and Opera with a config change).  Check out “LiveReload” in the Chrome Web Store.  Best out-of-the-box support for the filesystem piece is on Mac (and $$), but the Ruby project Guard and its various plugins allow for even more functionality for no charge (except the effort to figure it out).

On Linux, it’s quite simple, even for someone (like me) who knows very little about Ruby.

Install RVM and Ruby.

$ curl -L https://get.rvm.io | bash -s stable --ruby
$ source ~/.rvm/scripts/rvm
$ rvm use

Install Guard and Guard-livereload

$ gem install guard
$ gem install guard-livereload

Create a sample Guardfile (do this in your project dir).

$ guard init livereload

Then, edit the file for the filenames/patterns to watch. Here’s my simplest Guardfile.

guard 'livereload' do
  watch(%r{.+\.(css|js|html)$})
end

Finally, run guard from your project dir.

$ guard
19:16:14 - INFO - Guard uses NotifySend to send notifications.
19:16:14 - INFO - Guard uses TerminalTitle to send notifications.
19:16:14 - INFO - LiveReload 1.6 is waiting for a browser to connect.
19:16:14 - INFO - Guard is now watching at '/home/mghicks/...'

I haven’t tried Ruby on Windows, but there is a tutorial for using RVM with Cygwin. Git for Windows also has the option to work with Cygwin (selected during installation). I’ve tried Git on Windows with the Git shell, but not with the Cygwin option.

JSHint and JSONLint via NodeJS

Never sweat a missing comma or semi-colon again!  JSHint and JSONLint are the two biggest time-savers I’ve added to my toolset.  Thanks to the NodeJS community, NPM, and NVM, it’s trivial to install and use them.

Install NVM and the latest Node.

curl https://raw.github.com/creationix/nvm/master/install.sh | sh
echo "\n. ~/.nvm/nvm.sh" >> ~/.bashrc
source ~/.bashrc
nvm ls-remote
nvm install 0.x.x (use version from ls-remote)

Install JSHint and JSONLint.

npm install -g jshint jsonlint

Configuring Sublime Text next…

Linux Desktop Rebuild Checklist 1 – Base System

  1. Install via Unetbootin (I’m using Kubuntu these days)
  2. Fix monitors (important to do before next step)
  3. Install synergy
    sudo apt-get install synergy
  4. Add synergy to startup (see http://askubuntu.com/questions/15212/start-synergy-on-boot)
  5. Connect to NAS
    sudo apt-get install nfs-common
    sudo vi /etc/fstab
    
    10.1.1.3:/volume2/data	/home/mghicks/Data  nfs rsize=8192,wsize=8192,timeo=14,intr
    10.1.1.3:/volume2/media /home/mghicks/Media nfs rsize=8192,wsize=8192,timeo=14,intr
    
    mkdir /home/mghicks/Data /home/mghicks/Media
    sudo mount -a
  6. Additional repositories
    sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup
    sudo wget http://www.medibuntu.org/sources.list.d/$(lsb_release -cs).list --output-document=/etc/apt/sources.list.d/medibuntu.list
    sudo apt-get update
    sudo apt-get install medibuntu-keyring
    sudo apt-get install linuxmint-keyring
    sudo apt-get update</span>
    
  7. Additional codecs, fonts, tools, etc.
    sudo apt-get install ttf-mscorefonts-installer non-free-codecs flac libdvdcss2 vlc* mplayer dvdrip
    sudo apt-get install build-essential curl git-core
    
  8. Install Chromium
    sudo apt-get install chromium-browser

    and extensions: Adblock Plus, Reddit Enhancement Suite, LiveReload, Personal Blocklist

  9. Install Banshee (not the best, but the only one I’ve found that *easily* syncs my ratings between players–suggestions welcome)
    sudo apt-get install banshee
  10. Install Shotwell
    sudo apt-get install shotwell

Network Traffic on Ubuntu 12.04 vs Fedora 17

Recently, I started obsessing over network traffic on my (recently upgraded) Ubuntu 12.04 system.  This is what it looked like, with only a few user processes (just chromium and synergyc).

Ubuntu system monitor network history
Screenshot of Ubuntu’s System Monitor Network Traffic

I could never pin down a process that was causing the traffic; netstat -p and nethogs reported the bulk of traffic from an unknown PID.  It may have been malicious, or simply a weird Ubuntu issue.

So, naturally I did what I do when I’m paranoid.  I wiped and reinstalled.  Since I didn’t want the Unity desktop, anyway, I figured it was time to try something new.  Fedora 17, with Cinnamon.

Contrast the Ubuntu traffic above with the Fedora traffic here.

Fedora's System Monitor Network History
Screenshot of Fedora’s System Monitor Network History

That one blip is Banshee pulling the next song from my NAS.  And where I had no user processes running in Ubuntu, this Fedora traffic includes a torrent trickle (10kB/s limits), my media player, and synergy.  Nethogs shows very little traffic from sources it can’t determine.

Nethogs output on Fedora 17
Screenshot of nethogs on Fedora 17

I also have ntop running now. More on that, later.

And finding Cinnamon was a big silver-lining in this whole process.

What’s Causing that Network Traffic on Linux?

I noticed some periodic network traffic on my Ubuntu 12.04 box.

System monitor network history
Screenshot of Ubuntu’s System Monitor Network Traffic

Naturally, I wonder what was polling like that…

Then I realized netstat -ap  will tell me the process using a port, but just doesn’t provide enough info to figure out which is responsible for the traffic.

sudo apt-get-install iftop
sudo iftop -Pp

Iftop is closer to what I wanted, showing traffic, but it didn’t connect the ports with traffic to the PID.  At least it showed me all the traffic was local or to google’s 1e100.net, rather than to any of the other connections netstat showed. But I still didn’t know which process was responsible.

A little googling turned up Nethogs. Much closer to what I needed and easy to install.

sudo apt-get install nethogs
sudo nethogs

Nethogs console output
Nethogs console output

But here I get stuck.  I killed the synergy client and chromium, but the traffic pattern is still there.  Nethogs lumps all the packets it can’t associate with a PID in the ‘?’ row.

To sum up…

  • netstat connect ip-port to pid
  • iftop connects port to traffic
  • nethogs connects pid to traffic, but most traffic is lumped into ‘unknown’

Then, there’s ntop, which runs as a daemon, has a web interface, and produces incredibly detailed reports.  I installed it. We’ll see what it comes up with after it’s run for a while.

sudo apt-get install ntop

You have to create an admin password during the install.

Upstairs Bookcases Finished

N & I needed more space for books, and though the upstairs hall was wide enough to allow a row of bookcases. Are there ever enough bookshelves?

We brainstormed for ideas at IKEA, but the height was a problem. The wall was too short for most half-height media shelves (which are shallow enough) and the hall was too narrow for most half-height bookshelves (which are short enough). Because our rental walls are so dark, we also wanted white to brighten the hall a bit. N scoured the internet for solutions. We found $1200 units, $20 units, and no middle ground. We weren’t happy about it, but settled on four $20 units from Walmart. I wanted to save the $26 shipping cost, so arranged for (free) in-store pickup. I hope I never have to choose to use Walmart again, but if I do, I’m not sure whether the free in-store pickup is worth having to deal with a Walmart store.

Picture of hallway bookcases facing bedroom
Hallway bookcases facing bedroom

Picture of hallway bookcases facing office
Hallway bookcases facing office

To get around the baseboard trim, make it easier to clean, and for easier browsing of the bottom shelf, we built bases to lift the units up about 3.5 inches. These cost around $15 in wood, nails, and paint. Another $20 went into hardware to secure the shelves to the bases, wall, and each other. The total project cost was around $120 and took about 9 hours over 3 days (excluding the delay from ordering 1 bookcase to check it out before committing to the other 3). This would be a Saturday project except for waiting for paint to dry between coats on the custom bases.

Close-up picture of custom base
Close-up of custom base

Close-up picture of bookcases over trim
Close-up of bookcases over trim