serialized.net

Using R and `ggplot2` to Contemplate Relocation

2012-02-13T14:23:00-08:00

My wife and I are entertaining a still heavily “what-if” idea: migrating away from Southern California into the greener (and, it turns out, greyer) lands of the Pacific Northwest. (Portland seems particularly “us”.)

One big problem (and there are plenty of other problems that come up when you’re thinking about a major move) is the weather. Having both lived our lives calling Central and Southern California “home”, we’ve gotten spoiled by sunny, warm days being the norm.

I’ve been using R quite a bit at work, so answering the question of “how bad is the weather, really?” seemed like the perfect place to apply it.

Problem 1: Get some data.

I searched for a few minutes and found that the Weather Underground has historical information online, with handy links to getting it in CSV format. The only problem was that it seemed to only work a year at a time, so I bodged a quick perl script to download what was needed. (Cleaner links to all the code is available in my R github repo)

my ($day, $month, $year) = (localtime(time))[3,4,5];
$month += 1; $year += 1900;

my $uri_a = "http://www.wunderground.com/history/airport/$code";
my $uri_b = "1/1/CustomHistory.html?dayend=$day&monthend=$month&yearend=$year&req_city=NA&req_state=NA&req_statename=NA&format=1";

my %seen = ();

for my $i (1991 .. $year) {
  my $csv = get("$uri_a/$i/$uri_b");
  for my $line (split(/\n/, $csv)) {
    $line =~ s/<br\s+\/>//;
    my $first = (split(/,/, $line))[0];
    next unless defined($first);
    if(!defined($seen{$first})) {
      $seen{$first}++;
      print "$line\n";
    }
  }
}

The closest airport (which seems to be how the historical data is stored) to us is Van Nuys, so I pulled that CSV as well.

$ ./fetch_weather KPDX > KPDX.csv
$ ./fetch_weather KVNY > KVNY.csv

Problem 2: Import the data into R

Thankfully, the CSV data from the Underground is pretty clean. (A few NA’s, but nothing major.) So, basic loading of the file is cake.

citydiff <- function(city1_name, city1_logfile, city2_name, city2_logfile) {

  first_city <- read.csv(city1_logfile);
  second_city <- read.csv(city2_logfile);

  # Attach the name to each table
  first_city$city <- city1_name;
  second_city$city <- city2_name;

  # join the tables together in holy matrimony
  weather <- rbind(first_city, second_city);

  # … and make the city name an unordered factor
  weather$city <- factor(weather$city, levels=c(city1_name, city2_name))

  # Fix up precipitation (it came in as a factor)
  weather$PrecipitationIn <- as.numeric(weather$PrecipitationIn)

  # and parse the date field
  weather$PST <- as.Date(weather$PST, "%Y-%m-%d")

  …..
}

(It’s throwing a warning about a factor that I don’t understand, but it seems to be working overall, so I’m not really sure what’s going on there. I even broke out R’s awesome debugger to no avail.)

Let’s look at the data.

Basic Temperature distribution

The first thing to do is just toss all the temperatures (there are 14,000+ of them since 1991) into a probability distribution, just to see where they fall. This is good use for qplot:

qplot(Max.TemperatureF, color=city, data = weather, geom="density")

The curves are similar, obviously offset from each other by about 20 degrees F; but the Portland curve has a sharper back slope – meaning that temperatures do fall in the 30-50 range, but more often hit 50-90.

Ok, temps are interesting, but let’s see how they fall across the year. We could look month by month, or day by day, but that feels “too coarse” and “too granular” respectively. I chose to break things out by week of the year. The trick here is to use as.POSIXlt() to interpret our Date, then modulo the day of year by 7. The sapply is just to fix the case (leap year?) where a single date would end up in the 53rd week of the year – this just clamps them all to 52.

weather$weeknum <- sapply(1 + as.POSIXlt(weather$PST)$yday %/% 7, function(x) { min(x, 52)})

So, I really fell in love (all over again) with ggplot2 while making this graph. Check it out:

And the code:

ggplot(weather, aes(x=weeknum, Max.TemperatureF, colour=Max.TemperatureF))
  + facet_grid(city~.)
  + geom_point(alpha=1/6, position=position_jitter(width=3))
  + scale_colour_gradient(low="blue", high="red")
  + geom_hline(yintercept=90) + geom_hline(yintercept=60)
  + geom_smooth(method="loess", size=1)
  + scale_x_continuous(formatter=function(x) format(strptime(paste("1990 1 ", x), format="%Y %w %U"), "%B"))
  + xlab("Month") + ylab("Maximum Temperature (F)")

Breaking it down:

using the weather data frame we made, with weeknum on the x axis, and Max.TemperatureF on the y axis and keying the colorization
Facet the graph (make multiple sections), one for each city in our data set
Draw the data with points, at opacity set to 1/6, so points that get plotted a lot will end up darker, and jittered a bit to smooth things out
Color the data points with a gradient, from blue to red. (Nice for intuitive hot/cold display)
Put horizontal lines at 60 and 90 degrees (The points at which it starts to feel “chilly” and “hot”)
Run a smoothed line through the whole thing to make the center more obvious
Convert the x axis from being hard-to-comprehend “week numbers” back to months
Then put good labels on the X and Y axis

It’s really readable, beauitful, and was actually shockingly simple to make. (Other than remapping the X axis. That was fun. The trick is to take a week number, tag it into an arbitrary year with an arbitrary day (so strptime can parse it), then reformat that date as %B, being the full date string. There’s got to be a better way…. but that worked!)

So what does the image actually tell us about the weather? Hopefully, it’s obvious. In Portland, December to February are pretty consistently chilly. In the summer, it occasionally gets hot, but between May and October it’s pretty perfect.

In Van Nuys, it’s rare to be chilly, but in June to October, get ready to run your AC.

It’s not just about the temperature

So, it’s nice not to need a sweatshirt for the temperature, but what about the gray? It can be warm and still depressingly dark outside. Thankfully, the data set includes a cloud cover metric!

Cloud Cover is a number from 0-8, representing the amount of the sky which is covered…. by cloud.

Let’s take a quick peek at the distribution with qplot, sending a smooth line (and leaving the raw week numbers this time) through the data:

qplot(weeknum, CloudCover, data = weather, geom="smooth", color=city, span=1)

So, the cities have similar curves, also offset (by about 3, or 37% cloud cover). In the middle of the summer, Van Nuys typically has around 18% cover, and Portland 43%. So: Portland is Cloudy. Myth Confirmed.

Breaking out ggplot2, we can get a much clearer picture of how cloudy it is:

ggplot(weather, aes(x=weeknum, CloudCover, colour=CloudCover))
  + facet_grid(city~.)
  + geom_point(position=position_jitter(width=0.5, height=0.5), alpha=I(1/6))
  + geom_smooth(size=1)
  + scale_colour_gradient(low="blue", high="black")
  + scale_x_continuous(formatter=function(x) format(strptime(paste("1990 1 ", x), format="%Y %w %U"), "%B"))
  + xlab("Month") + ylab("Cloud Cover")

It’s using basically the same method as for the temperature graph, except I modeled the cloud cover as a color range from blue to black. (Awww, cute.)

You can see something interesting when you compare it back to the temperature graphs: the cold days correlate with the cloudy days pretty well! So, the winter is sucky-cold and sucky-gray all at the same time. At least you get it out of the way all at once (while the days are short, too!)

So how nice is the spring and summer?

It’s just a perfect day

What might a perfect day look like?

Comfortable temperature (60-90 degrees)
Not too cloudy (Less than 50% cloud cover)

It turns out that’s an easy query to make against the data frame.

weather$ideal <- F
weather <- within(weather, {
    ideal[Max.TemperatureF >= 60 & Max.TemperatureF <= 90 & CloudCover <= 4] <- T
  })
weather$ideal <- as.logical(weather$ideal)

Make a new column called ideal and fill it with F’s, then change the value to T wherever those conditions are met – then convert the whole thing to a logical (boolean) type. Easy!

So now we can see where “ideal” versus “not-ideal” days fall across the year. Week-by-week would probably be too noisy for this, so I added a column for month to the data frame as well.

weather$month <- factor(format(weather$PST, "%B"), order=TRUE, levels=c("January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"))

A much simpler incarnation of ggplot this time:

ggplot(weather, aes(x=month, fill=ideal))
  + geom_histogram()
  + facet_grid(city~.)
  + opts(axis.text.x  = theme_text(angle=90, size=10)))

And the results are again interesting; I would not have bet on November having the most ideal days here in Southern California, but it makes sense. And, July-September, over 1/2 the Portland days are ideal ones! (cough, like basically every month in Van Nuys.)

(Of course, anyone who’s been to Van Nuys can attest to the fact that, no matter how great the weather is, you’re very unlikely to have a perfect day in general. Because you’re in Van Nuys.)

So just to quickly summarize the data:

ideal_days <- weather[weather$ideal==TRUE,]
qplot(city, data=ideal_days)

Portland has about 42% of the “ideal days” that Van Nuys does. Which, given everything else that Portland’s got going for it is actually still a pretty good result.

That gives me an idea… I bet I can point R at some Crime, Traffic, School, and Cost of Living statistics to get a more well-rounded picture :)

Screencasting Presentations With Screenflow

2011-11-18T12:42:00-08:00

I’ve been really enjoying screencasting my presentations, and have come up with a nice workflow using the (commercial) Mac app ScreenFlow.

Watch the video for the full details, but the basic process is:

Record the screen while you give the presentation in “presenter mode” (so notes are available)
Also record video from your computer’s camera
Crop down to just the slide portion of the presenter view
Overlay the video on the slides and make it fully transparent (Opacity 0%)
Skim through the video, watching for slide changes as cues
Insert video actions (⌘K) and flip the Opacity between (100%/0%) to show the slide or the video.

At this point, I can come up with a very nice ‘canned version’ of a presentation in only slightly more than the time it takes to simply give the presentation. It’s also a great way to rehearse!

In the video I also recommend the Samson Go Mic, ($40) which I’ve been nothing but happy with.

(The full list of Screenflow Keyboard Shortcuts is good to keep handy.)

A technical side note: the video quality in the beginng of the video is a little off. The actual “presentation” was in 4:3, but the application demo was widescreen. This meant a bit of ffmpeg magic, (attaching ‘letterboxes’ to the sides to make it effectively widescreen) so I could glue the two videos together. When you actually use this method to simply give a presentation, it ends up looking even nicer.

$ ffmpeg -i Presentation.mov -s 560x416 -r 30000/1001 -padleft 46 -padright 48 -vcodec libx264 -vpre hq -acodec libf@c -ac 2 -ar 48000 -ab 192k PresentationLetterbox.mov

Going Fast With GMail’s Keyboard Shortcuts

2011-11-16T17:26:00-08:00

I’m frequently surprised by how many people don’t know about the awesome GMail keyboard shortcuts that are built in to the web application. (And yes, this is the 1 billionth google hit on this topic, but the battle against ignorance is won post by post!)

I use them constantly, and they really help churn through the inbox efficiently and effectively.

First, you need to turn them on:

Settings (Gear icon) -> General -> Keyboard Shortcuts On

Click the link above to see the overall list, but the basics I always use:

x => select a message
e => archive a message
j/k => Move to next/previous conversation

Just those 4 keys will let you BLAST through conversations that don’t apply to you. I often end up doing (x,j,x,j,x,j,e) (select 3 conversations, then archive the lot of them in one swoop.) If it’s spam, or something I know I’ll never need again, replace ’e’ with ’#’ (delete)

# => delete selected conversation

Ok, so sometimes you actually need to read a conversation:

o => open the conversation under the cursor for reading

Our good friend ’x’ will pop you right back to the main view, with the current message selected. So if there’s a lot of messages I need to read, (o,x,j,o,x,j,…. e) will be reading through them all, then archiving the bunch.

If it’s one of those conversations that you think will go on for a while and isn’t relevant, the wonderful m (mute) will pre-archive the rest of the conversation, without you having to deal with it popping back up later! (I have to imagine there are some monster and interminable threads that happen in the googleplex.)

m => mute the conversation

And then there’s replying! Using shortcuts gets you another gorgeous side effect, way lower risk of accidentally reply-all’ing.

When you have a message selected

r => reply
a => reply-all
f => forward

Boom. Tasty efficiency.

Update: @naydichev reminds me of the most important shortcut of all:

? => help (show all commands.)

Using Ogres to Improve Your Designs

2011-10-21T18:01:00-07:00

The problem

Creating new designs (in whatever domain) is always a bit of a tightrope walk – especially as an organization grows. How much do we try and figure out in advance, vs learning and adjusting as we go? How many people do we involve at what stage of the process?

When the design you’re working on will be largely owned and maintained by a 24x7 operations team, this “involvement” aspect can be particularly vital. It’s hard to imagine building a great and coherent design with 20+ people, but 20+ people still have to be full educated about how the system works, (largely) satisfied with the decisions that are made – and, more importantly, have years of collective experience that can help find weak points or make vital improvement suggestions.

An approach

Our current strategy is:

have a small team produce designs/prototypes/working code
have periodic larger-scale reviews for education, analysis and feedback.

What those “reviews” look like is a design that is, itself, undergoing incremental learning and feedback.

Our first stabs at it were fine, but clearly could be improved a lot. We brought the team together, walked through some slides, and did Q&A. In that format, though, there’s a pretty low upper bound on the number of people that can contribute at any moment. 20+ people in a “one person talks” scenario means a lot of standing around, which isn’t a great use of anyone’s time, and isn’t likely to even end up granting much attention and focus.

This week, we attempted at “version 2”, which while still not perfect, was a lot better.

Objectives

Actually make the product great. (Or, barring that, “better.”)
Create more (actual) shared ownership and understanding of the design.
Enhance the social network between the teams so that ad-hoc collaboration becomes more likely in future.
Improve shared vocabulary and methods around value and risk management (which we are bad at doing intuitively)
Make good use of everyone’s time: be engaging and effective.
Have fun!

How It Works, and what was that you were saying about Ogres?

Background

I love the book Gamestorming. It’s full of frameworks and patterns for having more than a few people interact in ways that’s both engaging and effective.

The review we did was based on the Gamestorming game Challenge Cards. In brief, you form two teams: one team creates challenges to the design, the other creates solutions.

The challenge team picks a card from the deck and plays it on the table, describing a scene or event where the issue might realistically arise. The solution team must then pick a card from their deck that addresses the challenge. If they have a solution they get a point, and if they don’t have a solution the challenge team gets a point. The teams then work together to design a card that addresses that challenge.

For fun, and to emphasize the game aspect (and dampen issues with taking criticism personally), I brought in a silly element of making the “solution team” be Knights, defending a castle – and the “challenge team” be gnarly Ogres.

The art and concepts were introduced in some pre-made cards for playing the game, and in the slide deck that introduced the game to the players. (All of that can be downloaded at the end of the post.)

Gameplay

The basic flow of the event (which took about 90 minutes total, each step being timeboxed) was:

Get together, and do a brief review of the current design (focusing on recent changes, and in-progress work.) People get handouts of the architecture to review through the game.
Split into two teams, Knights and Ogres, both of whom headed to their own rooms. (Anyone could choose to be on either team.)
As a group, use the Heuristic Ideation Technique (more on that below) to brainstorm approaches for solution/challenge.
Form smaller groups, and create the cards for the Challenge.
Meet back up, and play through the decks as described above.
Reconvene as a larger group and debrief.

Heuristic Ideation?

This is a clunky name for a simple and cool idea.

I used it to help brainstorm points of potential weakness in the design – breaking things down into “attributes and actions” and “components.”

Attributes are things about the system: Capacity, Security, Reliability, … Actions are things we know can change in the system: Upgrade, Deploy, Replace, Fail, ….

Components cover physical things: Hard Drive, Network Port, Switch, CPU, DIMM, …. They also cover logical things, like Authentication System, Filesystem, User Interface, …

I knew we’d end up with large enough lists that the grid layout called for in Heuristic Ideation wouldn’t work – our boards aren’t that big – so opted for this 2 column layout:

You can apply each attribute or action in the left column to each of the components on the right; by the time you get through all of them, you’ll have really thought through possible fragile areas of the system.

For example, take ‘Filesystem’ as a component; we can discuss it’s capacity, security, reliability, as well as what happens if we upgrade it, run a deploy of new code, if it fails, … and so on. Then, think about all of those attributes and actions applied to the next component! (Say, a network switch.)

Comparative Risk

Instead of just using blank index cards, I threw a bit of layout at it.

The extra fields help scope the risk a bit. (Obviously they are all going to be wild-ass guesses, but they still let us group things by order of magnitude.) Primarily, they’re just there to get people talking in these kinds of terms about relative possible impacts of unlikely things.

The #ragemode tag came from one of our customers who was on the bad end of a miscommunication surrounding some backups, and how fresh they were, (a bit too fresh, in his case), leading to them losing some data. It refers to the fact that some things, when they go wrong, give us the reaction “hey, it’s the internet, these things happen.” Others are infuriating. So it’s an attempt to let us weight possible failures by emotional impact.

When 2 cards are paired up, you can just do the math:

(Time to detect + Time to repair) * Customers Impacted * Expected times per year that this freakish thing might happen * #ragemode == A very fuzzy estimate of “customer-minutes of impact per year.”

Lessons

It was a lot of fun, and met (to at least some degree) all of the objectives I had for it. We’re still going through the cards and learning from them.

However, there was a lot of room for improvement.

First, Heuristic Ideation is awesome. We can use that in all kinds of different scenarios. I’m writing up a list of the Components/Actions/Attributes the teams brainstormed for the wiki, so we can use and further develop them in the future.

The biggest improvement to the game is in the Challenge/Solution dynamic. The Challenges tended to be very specific. (“Filesystem gets corrupted.”) However, the Solutions have to be pretty generic. (“Reinstall the server from scratch and restore from backup.”) Technically, that counted as a point-worthy exchange for the defense, but it didn’t really help us explore the problem.

I’d like to try it this way:

Still have the Knights do their defensive planning, but not actually make cards in advance
When the challenge card is played, come up with (on the fly) the best solution we can currently execute.
- If that is “good enough” (as decided by the players, or arbitrated by the facilitator), the Knights get a point.
- If it’s not, then the Ogres get the point, and then the two teams move on to collaboratively coming up with the solution.

That should keep the focus more on what really matters, and at the level of specifics, and less on the technicalities of “is your vague defensive card applicable here or not.”

A neat thing we discovered was that if someone proposed an attack for which our “defenses” were already good enough, that was a good data point that training/documentation/education needed to be beefed up around that aspect.

One of the Operations Managers had a great suggestion: now that everyone gets the basics, use this framework for fire drills. (Now known as “surprise attacks” or “Ogre Rush!”) Just pop into the NOC with a few of the “Attack” cards all filled out, and make sure people are solid on what the procedure would be to deal with that.

Share and Enjoy

Please feel free to rip off any part of this that was useful at all. I have compiled both PDF versions of the instructional slide deck and playing cards, as well as the original Keynote files.

Knights vs Ogres Game Materials

I made my best effort to track down royalty free art – please contact me if I’ve inadvertently used something of yours!

An Iteration of the Lean Meetings Concept

2011-09-28T09:44:00-07:00

The Personal Kanban blog ran a great pair of articles on “Lean Meetings” (part 1, part 2.)

I recommend reading through them to get more context for what follows, but I’ll re-share one of the biggest insights: the point of a meeting is to have a conversation. Pre-set agendas feel like best practice, but they can get in the way of that ‘true goal’ more often than not.

Control, agendas, and procedures impeded conversation, focusing on the
structure of the meeting rather than the topics at hand. If you want
people to engage in and feel they’ve derived value from your meeting,
make them feel respected, not restricted.

After having orchestrated a few meetings using this idea, some helpful practices have emerged, which seemed worth sharing.

The basic framework still holds:

Framework: Draw a Personal Kanban
Personal Agendas: Invite all attendees to write their topics on sticky notes
Democratization: Invite all attendees to vote on the topics on the table (each person gets two votes)
Group Agenda: Prioritize the sticky notes
Discuss

The riffs on that list are:

Framework

We found that converting (To Do, Doing, Done) to (To Do, Goals, Doing, Takeaways, Done) has been helpful.

Here is what those new substeps entail:

Goals

Before we actually start discussing a topic, create a shared understanding of what we hope to get out of it. Paraphrasing David Allen, “what does ‘wild success’ look like at the end of this?” What is the value we need to deliver to our customers or organization by the time we’re done?

Concretely, this can come down to basic things. Plan fall employee event? The goals might be “Define vision, date, coordinator, and budget.”

It can also be fuzzier: “Understand each other’s feelings about this topic.”

In practice, it’s been interesting how hard it can be for a team to get this articulated – and there’s a high correlation between “hard to find goals” and “conversations that would wander and go nowhere.”

Mechanically, as is hinted at in the diagram above, we found it natural to add a new sticky with the goals on it as a “rider” to the original topic note. That kept the goals clear and visible to everyone, and they were referred to frequently.

Takeaways

This is probably implicit in what it means to move a topic from “Doing” into “Done”, but it’s helpful as a reminder. “Who is responsible for taking things away from this item?” It’s a good hook for recording things, especially if there’s no formal note-taker.

We also use this as a hook to make sure to consider how any conclusions (or lack thereof) need to be communicated to the wider group that was not in attendance at the meeting.

Personal Agendas

At this point we’ve only really used this idea as-written, but I’m curious to play with various brainstorming/gamestorming or focusing techniques to help generate higher-value items for the team to choose from, rather than those that happen to be top-of-mind for the participants (or have been recorded in their own trusted systems.)

We did bring an idea from Kanban in – ‘item types.’ This was helpful at an all-day monthly meeting one of our teams have. We created a special designation (color of sticky works, as pictured above, or a separate section in the backlog) for items that are ”Today Or They Die” – things which will happen between now and the next meeting, regardless of if we discuss them or not. If it’s December 1, and your next Meeting is Jan 1, you should probably make your Christmas plans.

This can help with voting; those items may still not end up getting selected for discussion, but at least the team has a shared understanding that we’re letting the opportunity pass.

It’s mentioned in the linked articles, but the feature of being able to discover new items “on the fly” and work them seamlessly into the backlog is amazingly useful. In life things are fundamentally interconnected so it’s to be expected and encouraged that we’ll discover or create things that need discussion as a function of discussion. Agendas hate this idea.

Democratization

The ‘everyone gets N votes’ method works well. We also experimented with a few other methods of prioritization:

“If you only get to discuss one thing today…”: each member gets to select one item to bring into To Do. Perhaps more Socialist than Democratic, but it was a nice way to start off the day.
“Lightning Round”: Are there any topics remaining in the backlog we can get through productively in 5 minutes or less? This was a fun way to get back into things after a lunch break, but somewhat risky – one of the items selected turned into a 45 minute discussion.

One challenge with voting early in the discussion, particularly for all-day meetings, is allowing priority to evolve over time as perspectives and time pressures change. Using voting just to bring a few items into the ready queue, and then voting again (from scratch) to repopulate when it empties, seemed to work well.

Conclusion

This way of organizing meetings has really enhanced the quality of our discussions and the value they deliver. It ends up feeling much more natural – still providing the compass and context we’re frustrated about not having in a truly chaotic meeting, without the rigidity and creator bias that an agenda brings. After two separate meetings, participants made comments like “that was the most productive and engaging meeting I’ve ever been a part of.” That’s reason enough to continue exploring!

Thanks again to Jim and Tonianne for the education and inspiration.

Getting more Signal from your Noise

2011-02-24T00:00:00-08:00

At SCALE 9x I presented a talk in the DevOps track called Getting more Signal from your Noise.. You can download the slides (with notes, without notes), and this is a companion post which contains links and further information. I’d recommend reviewing that before reading more, as I won’t rehash what I covered there. Due to the 30 minute timebox, I cut out even discussing a few large areas that I can address (briefly) here.

The Point

Data + Open Source is an explosively growing field.
The complexity of our software systems – the way we deliver applications and services – is exploding.
The number of people trying/needing to build systems like this is exploding. (Not just Flickr and Facebook anymore.)
The demands on our time and attention are exploding.
The data that’s available to us from our infrastructures AND the “world around” (finance, customers, social media, etc) is exploding

The other thing that’s exploding are the number of businesses who will provide you some form of “shrink-wrapped” delivery of the kinds of tools (or at least results) discussed here. Depending on your business, going DIY and duct-taping together what you need may be the wrong idea. However, there are a few major reasons that DIY can be a good idea. * Flexibility: We are learning every day the kinds of things we need our systems to tell us in a hurry. Being able to quickly tune them makes a difference. * Latency and Connectivity: When you’re using a system for real-time decisionmaking, at least having it on-premise means you can throw GB/sec at it, and have results in seconds.

The Data Stack

In the talk, I introduced a model for thinking about what types of functionality the different tools available provide.

Collect
Transport
Process
Store
Present

Many tools provide just one part of this stack, but more than that are ‘hybrids’. Getting the data you need often means mixing and matching.

I called out graphite, collectd, OpenTSDB, reconnoiter, esper, and protovis as being particularly worth at least knowning about.

Other projects and ecosystems worth studying:

Hadoop

The literal elephant in the room that I discussed only tangentially, Hadoop (and the huge family of tools around it) can be an incredible asset to learning more about your world via storing/managing/questioning your data. Cloudera remains a great source of both software and education, and is a good place to start.

It’s now possible to get real-er time information from a Hadoop system, but historically it’s been essentially for things that are more time sensitive on the 1-day/1-month time range. (Trends, capacity, etc.)

Log Processing and Management

The state of the art with log management used to be syslog + logrotate = done. There are a lot more options today.

Many people are using HDFS (before or after processing) for both it’s scalability, resilience, and ability to integrate with the larger Hadoop family. Orbitz (awesome at sharing, first graphite, now this) have a great presentation about ‘Hadoop for Logs’ which is a good overview of what you’d be getting into.
Logstash and graylog2 can bring some of the utility of Splunk without the (ahem) cost structures. While graylog2 stores data in MongoDB, Logstash can optionally use ElasticSearch, which is a nicely packaged “throw text in and RESTfully search” engine. (Thanks to the author of Graylog2, @lennart, for clarifying that.) @timetabling suggests you can glue ElasticSearch to MongoDB, so that when the data changes you index it – but out of the box, that’s not an option.
A new entrant, Luwak runs on top of Riak, so you get the availability + scalability and a Map/Reduce interface to boot.
Flume and Scribe (as well as messaging systems like RabbitMQ) can replace syslog as the way to shovel raw logs around.
Google’s Sawzall is more on the processing side, but it allows you to describe patterns of information you want from your logs, and then in a map/reduce-like way aggregate them.

Data Analysis

A basic-to-advanced knowledge of statistics is becoming essential. There are powerful tools (like R, and many libraries available for different languages like SciPy) – but if you don’t know what operation you want them to do, they won’t help.

Our tax dollars have actually provided a pretty useful introduction, the NIST Handbook. I have been overwhelmingly happy with how useful the book Data Analysis with Open Source Tools has been – it takes some decent energy to get through, especially if your background is not so math/dev heavy, but it’s insanely rewarding.

Machine Learning

“Machine Learning” is still a pretty intimidating thing to Google for unless you’ve got a C.S. PhD. However, it’s starting to be packaged and democratized enough that mere mortals can start to play.

Apache Mahout has a lot of potential to be of tremendous use here. Many people are using it more in text-related spaces, but the ability to find patterns and trends across multiple disparate systems is exactly what we need for things like botnet combat. I’ve just started looking at this and so far am very inspired to dig further.

People to Watch

This is a by-no-means-comprehensive list of the people whose tweets, software, and writing I’ve found useful to keep exploring the possibilities and pitfalls here:

Jordan Sissel (author of Logstash, Grok and Fex)
Strata (The ORA Data Conference) Speakers: Lanyrd List (and Lanyrd’s awesome Writeup List).
Dr. Neil Gunther
Patrick Debois
Regis Giadot
Theo Schlossnagle
Percona (Including the very nice implementation of Dr. Gunther’s Universal Scalability Law)

And so

Short of a series of books, updated monthly, all I can really provide is an appetiser. Hopefully you’ve been inspired to move on to your own data hacking “main course.”

Testing Perl code that runs commands

2010-09-22T00:00:00-07:00

At the September Los Angeles Perl Mongers meetup, Tommy Stanton presented on an in-progress bit of code he’s working on, App::Git::HomeSync. (Presentation)

As you’d hope and expect from something headed CPAN-ward, he’s got lots of tests. As you might have guessed from the name, this module needs to run git quite a bit, with different command line arguments. Tommy’s approach to testing this is good – ship the module with some “fixtures” (a directory in a known state which gets unpacked into a temp directory) and then run the command line app in that directory.

There is another way to approach this, and I realized I didn’t have any “open sourceable” code which demonstrates this technique. I got a lot of the way through writing this before realizing I have blogged a more basic version of this idea before, but this is a new-and-improved take on things, with much deeper examples.

`tl;dr`

Use IPC::Run to run your command line apps
In your test code, intercept the calls to IPC::Run::run and return your own data, based on the command line used
Store sample command line output inside the test file using Data::Section

I’ve stashed a full, functional example of this idea in my Acme::System repository on GitHub.

Code Walkthrough

Using IPC::Run in your code

First, there’s the module code itself. This module does 2 very stupid things.

It returns the sum of all the PID’s (Process ID’s) on the system, and it calls ps to get this information.
It returns the value of one of the columns from the vmstat tool.

The important thing is to use IPC::Run::run to actually run the code, instead of a blind system() call. Because it’s a module call, and has a very simple interface, it’s much easier to mock it (“override the functionality with ‘fake’ functionality) for testing.

This is in lib/Acme/System.pm. Here’s the pidsum method (the vmstat_col method is basically the same, check out the full code if you’re curious):

    =method pidsum
        Return the sum of all the PID's on a system
    =cut

    sub pidsum {
        my @ps_cmd = ("ps", "-Ao", "pid,cmd");
        my ($stdin, $stderr) = (undef, undef);
        my $ps_output;
        IPC::Run::run(\@ps_cmd, \$stdin, \$ps_output, \$stderr);
        return sum(
            map { /^\s*(\d+)/ ? $1 : 0 }
            split(/\n/, $ps_output)
        );
    };

So, as I said, stupid code. I only care about getting the output, so I pass in undef for the other values – IPC::Run::run wants you to supply them anyway.

However, that little sum/map/split thing sure looks fancy. How much would be willing to wager that it’s bug-free? Probably not much. How would you even test code like that?

So, let’s cheat – run ps just the once, stash the results, and use those for testing from there on out.

Storing the command line output results

Check out the the whole test file to see what’s going on overall.

There’s a few bits of weirdness, for sure. Down the end you’ll see:

    __DATA__
    __[ ps -Ao pid,cmd ]__
    PID CMD
        1 init [2]
        7 [khelper]
    ….
    1887 /usr/sbin/acpid
    __[ vmstat -n 1 1 ]__
    procs ———–memory———- —swap– —–io—- -system– —-cpu—-
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
    0  0    648  85932 194240 119124    0    0     0     5    2   51  0  0 100  0
    __END__

This is the format that Data::Section wants. You just need to stash each command you run between the underscored square brackets, and follow it with some sample command line data.

The tricky thing is this – Data::Section was built to be used with Modules, not “plain jane .t files.” So to make that work, you need to:

Give your test a package name. I chose to just stick a Test:: in front of the module name.
Give Data::Section an instance of that object to stick it’s methods into.

    # magic; Data::Section wants this to be a module, not a test file.
    # trick into thinking this hashref is a member of Test::Acme::System
    my $data = {}; bless $data;

This is the only part of this whole technique that feels truly hacky. If anyone has suggestions for better ways to manage this, let me know.

Overriding `IPC::Run::run` in a test

There are lots of ways to override a module’s methods; I have had good experiences with Test::MockModule. It’s pretty easy.

Write a callback that emulates `IPC::Run::run`

Here’s the code that makes that evil hack above worthwhile. Here’s all you have to do to recover those canned program execution results, using the section_data method provided by Data::Section (and that hacky $data reference):

    sub mock_ipc_run {
        my($cmd, $stdin, $stdout, $stderr) = @_;
        $$stdout = ${$data->section_data(join(" ", @$cmd))};
    }

“Mock” that in place

    # override the real run object with one that will use the __DATA__ block
    my $module = new Test::MockModule('IPC::Run');
    $module->mock('run', \&mock_ipc_run);

Actually doing the testing

At this point, test away!

# Actually "run the tests", using the canned results from the __DATA__ block
cmp_ok(Acme::System::pidsum(), "==", 8278);

cmp_ok(Acme::System::vmstat_col("buff"), "==", 194240);
cmp_ok(Acme::System::vmstat_col("si"), "==", 0);
cmp_ok(Acme::System::vmstat_col("cs"), "==", 51);
cmp_ok(Acme::System::vmstat_col("swpd"), "==", 648);

done_testing;

So I can (independently) calculate what the results should have been, given the arbitrary data I’ve saved in the __DATA__ block, and test based on those values. Awesome.

jbarratt@dev:~/work/Acme-System$ prove -l t/00-fakerun.t
t/00-fakerun.t .. ok
All tests successful.
Files=1, Tests=5,  0 wallclock secs ( 0.01 usr  0.03 sys +  0.00 cusr  0.10 csys =  0.14 CPU)
Result: PASS

Trust me, the normal code will still actually call the system

Just for fun, I threw in a script that actually uses this module to get live data:

    use Acme::System;

    print "Sum of all system PID's: " . Acme::System::pidsum() . "\n";

    print "Current CPU user time: " . Acme::System::vmstat_col("us") . "\n";
    print "Current Free Mem: " . Acme::System::vmstat_col("free") . "\n";

And sure enough, if you run it, the data is getting updated live. IPC::Run really is working on the live system.

jbarratt@dev:~/work/Acme-System/lib$ ../bin/live
Sum of all system PID's: 367273
Current CPU user time: 0
Current Free Mem: 70448
jbarratt@dev:~/work/Acme-System/lib$ ../bin/live
Sum of all system PID's: 367291
Current CPU user time: 0
Current Free Mem: 70556

Wrapping it all up

Other than the hackish trick to get Data::Section working in what’s not really a module, this code is really clean, readable, and easy to maintain. It works well for pretty much any module you might care to use instead of IPC::Run – there are lots of options, but as long as you use one of the module-ized ones, you can hook the module name and go from there.

Especially if you write lots of sysadmin tools, and especially if they have costs or risks associated with running them (fsck? rm -rf?) this technique can be a lifesaver. It’s only as good as the inputs you give it, though. I made a mistake the first time I figured this workflow out of grabbing an output which, in real life, ended up with more whitespace than I’d accounted for, because the counters had gotten bigger between when I snagged my “output to test with” and when it was running on “live output.”

I hope it helps, and if you have any suggestions about how to improve the technique, let me know (or send a pull request!)

Up-Up-Updated ‘Inbox Zero with Mail.app’ Technique

2010-09-06T00:00:00-07:00

In March and December of 2009 I described how to do the Inbox Zero method with Mac Mail, often known as Mail.app.

Since publishing those articles I’d been fed up with the performance of all the methods I’d been using, and finally switched over to a true beast of a mail processing machine, Mail Act-On. The native plugin performance was great, and I was pretty happy with this solution.

The only thing I wasn’t happy with was that I was still only using the single feature – the ability to, with a hotkey, move mail into an archive folder. And for this, I’d happily paid.

However, I was recently contacted by the developer of the perfect plugin for me – Archive, a button for Apple Mail.. It does Archive and nothing else. It’s a native plugin so it’s fast, fast, fast. You don’t have to jump through any of the hoops of Applescripting and Service menus and Quicksilver.

I’ve been using it for a few weeks now and it’s been perfect – clean, simple, and exactly what I wanted all along.

It’s free, and you can download it from the author’s site.

Why using packages makes sense in a configuration management world

2010-09-01T00:00:00-07:00

I woke up this morning to a discussion on twitter between two of my favorite internet people, Andrew Shafer and R.I. Pienaar.

Andrew I know from his previous job with Reductive (now Puppet Labs) and I love what he has to say. (I really liked his DevOps Cafe episode – in particular making me change my opinion about “commitments” in Agile contexts.)

R.I. is a force of nature – his blog is great, and full of years of hard-earned wisdom, and mcollective project is something I can’t wait to roll out.

The discussion centered around the question that I’ll paraphrase:

In the era of configuration management tools like Chef and Puppet, what value do packages provide? What are the pros and cons of packaging?

It was clear that both of them were feeling the pinch of expressing themselves in 140 characters. It’s a topic I’m pretty passionate about, after 15+ years fighting to keep systems under control, so I figured I’d write up my take on it.

Packages vs `wget && tar`

What are the pros and cons at the utility level?

(As dogma-free and objective as I can be, of course…)

The cost of building packages

Let’s start with the only downside I can think of to having to build packages – it’s an extra step, and takes some time.

Packaging your own code is easy – you solve it once, and then have something like Hudson or BuildBot take it from there. However, packaging upstream code that’s not in your distro is a pain in the butt. That’s a given.

Both of these get worse if (like me) you’re stuck running multiple distributions. Right now we have to build .rpm, .deb and Solaris packages.

Depending on what language you’re using, there might be some tools that help package things the right way. For Debian+Perl, for example, dh-make-perl is getting to the point of being awesome and very usable.

One way to get packages for upstream stuff that’s not very painful is with a tool like CheckInstall – you do the equivalent of a make install one time in a sandbox, and that gives you a package you can install at will and get all the benefits I’ll elaborate below.

No matter what, it’s an extra step.

@AshBerlin points out that, in the case that you’re managing some upstream software, that this is a cost you have to take on for every version they release, not just a 1 time cost.

There is no question that it’s ‘easier’ to do a make install than to build a package (every time the software is updated) and get that installed.

So here’s why that’s worth putting up with.

The value of building packages

Redundant, Version-aware Repositories

What’s can go wrong with code that looks like this? (Arbitrary link I happened to have in my .bash_history)

$ wget http://yuilibrary.com/downloads/yuicompressor/yuicompressor-2.4.2.zip

What happens when the version changes? You have to update your configuration. This can be a good or bad thing, but in some cases you really want $latest to be installed, rather than the hard-coded version someone supplied the last time they edited the configuration manifest.
What happens when the yuilibrary folks change their (arbitrary) download URI’s?
What happens if http://yuilibrary.com is down the next time you want to do a build?
What if you want to be shipping yuicompressor-2.4.1.zip? Is that still available for download?

When you want to install a package, the more reliable you can make the process, the better. Upgrading all the servers in a cluster? You want all the servers to be upgraded. Trying to bring a new server online? You want to be able to do that with a very low probability of anything going wrong. (The more you can trust your deploys, especially in the age of automated infrastructure, it saves you money to be able to bring servers up as “Just in Time” as possible.)

The “repo” model provided in at least the Red Hat and Debian packaging system handles all of these cases really, really well.

You can provide a list of repositories that an attempt to install a package will try from until they find one working
It’s a trivial sysadmin task to have several repos with the same content available. Each one doesn’t even have to be fancy and “HA”.
It’s 100% predictable (and an implementation detail you don’t have to worry about) what will happen when you say you either want a specific version of a package, or the latest version.

Built-in “tripwire”-like functionality

Apt and Yum both keep checksums of all the files on the system installed by packages. So at any time you can ask “have any files supposed to be managed by packages been modified”?

debsums
rpm -V -a
pkgchk on Solaris (/via @builddoctor)

This is a useful thing to know for security reasons, of course. However, it’s even more important for helping people adapt their behaviors.

In an environment that’s moving from “not configuration managed” to “configuration managed”, and the status quo has been “modify the files on production servers”, it’s great to be able to get a nagios alert that one of the servers is now out of configuration, check the sums, and find out exactly what file(s) and when were modified.

(If you couple that with a nice ‘everyone logs in as their user and sudo’s when needed’ policy, you can find out exactly who and when, as well.)

The package manager knows how files got on your system

Knowing what files got spewed into your system from your average make install is pretty predictable, but certainly not always.

This is useful for 2 cases:

Troubleshooting, and knowing where to make changes when you find the problem

The server keeps throwing 500 errors. Why? Ah, an untrapped exception in /some/file/deep/on/the/system. Ok, I can fix that. Where do I need to go fix that? dpkg -S <filename> tells me the exact package responsible for that file.

$ dpkg -S /usr/bin/factor
    coreutils: /usr/bin/factor
$ dpkg -S /usr/bin/facter
    facter: /usr/bin/facter

That’s one area in particular where configuration management systems can add an extra layer of value. R.I. actually wrote a tool that helps you discover which puppet module is responsible for configuring a given resource on the server. (localconfig.yaml parser) So if the answer is, rather than “it’s something shipped with a package”, “it’s a config file that puppet wrote with your module called my_module”, you can easily find and fix it.

Uninstalling

I’ve had some concrete problems from this, where I did an upgrade, and cruft left over from the previous version interfered with the new versions. For dynamic languages which build up large library trees, this can be particularly nasty, since default search paths might end up including remnants of an old version.

When the package manager knows the location of every file, it can rip them out as happily as it put them in.

Dependencies are built in

At the level of configuration management, I really care about the application I’m configuring.

I want to run lighttpd. I tell the configuration manager to install it. I don’t want to have to do a research project to find all the supporting libraries required for it. Also, I really don’t want to track down the -dev versions of all of those libraries by hand.

This is especially important for upgrades – if an application starts using a new library after an upgrade (or depends on a newer version of a library when upgrading) that’s all handled (and expressible) at the package layer.

Discovery of available updates are built in

This is one of the most compelling reasons.

If you’ve got a system with 9 tarball’d packages,

What versions are currently installed?
What software has updates available?

It’s bad enough if you’re installing software you know about, but I assume we’re going to be using a distribution at some point. You also REALLY want to know about updates your distribution is providing, right? Especially when it’s things like critical kernel issues, openssl problems – anything that can be remotely triggered, at the very least.

Knowing what versions are available can be easily automated. You can use things like

to handle this, and even do things like automatically install security patches if you’d like.

Since you need to solve this ‘sysadmin problem’ at large anyway, why not leverage the tools and practices you build around this to learn about your own software, as well? It’s just as valuable to know that there’s a new build of apache2 available from the core Debian repo as it is to know that our_custom_app, which we expected to be at version 2.6 everywhere, is pinging because host25 is still running 2.5.

Cryptographic signatures are built in

Packages give you a way to trust that the bits you’re about to install are the ones you should be installing.

In this time of huge malware infestations, attempts to trojan even things like the linux kernel, and a large black market for “owned boxes”, you have every reason not to trust that the software you download isn’t compromised in some way.

Most places you can download tarballs also post checksums of what those files should be. There’s two problems with this:

If you can’t trust the place the download is hosted, why would you trust the sum?
That adds a lot of complexity to the automated installer.

You go from

Download the package
Decompress, configure, make

Download the package
Scrape for the latest checksums
Verify the checksums
THEN install the packages

Packages solve both problems.

They distribute a public key asynchronously from the hosting of the packages. Unless you are owned already, you can verify that a package was signed by the person who holds the matching private key.
Checking signatures is built in. Your package install will fail if something goes wrong.

Binary-identical code on every server

There are lots of things that can go wrong when you try and build a package from source.

You may not have some development libraries you need. Great, now you’re stuck managing those explicitly in configuration management as well. This also means that
Every server has to have the full stack of software needed to be able to build your software
Compilation may fail, especially if the package was updated from the last time you tried to build it.
Things that varied from build-to-build need to be accounted for when troubleshooting. “Huh, redis on host25 keeps crashing. Do we need to rebuild it?”

In general, it’s very nice to be able to completely decouple the tasks of

Creating a ‘build’ of your software and stack
Deploying a ‘build’ of your software and stack

Environment Support

One reason we like packages and repos is that it lets us define a configuration as:

A set of which packages
A set of configurations of those packages

Then when we’re working in a topic or feature branch, we can create a repository just for that branch (this is also automated), and the repo configuration is the only thing that needs to be modified in the configuration management code.

Also, because you only need a subset of all the packages you need in a repo, this lets us “stack” them.

Prefer my project repo (which only has my project-sensitive packages getting built into it, the stuff I’m modifying)
Fall back to the production repo for anything else you need

The End

Packages may add some pain and complexity up front to the install process, but they add a tremendous amount of value to the “lifecycle management” of your applications. Most of the hell we go through as people running servers doesn’t happen the first month we set them up, it’s month 6, 18, and 24 that are the real problem. And those problems (“servers are graveyards of state”, etc) that make Configuration Management the right thing to do in the first place.

Use them. Love them.

Deploying self-contained Perl Dancer applications

2010-08-30T00:00:00-07:00

I’m really enjoying using Perl’s Dancer for building lightweight web applications. It’s heavily inspired by Ruby’s Sinatra framework, but clearly Perlish.

The only thing I’ve been bothered by so far is getting my applications from my development environment out to production. It’s pretty easy to actually do deployments in terms of actually getting your app up and handling web requests, but shipping the software to the remote system has always bothered me.

Update: I wanted to clarify that Dancer itself is very lightweight and has as close to zero dependencies as is reasonable to have.

However, the Dancer apps that I’ve been building tend to require a lot of pretty fresh CPAN goodness. (Task::Plack, Moose, DBIx::Class, AnyEvent, and more.) This is a problem if you’re trying to avoid using CPAN as root to just install system packages, which I like to avoid – it makes systems harder to define with something like Puppet, and can cause weird interaction problems between multiple applications running on the same machine when they use the same modules.

Dancer builds you a nice starter application container when you run dancer -a – it’s made to look like a Perl module, with a Makefile.PL and everything. This initially excited me, because I could just turn it into a debian package it with something like dh-make-perl. Here’s the problem – when you perl Makefile.PL && make dist none of your non-perl-module assets make the trip. I’m not really interested in deploying applications that don’t have templates, CSS, Javascript, or images.

From what I can tell in the docs and on IRC, most people solve this by just checking out their application from their version control system on the production box.

This is more or less this idea:

If you’re like me, and living life with hundreds or thousands of servers, that approach doesn’t really work. It also doesn’t solve the first problem above of how to handle all the dependencies (or dependency clashes on a single machine.)

If you’re running lots of servers, you’ll end up with this problem. You install your first app on a single box or two, and it’s running along fine. Then the people come, and you need more horsepower. Time to build a new box.

Oops! DBIx::Class isn’t passing tests right now. I guess we can’t deploy a new server. Or, more subtly, everything installs, but when you add your app to the load balancer, something is “wacky” one 1/2 of the web requests. Pleh.

You really don’t want to be in that trap if you’re trying to auto-scale your app on, say, EC2.

Ok, so we don’t want to install from CPAN on the fly.

I really like brian d foy, and he’s got a strategy to handle this problem: run your own CPAN!

This is actually a pretty good idea, and in some environments, I’d be using it right now.

That turns the model to this:

We have our own minicpan to use to buffer the volatility of the CPAN. Upgrades can happen to minicpan when we want and need them to. If the minicpan didn’t change, we can install our application on as many servers as we want, and trust they’ll be getting the same code.

There’s still a big problem with this: what if you have multiple apps that have different dependencies? You can’t use CPAN’s “new hotness” for one simple app that could really use from it, without worrying about if all your other applications will be able to work with all that new code. So we’ve given ourselves the ability to add a ‘buffer’ between our usage and CPAN’s potential volatility, but we haven’t bought much independence for our applications.

Brian’s started working on extending minicpan to handle multiple minicpan’s.

However, there is another approach, which brings some other nice features as well.

Enter Shipwright.

Shipwright lets you keep a local, version-controlled copy of all the source (from CPAN or otherwise) that your application needs. It keeps the information about where all that source “came from”, be it CPAN or a local file, so you can keep it as up to date as you want to, and when you want to. It nicely decouples the “application building and packaging” problem (“make me a new package”) from the “application maintenance” problem (“update some components.”)

So now we’ve got a version controlled “CPAN Cache” per application we’re managing.

The other things I really like about Shipwright are:

It doesn’t just handle the CPAN problem – it also makes your code into a little self-contained unit which can be dropped into any directory on the target system.
You can tweak any module’s “build process”. (As I take advantage of below.) If the CPAN installer doesn’t work the way you want or need, you can do some pre/post-install hacking. Again – in a nice, version controlled and repeatable way.
You can ship autoconf style applications along with it. Want to also deploy a patched version of nginx, or a redis server? You can do that here. (I wouldn’t, but you can.)

Essentially, all I’m saying is “use shipwright” – but there are a few tricks to make it work for Dancer applications.

MANIFEST

First, you’ll need to make sure all the files you care about in your Dancer app are going to be included at all. This means getting them in the MANIFEST file. I just did a simple find . -type f > MANIFEST and cleaned out the entries I didn’t need or already had. If you’re doing this a lot, or modifying the file contents of your applications a lot, I’m sure there’s a more elegant approach.

shipwright `build` file

One of the nice things about shipwright is that it allows you to tune up a build script for everything you’re installing.

Even though the Dancer packages now contain all your files, they still don’t know where to get installed.

Normally the scripts/MyApp/build contents look like this:

install: %%MAKE%% install

If we add a simple extra step, that gives us a copy of all the module’s assets rooted off our package’s ‘/www’ path.

install: %%MAKE%% install ; cp -av . %%INSTALL_BASE%%/www

Walkthrough

First, you’ll need to install Shipwright. I am in love with the mighty combination of local::lib and cpanm, so I’d recommend starting there.

Build your Dancer app

Once you’ve fixed the MANIFEST file as described above, you need to build a distribution of your Dancer app.

$ cd ../MyApp
$ perl Makefile.PL
$ make dist

Prepare the ‘vessel’

I’m going to be doing all the work in a directory called ~/home/work/shipwright. I’m also using the git shipwright backend here – it works with svn, plain filesystem, and other options as well.

# you might need to mkdir -p "$HOME/work/shipwright/" first.
$ export APPNAME="MyApp"
$ export SHIPWRIGHT_SHIPYARD="git:file:///$HOME/work/shipwright/$APPNAME-vessel.git"
$ shipwright create

Ok, now you’ve got the vessel. It’s time to load it full of CPAN’y goodness. Since this is a tutorial for Dancer, I’ll include some of the basics I like to have when deploying Dancer apps.

Fill the vessel with software

I’m using –no-follow here because I had some errors trying to follow dependencies on my internal applications that I also install via distribution file. If you’re only loading CPAN modules from your Dancer app, you can take this off.

$ shipwright import cpan:Template cpan:Dancer cpan:YAML::XS cpan:Task::Plack
# put the full path, and right version number of, the file here
$ shipwright import file:~/work/$APPNAME/$APPNAME-0.004.tar.gz –no-follow
# REPEAT importing for any of your other in-house modules/code
$ cd ~/work/shipwright
$ git clone $APPNAME-vessel.git
$ cd $APPNAME-vessel
$ vi scripts/$APPNAME/build
# change the install line from:
#       install: %%MAKE%% install
# to
#       install: %%MAKE%% install ; cp -av . %%INSTALL_BASE%%/www
$ git add scripts/$APPNAME/build
$ git commit -m "tweaked build script for $APPNAME"
$ git push origin master

Build the vessel

Cool. Now we’ve got a self-contained, versioned repository. Time to build it.

$ ./bin/shipwright-builder –install-base ~/work/shipwright/$APPNAME –force

The –force is because some modules don’t pass tests. Shipwright does have a ways to go with dependency management (or I’m doing something wrong) – if I’ve install a module into the ‘vessel’, sometimes other modules that depend on it can’t use it at build/test time.

Now you’ve got a directory (~/work/shipwright/$APPNAME) which can be deployed repeatably on your servers. You can wrap it up in a Debian or Red Hat package if you’d like, tar it, rsync it, BitTorrent it – up to you.

Maintaining the Vessel

When you build a new version of your Dancer app, all you have to do is update the vessel, then build.

$ shipwright relocate $APPNAME file:~/….new.tar.gz
$ shipwright update $APPNAME
$ cd $APPNAME-vessel
$ git pull
$ rm -rf ~/work/shipwright/$APPNAME && ./bin/shipwright-builder –install-base ~/work/shipwright/$APPNAME –force

Using the Vessel

Shipwright has some nice features to set up all the environment variables needed so you can use your app. All you have to do is source the appropriate script, like so:

# set up your environment so the '$APPNAME' libraries and binaries are in your path
$ . /opt/yourstuff/$APPNAME/tools/shipwright-source-bash /opt/yourstuff/$APPNAME/

What’s cool is you can do the same thing from SYSV-style init scripts. Let’s say you’re launching this as a fastcgi application. Your startup script can look like this example. The magic line is source $APP_BASE… which uses the shipwright shell config to set the variables used by the rest of the script.

#!/bin/bash

NAME=$APPNAME
APP_BASE="/opt/mt/$NAME"
source $APP_BASE/tools/shipwright-source-bash $APP_BASE
APP_BIN="$APP_BASE/bin"
APP_WWW="$APP_BASE/www"
APP_PSGI="$APP_WWW/app.psgi"
FCGI_LISTEN=127.0.0.1:55900
DAEMON="$APP_BIN/plackup"

# Defaults
#RUN="no"
OPTIONS="-s FCGI –listen $FCGI_LISTEN -E production –app $APP_PSGI"

PIDFILE="$NAME.pid"

[ -f /lib/lsb/init-functions ] && . /lib/lsb/init-functions

start()
{
    log_daemon_msg "Starting plack server" "$NAME"
    start-stop-daemon -b -m –start –quiet –pidfile "$PIDFILE" –exec $DAEMON – $OPTIONS
    if [ $? != 0 ]; then
        log_end_msg 1
        exit 1
    else
        log_end_msg 0
    fi
}

signal()
{

    if [ "$1" = "stop" ]; then
    SIGNAL="TERM"
        log_daemon_msg "Stopping plack server" "$NAME"
    else
    if [ "$1" = "reload" ]; then
        SIGNAL="HUP"
            log_daemon_msg "Reloading plack server" "$NAME"
    else
        echo "ERR: wrong parameter given to signal()"
        exit 1
    fi
    fi
    if [ -f "$PIDFILE" ]; then
        start-stop-daemon –stop –signal $SIGNAL –quiet –pidfile "$PIDFILE"
     if [ $? = 0 ]; then
            log_end_msg 0
        else
        SIGNAL="KILL"
        start-stop-daemon –stop –signal $SIGNAL –quiet –pidfile "$PIDFILE"
            if [ $? != 0 ]; then
                log_end_msg 1
                [ $2 != 0 ] || exit 0
            else
            rm "$PIDFILE"
                log_end_msg 0
            fi
        fi
    if [ "$SIGNAL" = "KILL" ]; then
        rm -f "$PIDFILE"
        fi
    else
        log_end_msg 0
    fi
}

case "$1" in
    start)
    start
    ;;

    force-start)
    start
    ;;

    stop)
        signal stop 0
    ;;

    force-stop)
    signal stop 0
    ;;

    reload)
    signal reload 0
    ;;

    force-reload|restart)
    signal stop 1
    sleep 2
    start
    ;;

    *)
    echo "Usage: /etc/init.d/$NAME {start|force-start|stop|force-stop|reload|restart|force-reload}"
    exit 1
    ;;
esac

exit 0

Conclusion

Dancer is great. Shipwright is great. CPAN is great, but I want a buffer from all that awesomeness.

Relaunched blog, sorry for any dust.

2010-08-29T00:00:00-07:00

I’ve migrated my site from WordPress to Jekyll. You can see where the donuts are made in the source. This is a much more programmer-y way to blog, including very nice baked in server-side syntax highlighting.

I apologize if your feed reader gets my entire posting history back, I’ve got a few nice articles in draft that I hope to get up soon so it should be worth staying subscribed.

Notes from my Jekyll Migration

2010-08-29T00:00:00-07:00

I had a few snags moving to Jekyll that I thought worth sharing.

Lenny Support Packages

If you’re using Debian Lenny, you won’t complain about having these packages installed:

$  sudo apt-get install ruby-dev  libyaml-ruby libzlib-ruby python-pygments libgsl-ruby1.8 sun-java6-jre

Local install of Gems

I didn’t want to mess with my system gems, and wanted to have the gems installed in my local directory like I’m used to with Perl’s local::lib. This turned out to be harder than you may think. Here’s the environment incantation that did the trick:

export RUBY_PREFIX="$HOME/.ruby"
export PATH="$RUBY_PREFIX/bin/:$RUBY_PREFIX/lib/ruby/gems/1.8/bin/:$PATH"
export GEM_HOME="$RUBY_PREFIX/lib/ruby/gems/1.8"
export GEM_PATH="$GEM_HOME"
export RUBYLIB="$RUBY_PREFIX/lib/ruby:$RUBY_PREFIX/lib:$RUBY_PREFIX/lib/site_ruby/1.8"

And then I grabbed RubyGems and just

$ ruby setup.rb all –prefix=$RUBY_PREFIX

Importing Wordpress

I also wanted to use a remote database that I didn’t have direct network access to. Step 1, use ssh to create a nice tunnel (to the arbitrary local port 11122):

$ ssh -L 11122:host.of.database.com:3306 user@domain.com

I needed to tweak the importer tool that ships with Jekyll to support using a host and port:

def self.process(dbname, user, pass, host = '127.0.0.1', port = 11122)
    db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :port=> port)

There was a whole lot of hideous perl going on to slap around my posts once they were imported since the post bodies were in a strange combination of raw HTML, textile, and other legacy markup fragments. Things seem fairly solid at this point.

Optimizing CSS

I’m using some CSS magic from the inspirational OOCSS project, and that style leaves you with lots of individual files. My Rakefile is mostly snaked from people like avdgaag, but I added in a useful CSS optimization section.

file "style/merged.css" => ["style/syntax.css", "style/template.css", "style/libraries.css", "style/grids.css", "style/content.css", "style/mod.css"] do |t|
    sh "cat #{t.prerequisites.join(' ')} > #{t.name}.devel"
    sh "java -jar ~/.java/yuicompressor-2.4.2.jar –type css -o style/merged.css #{t.name}.devel"
end

I’ve had to elaborate files by name so they end up in the correct order (one of the least fun things about CSS, aside from ‘everything else’, is that order matters). So you do have to update the list by hand when adding files, but most of the time I’m sure you’ll tend to tweak them instead. I’m using Yahoo’s YUI Compressor which is nice, since if my Javascript gets more complex I can use it for that as well.

For now it shrinks the payload from 14k to 8k, not worth doing super backflips over – the majority of the gain is in the simple concatenation, turning 6+ HTTP GET’s into a single one.

Capturing short-lived programs on Linux

2010-06-14T00:00:00-07:00

One of the things I love most about DTrace is your ability to do things like this. Because you can ask the kernel to let you start tracking when something cool happens (like forking a new program), you can instrument it.

// Measure parent fork time
syscall::*fork*:entry { /* save start of fork */
  self->fork = vtimestamp;
}

Sadly, those tools aren’t available on a standard issue Linux machine. (You can do it with SystemTap, but it’s not on every server I end up looking at.) Today I was trying to track down some processes that were making very odd DNS lookups. I isolated the user ID making these calls via iptables logging:

iptables -I OUTPUT 1 -m string –string "BADZONE" -d 127.0.0.1 -p udp –destination-port 53 –algo bm -j LOG –log-uid –log-prefix "BADZONE: "

But this user’s PHP scripts were getting launched and dying again quickly. How to find out what was going on inside them? This hacky little trick actually worked really well. You can download it from my git utilities directory.

Basically, we check the process list as fast as we can for any owned by that user; If you find one, stick an strace on it. The script will ‘hang around’ until all the strace processes have finished. You end up with a directory full of trace files which you can then post-grep through to get what you need.

The short version is I was able to catch one of that user’s programs doing the odd behaviour within about 30 seconds of running this tool, which was great!

#!/usr/bin/perl

use warnings;
use strict;
use Getopt::Long;

my $opt = {};
GetOptions($opt, "uid=i", "number=i", "help", "man");

if($opt->{help} || $opt->{man} || !defined($opt->{uid})) {
    print "Usage: auto_stracer –number <processes to capture> –uid <numeric uid>\n";
    exit;
}
$opt->{number} ||= 5;

my $traced = 0;
my %seen = ();
my @pids = ();
my %ourpid = ();
$ourpid{$$}++;

$SIG{INT} = sub { kill 9, @pids; print "Interrupt: Killed all running strace processes and quitting.\n"; exit;};

while ($traced < $opt->{number}) {
    for my $line (split(/\n/, `ps -Ao 'uid,pid'`)) {
        $line =~ s/^\s+//g; # eat leading spaces
        my($uid, $pid) = split(/\s+/, $line);
        next if($ourpid{$pid}); # don't run on something we are tracking
        if($pid && $uid eq $opt->{uid} && !$seen{$pid}) {
            $seen{$pid}++;
            my $st_pid = do_strace($pid);
            $ourpid{$st_pid}++;
            push(@pids, $st_pid);
            $traced++;
            last if ($traced >= $opt->{number});
        }

    }
}

print "Traced $opt->{number}, waiting for all to finish\n";
my $kid = 0;
do {
      $kid = waitpid(-1, 0);
} while($kid>0);
print "All processes completed, exiting.\n";
exit;

sub do_strace {
    my($pid) = @_;
    my $ourpid = fork();
    return $ourpid if($ourpid != 0);
    my $cmd = "strace -p $pid -f -s 65535 -o trace.$pid.$$ -v";
    print "\tcmd: $cmd\n";
    system($cmd);
    exit;
}

So, very brute force compared to the elegance of systemtap or DTrace, but when you need it, it’s still handy.

Instantly share local directories with 2 command lines

2010-06-11T00:00:00-07:00

Just came across this useful trick to let anyone browse/download directories you want to share in seconds.

All you need is a Mac or Linux machine you are logged into, and ssh access to a server on the internet. It uses this trick I just saw the other day.

$ python -m SimpleHTTPServer &
$ ssh -R 8080:localhost:8000 my.remote.host

Now someone can go to http://my.remote.host:8080/ and be browsing your local directory. Rad. And you see the access that’s happening come up in your screen – SimpleHTTPServer spits it out to you.

When they’ve got what they need, just

my.remote.host$ exit
$ fg # Brings python back, CTRL-C to exit

Sharing over, nice and securely back to normal.

If this isn’t working, the most common reason is that on the remote server, you do need the ‘GatewayPorts’ set to ‘yes’ in your server’s /etc/ssh/sshd_config for this to work. (And you’ll need to restart sshd after you make that change.)

Optimal web-sized identifiers

2010-05-14T00:00:00-07:00

As part of a project I’m working on for fun (shhhh) I’ve been trying to solve an interesting problem – what is the most compact way I can uniquely refer to something in a URI?

This one’s going to get complicated, so here’s the tl;dr version. To get the shortest possible, web-safe identifiers:

encode identifiers with base62 or a customized version of base64 that uses valid characters.
Use a big enough hash space to get an acceptable level of collisions.

Ok. Here’s the long-winded version.

It’s a little like the “URL Shortener” use case, where you want to take a big fat link and describe it in a tiny link. In this case, I’m converting individual sentences into a tiny, linkable representation.

So, I need a very short identifier that obeys certain (conflicting) properties:

As much as possible, the identifier should be the same regardless of the order that I create the sentences in
It should be as short as possible, because I may need to refer to a lot of these in a single URI

In the URL shortener mode, they tend to solve this by just keeping a global “ID” counter and incrementing it every time someone creates a new link. I need to use something a lot more like a hash. (MD5 or SHA1 are typical choices here.)

So what are the constraints?

How long can a URI be?
What characters are we allowed to use?
How much unique data is going to be represented?
How tolerant can we be of collisions? (When 2 “objects” might map to the same shortened identifier?)

URI constraints

The first 2 are simple and global. There’s no real standard on URI length, but smart people have done the legwork for us so we’ll steal their conclusions, and say “keep URI’s shorter than 2,000 characters.”

As far as the characters that can be used, we can consult RFC 3986 in the “Unreserved Characters” section and find we’re allowed 0-9, a-z, A-Z, and “- _ . ~” (dash, underscore, dot, tilde.) I’m going to avoid using those because I’d like to keep them for separating my identifiers. So, if we just go with the “normal characters”, there’s 62 of them.

Collision Constraints

If you assume a hashing function generates numbers pretty randomly spread throughout the number space available to it (which I will for this) there’s actually some pretty good math we can use to figure out how likely a collision will be. If you’re interested in where this comes from, check out the Birthday Paradox, which is nicely written up at Wikipedia. Using the formulas there you can calculate the probability of having 2 keys that collide, as a function of how many things you’re trying to insert, and the size of the overall hash space.

Putting it all together

First, figure out how many items you want to be able to store. (for me, 20,000 is a high estimate). Second, figure out how comfortable you are with having some keys collide. 50/50 odds? (0.5) 1/1000 chance? (0.001) I’m actually pretty ok having a very small number of collisions – I’m going to use 0.99, a 99% chance that SOMETHING will collide. (That’s still going to be a small number, less than a 100% chance that anything at all will collide, let alone more than one.)

I sketched up a little script for this article (which you can grab from my git repo. It’s pretty self-documenting, so I’ll let it speak for itself.

The goal is, given our constraints, and the fact that we need to use only the characters 0 .. 9, a .. z, A .. Z, (a) how many of those characters do we need, and (b) given that it probably won’t be an exact fit, what will our collision probability end up being?

my ($items, $probability) = (0,0);
GetOptions("items=i" => \$items, "probability=f" => \$probability);

printf("Trying to store %d items with a %0.5f chance of collision\n", $items, $probability);

# solve the birthday paradox in terms of 'how big the range should be'
# http://en.wikipedia.org/wiki/Birthday_paradox#Calculating_the_probability => wolfram alpha, 'solve for d'
my $max = -1*(($items - 1)*$items)/(2 * log(1-$probability));

print "To do that, we'd need to be working in the range 1:$max\n";

# compare that back to the original formula from Wikipedia and make sure that worked
my $calculated_prob = get_probability($max, $items);

printf("probability %0.10f should match your requested probability of %0.10f, or something went wrong.\n", $calculated_prob, $probability);

# so it does, cool
# how many bits would it take to represent $max?

my $frac_bits = log($max)/log(2);
my $real_bits = int($frac_bits)+1;

printf("This can be represented by %0.2f bits (%d)\n", $frac_bits, $real_bits);

# ok, so how about encoding those into URI's? We can use 0-9, a-z and A-Z, which gives 62 "digits"
# aka base62. $max in base62 can be calculated like we did "how many base2 digits were needed"

my $base62_digits = int(log($max)/log(62))+1;


print "Using 0..9, a..z and A..Z we can represent each object with $base62_digits digits\n";

# ok, if we're using 5 digits, what's our actual "space" and what's the probability of a collision?

my $newmax = 62**$base62_digits - 1;

# which is actually how many bits?
my $newbits = int(log($newmax)/log(2));

printf("Given that we'll have to use %d digits, (%d bits), the real probability of a collision is %0.10f\n", $base62_digits, $newbits, get_probability($newmax, $items));

sub get_probability {
    my($max, $items) = @_;
    return 1 - (($max - 1)/$max)**(($items*($items-1))/2);
}

The most mysterious parts of this are probably the formulas. The one in the get_probability subroutine is transcribed right from the Wikipedia page, but the other one is the same formula, solved for a different value. In general, if you need to do this, WolframAlpha is a math nerd’s dream come true. I just asked it to “solve (the equation) for d” and got the new formula I needed.

The solution actually comes from “show your steps” – I can find an intermediate form that’s easier to represent in a non-math-centric programming language. (I’m sure you can do imaginary numbers in perl, but it was kind of outside the scope of my plans for this evening.)

Here’s the formula I ended up using:

Here’s a few sample runs of the script:

First, using my personal constraints for this project:

$ ./hashspace_model -i 20000 -p 0.99
Trying to store 20000 items with a 0.99000 chance of collision
To do that, we'd need to be working in the range 1:43427276.7179157
probability 0.9900000006 should match your requested probability of 0.9900000000, or something went wrong.
This can be represented by 25.37 bits (26)
Using 0..9, a..z and A..Z we can represent each object with 5 digits
Given that we'll have to use 5 digits, (29 bits), the real probability of a collision is 0.1961141788

Cool! So I said I’m ok with a 99% chance of a collision, and the algorithm figured out that in order to do that, I’d need to be using 5 digits of base62. And if I’m using 5 digits of base62, I get 3 more bits than I strictly “need”, which means I end up with only about a 1/5 chance of EVER getting a collision.

Let’s say I wanted to be more strict, and go “one in a million”.

$ ./hashspace_model -i 20000 -p 0.000001
Trying to store 20000 items with a 0.000001 chance of collision
To do that, we'd need to be working in the range 1:199989899999232
probability 0.0000009992 should match your requested probability of 0.0000010000, or something went wrong.
This can be represented by 47.51 bits (48)
Using 0..9, a..z and A..Z we can represent each object with 8 digits
Given that we'll have to use 8 digits, (47 bits), the real probability of a collision is 0.0000009103

In this case base62 comes pretty close to exactly the dimensions that we want, so we more or less get 1/1,000,000 on the nose with 8 digits.

If you grab the script to play with, you may need to tweak the printf’s and make sure they have enough resolution for the data you’re trying to examine.

Ok, so that tells us how bit our bitspace needs to be, but not how to get a hash, or how to do base62.

Hashing Function

I chose md5 because…. it seems to work fine. I’m sure there’s a better option, but this is working for now. However, md5 has a lot more bits available than I need. How to just steal a few of them?

First, you need to know how many bits you want. Thankfully, I know I want 29 bits (thanks, helper script!). I can extract 29 bits of information from it by making a “mask” of 29 1’s, which can be done easily like so:

my $mask = 2**29 - 1;

So, now I just need a raw integer slice of an md5, and do a “logical and” of that:

use Digest::MD5 qw(md5);
# unpack makes this back into an integer for us
# L == interpret the data as a 32 bit unsigned long. 
# See 'perldoc -f pack' for a ton of other options
my $value = unpack("L", md5("the string we want"));
$value = $value & $mask;

Groovy. Now we know what number we want to represent, we need to actually represent that in this weird “base62” format.

The algorithm for converting to base(anything) is actually pretty easy. We’re used to thinking in base10, so I’ll show an example of running the algorithm to from and to base10, just so the flow is clear.

125	Starting Value
125 % 10 = 5	Find the remainder when dividing by the “destination base.” Keep “5” as the “new number”
125 / 10 = 12	Divide by the “destination base”. This chops the number off the end that we just snagged as a remainder
12 % 10 = 2	Do it again with the result of the division. Stick the result to the front of number we’re keeping track of (now 25)
12 / 10 = 1	And do the division again
1 % 10 = 1	Last remainder, glue to the front again, and we have the answer “125”. (Back where we started.)
1 / 10 = 0	As soon as we get 0 from the division, our work here is done.

Simple enough to understand in base10, but the exact same technique works when going from base10 to any other base (2 would work to convert to binary, we’re using 62, and if you could find enough characters, you could go to something crazy like base300 using the same idea.)

Here’s the source for it:

# even though they are letters we are using them here in the role of "digits"
my @digits = (0 .. 9, 'a' .. 'z', 'A' .. 'Z');
my %digits = map { $digits[$_] => $_ } 0 .. $#digits;

sub to_base {
    my($base, $value) = @_;
    die "base $base out of range 1-62" unless ($base > 0 && $base <= 62);
    return $digits[0] if $value == 0;
    my $rep = "";

    while($value > 0) {
        # prepend the "digit" we get when dividing by the base        
        $rep = $digits[$value % $base ] . $rep;
        # then "shift right" the working value
        $value = int( $value / $base );
    }
    return $rep;
}

sub from_base {
    my($base, $rep) = @_;
    die "base $base out of range 1-62" unless ($base > 0 && $base <= 62);
    my $value = 0;
    # this pattern grabs a character at a time, left to right
    for ( $rep =~ /./g ) {
        $value *= $base;
        $value += $digits{$_};
    }
    return $value;
}

So there you have it. Provably optimal URI-compatible identifiers with 3 easy steps:

Figure out what the constraints for your problem space are
Grab enough bits from md5
Convert to (and from) base62

Quickie: Keeping my OSX downloads folder clean

2010-05-04T00:00:00-07:00

My Downloads folder consistently gets large and cluttered. I considered an app like Hazel but it seemed like overkill.

I added the following alias to my .bashrc file:

alias clean_downloads="find ~/Downloads -mtime +30 -maxdepth 1 -print -exec rmtrash '{}' ';'"

which depends on the free tool ’rmtrash’ being installed – a command line way to elegantly move files to the OSX-standard trash can.

The nice thing is when I think ‘that looks messy’, I just run clean_downloads, and get a nice list of all the things that it just “trashed” – so, worst case, I can go and fish it out right then and there. Actually a bit nicer than things magically getting trashed while I’m not looking.

Getting around tmpfs ‘noexec’ problems

2010-03-16T00:00:00-07:00

In general, running your /tmp (or /var/tmp) without the execute bit set is a good idea. And sometimes, you don’t have a choice – for example, when running in a hosting environment running Virtuozzo.

You’re liable to see a mount that looks like this:

$ mount | grep noexec
/dev/simfs on /tmp type simfs (rw,noexec)
/dev/simfs on /var/tmp type simfs (rw,noexec)
devpts on /dev/pts type devpts (rw,nosuid,noexec)

However, sometimes that’s a problem, for example if you run the fantastic checkinstall tool to package software. You might see an error message like this:

/usr/bin/installwatch: /var/tmp/tmp.SuogJyYftZ/installscript.sh: /bin/sh: bad interpreter: Permission denied

The noexec permission has gotten us. How to work around it? Here’s a quick, easy to roll back method:

# make a new temporary directory for this use
mkdir ~/tmptmp
# use mount –bind to overlay this on our 'real' /var/tmp for now
$ sudo mount –bind ~/tmptmp /var/tmp
# do your work
$ sudo checkinstall make install
# restore the natural order to the universe
$ sudo umount /var/tmp

You could also (potentially) remount /tmp without that option temporarily, but that isn’t always possible. (See ‘virtuozzo’ above.)

Adding iSCSI exports to OpenSolaris for remote Time Machine

2010-03-15T00:00:00-07:00

There are 3 ways to Time Machine over the network (that I know of)

Use a Time Capsule
Run Snow Leopard Server, which allows you to set remote drives as Time Machine volumes
Mount the iSCSI share via an “initiator”, and use it like it was a normal hard drive.

I’m not sure I’d want to use the iSCSI option with a laptop, so I use a hybrid of the second and third options.

I’m running Snow Leopard Server on a Mac Mini with limited disk space. I’ve got a large server running OpenSolaris about which I blogged previously.

Here are the brief steps you need to take to get it working.

On the Snow Leopard Server, install the ISCSI initiator from GlobalSan.

On your OpenSolaris server, make sure you’ve installed the support packages needed.

# pkg install SUNWiscsi SUNWiscsitgt

Now, from one of your available disk pools, create the virtual hard drives for your servers to mount, (making sure you make them about 1.5x larger than the disks you’re trying to back up, so you can get historical information from them.

# zfs create -o shareiscsi=on -s -V 160GB mypool/laptop_tm
# zfs create -o shareiscsi=on -s -V 140GB mypool/server_tm

To find out what the “target ID’s” are, run this:

# iscsitadm list target

shows the target names to paste into the initiator.

You can follow the instruction at this blog on sun.com which has screenshots of the rest of the process.

At this point the drives are just like regular disks, from the perspective of your Mac (desktop or server) – and can be formatted and used for Time Machine. Very convenient and the data that you’re backing up will be in the safe, safe hands of ZFS in the event of a disk failure.

Using custom functions in Puppet templates

2010-01-01T00:00:00-08:00

This eventually required a support ticket to figure out, so I’m documenting it here in case it’s useful to others.

The problem: You have a puppet function you find useful. At this point custom functions are really easy to create and distribute via modules so there’s really no reason not to be using them. However, you want to use the result of a function in your templates.

A real-world example of this is sharing authentication information. (For example, crypted password hashes. Plain text, as always, would be evil no matter where it was living.) You really have 3 choices:

Coding it right into your puppet code and checking that into your version control system
Not distributing that information with puppet
Using puppet to distribute the information, but store it outside of your puppet modules somehow.

We chose the third way, and store this kind of information in YAML files outside the puppet modules tree. Anyway. So we want to access this info inside a template.

Assume you have a function called ‘get_extdata’. You normally call from your ‘.pp’ manifests with something like

    $data = get_extdata('mymodule', 'path:to:data')

You can make a call like that from your template by doing this wonderful fragment of magic-wand-waving to get this function accessible:

In mytemplate.erb:

    <% Puppet::Parser::Functions.autoloader.loadall %>
    Now you can use the 'scope' variable to get at your function.
    This is the equivalent of the call get_extdata('mymodule',  'path:to:data'):
    <%= scope.function_get_extdata('mymodule',  'path:to:data') %>

This works like a charm, and means you don’t have to create a bunch of variables that you don’t really need before you load the template.

So if you were creating an .htpasswd style file with puppet, you could do

In your init.pp:

    $users = ['steve', 'paul', 'stu']

htpasswd.erb:

    <% Puppet::Parser::Functions.autoloader.loadall %>
    <% users.each do |user| -%> 
    <%= user %>:<%= scope.function_get_extdata('authmodule',  user) %>
    <% end -%>

Updated “Inbox Zero with Mail.app” technique

2009-12-02T00:00:00-08:00

Update: I have updated to a new method since writing this – I now use a free native Archiving plugin. More information in my most recent article.

Back in March, I wrote about my technique for implementing Inbox Zero with Mail.app.

Since then, the world has changed. I’ve upgraded to Snow Leopard, and Quicksilver has essentially stopped working out for me. (I’ve been trying to keep it up to date, and sometimes it kind of works, but my confidence levels are really low.) On the other hand, I’m loving me some Google Quick Search Box (a new project from the original creator of Quicksilver). The only feature that I still wanted Quicksilver for was triggers. And until today, I was still running it for trigger support to implement my Inbox Zero trick.

And then triggers broke when I upgraded it. And I’m out.

So here’s how to get Inbox Zero magic in Mail.app using only tools native to Snow Leopard.

First, launch Automator and create a new Service workflow.

Select “Utilities” and then drag in “Run Applescript” to the window on the right.

Fill in your applescript, replacing only the center “comment” block – leave the other autogenerated lines there.

The core code again, (tweak to fit your accounts and archive mailbox names)

    tell application "Mail"
        set theSelectedMessages to selection
        set myAccount to "zimbra"
        set myMailbox to "Archive"
        repeat with theMessage in theSelectedMessages
            move theMessage to mailbox myMailbox of account myAccount
        end repeat
    end tell

It should look something like this:

And that’s it! Save it, and give it a name you’ll remember. (I chose “Archive Selected Mail.”)

Now it’s actually in Mail’s Menu (under ‘Services’) and we can use the built in OSX hotkey support to launch it.

Go to the System Preferences Menu, click “Keyboard Shortcuts”, and add one for the menu item you created. (It needs to be the same exact name you saved your service as.)

Badda bing. Feature complete, and I can (sadly) retire Quicksilver forever. This technique should be extensible to all kinds of other cases when you want to add new features to applications, for access either through the menus or with hot keys.

serialized.net

Using R and `ggplot2` to Contemplate Relocation

Problem 1: Get some data.

Problem 2: Import the data into R

Basic Temperature distribution

It’s not just about the temperature

It’s just a perfect day

Screencasting Presentations With Screenflow

Going Fast With GMail’s Keyboard Shortcuts

Using Ogres to Improve Your Designs

The problem

An approach

Objectives

How It Works, and what was that you were saying about Ogres?

Background

Gameplay

Heuristic Ideation?

Comparative Risk

Lessons

Share and Enjoy

An Iteration of the Lean Meetings Concept

Framework

Goals

Takeaways

Personal Agendas

Democratization

Conclusion

Getting more Signal from your Noise

The Point

The Data Stack

Hadoop

Log Processing and Management

Data Analysis

Machine Learning

People to Watch

And so

Testing Perl code that runs commands

tl;dr

Code Walkthrough

Using IPC::Run in your code

Storing the command line output results

Overriding IPC::Run::run in a test

Write a callback that emulates IPC::Run::run

“Mock” that in place

Actually doing the testing

Trust me, the normal code will still actually call the system

Wrapping it all up

Up-Up-Updated ‘Inbox Zero with Mail.app’ Technique

Why using packages makes sense in a configuration management world

Packages vs wget && tar

The cost of building packages

The value of building packages

Redundant, Version-aware Repositories

Built-in “tripwire”-like functionality

The package manager knows how files got on your system

Troubleshooting, and knowing where to make changes when you find the problem

Uninstalling

Dependencies are built in

Discovery of available updates are built in

Cryptographic signatures are built in

Binary-identical code on every server

Environment Support

The End

Deploying self-contained Perl Dancer applications

MANIFEST

shipwright build file

Walkthrough

Build your Dancer app

Prepare the ‘vessel’

Fill the vessel with software

Build the vessel

Maintaining the Vessel

Using the Vessel

Conclusion

Relaunched blog, sorry for any dust.

Notes from my Jekyll Migration

Lenny Support Packages

Local install of Gems

Importing Wordpress

Optimizing CSS

`tl;dr`

Overriding `IPC::Run::run` in a test

Write a callback that emulates `IPC::Run::run`

Packages vs `wget && tar`

shipwright `build` file