Notes: Josh McArthur

How to handle Cloudflare Challenge pages when making Turbo requests

Mon, 13 Apr 2026 00:00:00 +1200

Turbo Drive intercepts link clicks and form submissions, fetching the next page with fetch() and swapping the <body> in place rather than doing a full browser reload. That’s great for smoothness, but it means the browser never performs a real navigation — and some things depend on a real navigation to work.

Cloudflare challenge pages are one of those things. When Cloudflare decides a request needs to be challenged, it returns a special HTML page that runs JavaScript to verify the visitor. If Turbo swaps that HTML into the existing <body> the challenge scripts fail to initialise and the user ends up stuck on a broken page.

Cloudflare marks challenge responses with a Cf-Mitigated: challenge header. Turbo fires a turbo:before-fetch-response event after a fetch completes but before it processes the body, which is the right place to catch this. Calling event.preventDefault() stops the swap, then window.location.assign() upgrades the visit to a real navigation:

document.addEventListener('turbo:before-fetch-response', (event: Event) => {
  if (
    !('detail' in event) ||
    !event.detail ||
    typeof event.detail !== 'object'
  ) {
    return;
  }

  const { fetchResponse } = event.detail as {
    fetchResponse: { header: (name: string) => string | null; location: URL };
  };

  if (fetchResponse.header('cf-mitigated') !== 'challenge') {
    return;
  }

  event.preventDefault();
  window.location.assign(fetchResponse.location.href);
});

The detail guard is there because TypeScript types the event as a plain Event — Turbo’s custom properties aren’t in the standard lib, so a cast is needed. The early return keeps the handler cheap for the overwhelming majority of requests that aren’t challenges.

Once the challenge is solved, Cloudflare redirects back to the original URL, which Turbo handles as a normal navigation.

How to test new Github Actions workflows

Mon, 29 Sep 2025 00:00:00 +1300

I recently build out a data processing pipeline that uses Github Actions, and so needed to add several new workflows. I couldn’t for the life of me figure out how to test them. They were going to be initiated either as part of a pipeline (using workflow_call), or manually, using workflow_dispatch. In most cases, each workflow had inputs which needed to be provided.

Github Actions shows workflows that can be called, but only once they’re on main. I didn’t really want to sit here committing directly to main until they worked the way I needed, so I came up with this flow which works well:

Create the workflow files you need. Make sure they have the workflow_dispatch trigger at least so they can be invoked
Put a minimal job into the workflow file - it doesn’t matter what the job does, it just needs to be syntactically valid
Via a pull request, add these new workflows to main- noting that they are basically no-op workflows at the moment
Now that the workflows are on main, they are available to be dispatched from the Github Actions UI, and you can now choose which branch to run the workflow from.
Modify your workflow on a branch, committing as necessary and invoking the workflow from that branch.
Once you have the workflow working the way you expect, you can open a pull request to add the ‘implementation’ of the workflow back to ‘main’.

This pattern still requires a pull request onto main, but only with no-op, ‘empty’ workflows. Once they are on main, they can be actually implemented and tested in a feature branch before merging the implementation back to main.

Talk on Plain Language Law at the AI Engineers meetup in Wellington, NZ

Thu, 04 Sep 2025 00:00:00 +1200

Last night I had a great opportuntity to talk about Plain Language Law, a site that I built using my own time and personal development time provided by Ackama.

Plain Language Law is a site that turns New Zealand legislation into plain language using LLMs. This was the first meetup I’d presented at, so I really appreciated the invite, and the chance to do a bit of personal development with public speaking. I found I got a lot of value in conversations and questions, and I have some great ideas about ways I can build improvements into the data processing and QA pipeline.

If this talk interests you, my slides can be found below:

Handy access to OS environment variables on Debian

Thu, 04 Sep 2025 00:00:00 +1200

This morning, while checking (yet again) how to install Postgres inside a Debian container, I stumbled across a file that I didn’t know existed - /etc/os-release - this file contains a bunch of handy variables which can be sourced to chuck them into your environment as environment variables:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$VERSION_CODENAME is particularly handy for building apt repo release strings, but there’s a lot useful stuff in there!

How to set up Rails directories for MCP servers

Thu, 28 Aug 2025 00:00:00 +1200

I’ve recently started on a new Rails application which will define at least one MCP servers, and potentially 2-3. To support this, I wanted to set up a new namespace in my app/ folder to contain all the MCP ‘stuff’ - I also do the same thing for GraphQL mutations, resolvers, queries etc.

Zeitwerk has made autoloading and eager loading much easier in Rails, but this pattern is still something I always have to figure out - I want an app/mcp folder, and I want all the constants in that tree to sit within the MCP module.

So a folder structure like:

app/
├── mcp
│   ├── prompts
│   │   └── base_prompt.rb
│   ├── README.md
│   ├── resources
│   │   └── base_resource.rb
│   ├── servers
│   │   └── base_server.rb
│   └── tools
│       └── base_tool.rb

Defines constants named like MCP::Servers::BaseServer.

The magic for this is to tell Rails’ autoloader about a new top level directory and the namespace to use:

# config/application.rb
# Autoload and reload these paths in dev, eager load in production
Rails.autoloaders.main.push_dir(
  Rails.root.join("app/mcp"),
  namespace: MCP
)

With this in place, both autoloading, dev reloading and eager loading all work as expected:

require "test_helper"

class MCPAutoloadTest < ActiveSupport::TestCase
  test "MCP classes are autoloaded correctly" do
    assert MCP::Servers::BaseServe
    assert MCP::Tools::BaseTool
    assert MCP::Prompts::BasePrompt
    assert MCP::Resources::BaseResource
  end
end

Bikeshed #470: All about Queues

Mon, 25 Aug 2025 00:00:00 +1200

I listened to https://bikeshed.thoughtbot.com/470 in the weekend. There was a fair amount of talking about judoscale, but there was also some interesting stuff in there about measuring queue performance. A couple of key takeaways for me:

It seemed like a really good suggestion to me was to name queues by their SLA, rather than their purpose - for example, instead of naming queues “emails”, “exports”, “push_notifications” or whatever, go with when messages should be handled within - Adam suggested starting with “5 seconds”, “5 minutes” and “5 hours”.
Having queues named for SLA gives you a really good metric for monitoring - messages shouldn’t sit in any of those queues for longer than the time they’re named for.
If you have the infra for it (e.g. Heroku was mentioned), you don’t have to handle all queues with the same infra - an example they gave was that if you have a big expensive job that takes loads of resources (memory) - put that job on it’s own queue and then you can have a dedicated dyno or instance or whatever that only works on jobs in that queue and is otherwise idle. The idea being not to waste your expensive compute on trivial jobs

Connecting to a locally Running app with hotwire native

Wed, 06 Aug 2025 00:00:00 +1200

title: Connecting to a locally-running app with Hotwire Native category: TIL –

I’ve just started a new Hotwire Native application. The Getting Started docs for Android use a deployed domain name, so while the demo app worked just fine, I had a bit of trouble figuring out how to connect to my locally-running web app from an Android emulator. In my case, it was a Rails app, but any other web framework would also have this problem.

There were two useful things I found:

Android emulators make your computer available at 10.0.2.2. This means that if you run your web app on port 3000, you can connect in a browser at http://10.0.2.2:3000. Neat!
If you are connecting to insecure origins, you need to flag this in your AndroidManifest.xml - specifically, you need to add android:usesCleartextTraffic="true" to the <application> tag.

With this configuration in place, I was able to set up Hotwire Native to connect in debug builds to my locally-running web applicaiton, and I was off!

A simple redirect to next pattern for Rails

Thu, 24 Jul 2025 00:00:00 +1200

In Rails controllers, it’s pretty common to redirect somewhere when an action is completed. Sometimes this location is unknown, and therefore straightforward. Other times I just want to redirect ‘back’ - wherever that might have been - redirect_back(fallback_location: wherenver_path) is perfect for that.

Occasionally though, I’ll want to determine the next path from the caller - often this will be for some sort of generic or shared action, that might need to go one of several places when it’s done. For that, I like to use this simple redirect_to_next pattern:

module RedirectToNext
  extend ActiveSupport::Concern

  def redirect_to_next(fallback_location:, **args)
    return redirect_to fallback_location, **args unless next_path

    redirect_to params[:next], **args
  end


  def next_path
    return nil unless params[:next]&.start_with?("/")

    params[:next]
  end
end

# app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
  include RedirectToNext

  # in a controller
  def create
    # ...
    redirect_to_next(fallback_location: whatever_path, notice: "Created a thing")
  end
end

This pattern essentially follows the logic of redirect_back, but sources the redirect location from a param called next. The param should be a path (though Rails prevents other-host redirects now by default anyway). It splats other args so that anything that can be passed to redirect_to is forwarded.

The usage of this is nice and simple. For controllers that are POSTed to directly, I just provide the next param with the path to send to. For new actions, I include the next param as a hidden field:

<%= form_with model: @widget do |f| %>
   <%= hidden_field_tag :next, whatever_path %>
   <%= f.label :name %>
   <%= f.text_field :name %>
   <%= f.submit >
<% end %>

How to run a command via AWS SSM with live output

Wed, 23 Jul 2025 00:00:00 +1200

This little technique is a great example of how simple *nix utilities can be (mis)used to accomplish some interesting things:

 (echo "./deploy.sh" && cat && exit && exit) |  \
   aws ssm start-session --target $INSTANCE_ID

This:

Echos the command into the shell session once started (echo "./deploy.sh" - obviously whatever command you want to run goes here)
Runs cat to capture any output from that command until the command exits
n exit calls, where n is the number of shell sessions you’ve got. I have two, because SSM by default runs sh, then I have a Linux profile set up in SSM that runs bash. This will drop you back where you started, in your shell.

This accomplishes the same result as using send-command with the built-in AWS-RunShellScript document (which is what start-session uses under the hood), but shows output of the command in real-time, just like ssh ... -c would, without the need to wait for the SSM command to finish executing before output is visible.

A Stimulus controller to auto-submit authentication codes

Tue, 22 Jul 2025 00:00:00 +1200

I think this is neat - very succinct. This controller expects an input that calls formatOnSubmit on input, which removes invalid chracters, strips the input (e.g. of spaces), and then automatically submits it. Works great with pasting, manually typing, and provides fast feedback to users.

import { Controller } from "@hotwired/stimulus";

export default class extends Controller {
  static targets = ["input"];

  formatAndSubmit(event) {
    const input = event.target;
    let value = input.value.replace(/\D/g, ""); // Remove non-digits

    // Limit to 6 digits
    if (value.length > 6) {
      value = value.slice(0, 6);
    }

    input.value = value;

    // Auto-submit when 6 digits are entered
    if (value.length === 6) {
      input.form.requestSubmit();
    }
  }
}

I use this controller for things like email and 2FA codes, bound with a simple bit of markup:

<form data-controller="verification-code">
  <input
    type="text"
    pattern="[0-9]{6}"
    maxlength="6"
    autofocus
    required
    data-verification-code-target="input"
    data-action="input->verification-code#formatAndSubmit"
  />
  <button type="submit">Verify Code</button>
</form>

Open-sourcing a reference implementation of native iOS Google sign-in flow with Hotwire

Tue, 22 Jul 2025 00:00:00 +1200

I’ve just open sourced a reference implementation of native iOS Google sign-in using Hotwire Native. I developed this flow when building Virtualtrails, where I wanted to support both web and native authentication flows for Google Sign-in. Using Hotwire for this worked great - when native support is available, it’s automatically used. When not, it falls back to a web-based OAuth flow.

https://github.com/joshmcarthur/hotwire-native-google-sign-in

About Trove Project

Fri, 04 Apr 2025 00:00:00 +1300

I’ve been working this week on Trove (Github, Docs). Trove is meant to be a self-hosted system that can store arbitrary (but schema validated) JSON data, files, and relationships between events. By events I mean things like:

Bookmarks
Photos I like
Colours/patterns I’ve seen out and about that I like
Storypark updates (from daycare/school)
etc.

Basically all the digital miscellanery that I want to hang on to, but currently exists as a google keep note, or read email that I don’t ever archive. I don’t plan on trying to replicate services I already use like Google Photos, Google Drive, iCloud, and the like, but rather might cherry-pick particular bits of content from those systems to hang onto separately.

I’ve just completed the core, which is basically an event bus with a plugin system. Plugins (will be able to) register new schemas (event types), web/API endpoints, or the ability to process specific event types. Processing like:

Screenshotting a bookmarked website/extracting a MHTML archive
Label/OCR/face detection on images
Thumbnailing images
Generating colour palettes from a saved color
Extracting and saving attachments from an email
Extracting a note from an email
Visiting a link from an email and scraping the resulting page

This particular project is massively ambitious, but it’s also a good exercise in trying to stay really focussed on a small core and try and architect it so that I can achieve all of the above, and any other ideas I have, just using the plugin system. I’m taking a lot of inspiration from HomeAssistant in being very modular and trying to be very adaptable. I’m using Deno, which is an alternative JS/TS runtime to Node, and I’m feeling really good about that. I’m doing my best to not have any dependencies outside the Deno standard library, and automating builds and releases. I want self-hosted data I put in to be safe and private to me, but it’s also a fun and challenging constraint to think about designing Trove to be able to be useful 10 years from now without a tonne of maintenance intervention.

I don’t have a stable release yet, but I will soon. Right now, the core works as a library, but I’ll be setting it up as a precompiled CLI program and setting up Github releases to distribute Mac OS/Windows/Linux binaries and Docker images. The CLI program will accept a typescript configuration file which determines what plugins to load, what configuration to pass to those plugins, and how to store events, files, and event links (which can be a single storage, or an individual storage for each).

I also have plans to create an API (JSON and JSON-over websocket), and a web and native frontend. I’ll be utilising plugins for each of these, and don’t expect to need to extend the core. This means that if myself, or someone else does not want or need a web frontend, or a native frontend, or an API, that plugin can just be omitted from the configuration. I’m also expecting API and web core plugins to allow other plugins to build on them, allowing for authentication, middleware, and routing within plugins to create new capabilities without having to extend the core.

OpenNext on Cloudflare

Mon, 03 Mar 2025 00:00:00 +1300

I’m trying out OpenNext with Cloudflare. One kind of annoying thing already I’ve run into is that libraries that use cross-fetch end up running into errors, because OpenNext patches the deployment package enough that cross-fetch tries to use Node libraries, when actually it can just use native fetch.

In the case I ran into, I could work around this by performing a native fetch, then processing the response. Alternatively, I can use patch-package to resolve cross-fetch’s fetch to the built-in version.

Hutt Forks/Eastern Hutt Hut

Sun, 09 Feb 2025 00:00:00 +1300

I tried to get to Eastern Hutt Hut on Friday. Didn’t quite make it, just because I’d committed to a half day rather than full day trip, but it seemed doable. I turned around here with 2km to go, it took me 2 hours to get there after biking over from Kaitoke on the 4WD road - 1 hour biking, just over 1 hour walk/running. It felt like another 1-1.5 hours to the hut in the riverbed or game trails. Definitely no track, so it’s a case of bush bashing or rock hopping in the river.

Easter Hutt Hut is for emergency use only, since it’s in or near the water catchment area - visitors are asked to clear the area in a day, rather than staying.

View Larger Topographic Map

Restart rails quickly with bin/rails restart

Thu, 16 Jan 2025 00:00:00 +1300

Just a quick tip! If you’re like me, you start with just one or two tabs (or tmux panes), and things get more and more disorganised as the day goes on. If you need to restart your Rails server quickly, and don’t feel like hunting out your Rail server terminal, no worries! Just running bin/rails restart will signal the server process, wherever it’s running, to restart itself. This can also be used on a deployment server, to restart the server process without actually knowing what the process is (though you should really be using proper process/service manager on a server!).

Technically, this also means that if you really don’t want your server visible in your terminal, you could also run the server daemonised (e.g. in the background), and just restart it when you need.

Check that an enumerable satisfies matchers with args in RSpec

Wed, 15 Jan 2025 00:00:00 +1300

Note: I discovered this API based on a suggestion from an AI editor assistant, just to disclaim that.

When creating an RSpec test against an enumerable, rubocop-rspec will enforce that the all matcher be used instead:

it "only returns odd numbers" do
  odd = [1 3 5]
  expect(odd).all(be_odd)
end

This is fine for a matcher that doesn’t require any args be passed to it, but what do we do in the case where we do require args?

Take this example:

it "shows the family member names on the page" do
  family_names = %w[Alice Bob Kenny]
  expect(family_names).all(have_content)
end

Now that fails, because, quite rightly, has_content? expects an argument. So how do we pass an argument to the matcher? We can use satisfy, a block matcher:

it "shows the family member names on the page" do
  family_names = %w[Alice Bob Kenny]
  expect(family_names).all(satisfy { |name| page.has_content?(name) })
end

Using matchers like this means that your test output will be most useful on failure. Instead of a .each “failing fast” when it encounters an item that doesn’t pass the test, RSpec will report on all more comprehensively:

  expected ["Alice", "Bob", "Kenny", "Quentin"] to all satisfy expression `has_content?(page_name)`

          object at index 3 failed to match:
             expected "Quentin" to satisfy expression `has_content?(page_name)`

There are lots of other checks you can do within satisfy: https://rspec.info/features/3-12/rspec-expectations/built-in-matchers/satisfy/

Detecting the origin of an ActiveRecord record's destruction

Tue, 14 Jan 2025 00:00:00 +1300

In a before|after_destroy callback, you can use the destroyed_by_association method to access the association (therefore the parent record or parent record type) that is causing this record to be destroyed. This is also a handy way to tell the difference betwee a ‘direct’ destruction (widget.destroy), and a ‘dependent’ destructionhas_many :widgets, dependent: :destroy`).

I had a use case for this recently with a before_destroy callback intended to prevent a record from being ‘orphaned’ - that is, a record that should always have at least one of something, where destroying the record would cause it to have none.

I implemeneted a ‘validating’ callback:

before_destroy :prevent_orphaned_record

private

def prevent_orphaned_record
  return true unless parent.children.size == 1
  errors.add(:base, :destroy_would_orphan_record)

  throw :abort
end

The problem I have with this, is that when the parent record has an assocaition with dependent destroy, we don’t actually want this validation to be triggered, since, while we are orphaning the record, we are also about to destroy the parent. An example of an association like this would be:

has_many :children, dependent: :destroy
# or
has_one :child, dependent: :destroy

What we want to do, is guard the callback, so that if the parent record is being destroyed, we can allow the record to be orphaned, since it’s about to be destroyed. I tried a few different variations, but eventually found destroyed_by_association. This method returns the association, which allows access to the parent record, as well as the parent record class.

With this method, the guard clause can easily be implemented:

def prevent_orphaned_record
  return true if destroyed_by_association # Could also check the specific parent here, etc.
  return true unless parent.children.size == 1

  # etc
end

How to use assigns() in a Rails request spec without a dependency

Fri, 10 Jan 2025 00:00:00 +1300

Several years ago, the Rails core team advised that controller tests/specs were being discouraged, in favour of request specs. The basis for this was that controller tests had more visibility into the internals of the controller, and were not purely testing the HTTP interaction as request specs were.

This is valid, but one method I end up missing a lot from controller specs is assigns - as in (expect(assigns(:widget)).to be_published). While the assertions available to controller tests are easily installable as a gem, getting the assigns method back is super simple.

It’s pretty common to put support modules into place - for example, in spec|test/support/request_spec_helpers.rb:

module RequestSpecHelpers
  def assigns(key)
    controller.view_assigns.fetch(key.to_s)
  end
end

In Test::Unit, this can just be included into your ActionDispatch::IntegrationTest:

class WidgetRequestTest < ActionDispatch::IntegrationTest
  include RequestSpecHelpers
end

And in RSpec, it can be included with config:

RSpec.configure do |config|
  config.include RequestSpecHelpers, type: :request
end

Generating ActionMailer Previews inside a transaction

Thu, 09 Jan 2025 00:00:00 +1300

ActionMailer Previews are a great way to provide some living documentation in the form of rendered emails. They’re also great for testing.

Often, in order to generate a preview, you might need to create a database record - perhaps using fixtures, or a tool like factory_bot. When you do this, you might be surprised to see these records then show up and remain in your database. This is because previews are not generated inside a rollback transaction, so once they are in your database, they stay there.

To avoid this, I have a small patch to Rails which I apply using an initializer to all the projects I work on that need previews:

module RollbackingAfterPreview
  def preview
    ActiveRecord::Base.transaction do
      super
      raise ActiveRecord::Rollback
    end
  end
end

Rails.application.config.to_prepare do
  class Rails::MailersController
    prepend RollbackingAfterPreview
  end
end

It simply wraps the preview action of the Rails::MailersController in a transaction which is rolled back after the action executes. This means that any database updates made in the process of generating the preview are rolled back, and not committed to the database.

I’ve been interested in committing this to Rails, as I think it’s a reasonable default behaviour to have. Unfortunately, when I tried to dig into it, I found that I couldn’t quite get my head around how to test the transactional part of the patch, so it’s remained something that I just patch myself when I need it. Maybe it’s useful to someone else.

Asserting method calls with Minitest

Sat, 05 Oct 2024 00:00:00 +1300

Minitest has support for stubbing via Minitest::Mock. This is a good fit for providing fake results to a caller, but what about asserting that a method is actually called?

To do this, you can leverage the fact that if the stubbed value is callable, it will be called. This means that you can provide a mock, then expect that the mock will receive call with the expected args (and return the return value that you need):

perform_job = Minitest::Mock.new
# expect(method name, return value, [args])
perform_job.expect(:call, nil, [my_job_arg])
SlowJob.stub(:perform_now, perform_job) do
  method_that_triggers_slow_job
end

perform_job.verify

Converting geofiles to WKT using ogr2ogr

Thu, 05 Sep 2024 00:00:00 +1200

I love gdal, particularly ogr2ogr. Like ffmpeg, and convert, it’s just one of those convert anything to anything commands that is so handy to know about.

ogr2ogr is part of the GDAL library, and is particularly suited to converting files containing geospatial data from one source to another. I often use this tool to convert shapefiles into GeoJSON for example, load data into Spatialite or PostGIS, or use query and filter operations to reduce a result set into a smaller file.

I built, and operate, an application (web + iOS) called Virtualtrails. This application integrates with a bunch of fitness platforms to track the distance travelled by walking, running, or using a wheelchair, and then superimposes that distance onto a trail from New Zealand and around the world. Along the way, points of interest are virtually reached and unlocked, and so it’s a really fun way to both meet fitness objectives and find out more about a trail.

I create all the trail content myself, but often trail geospatial paths need to get stitched together from a number of sources. Sometimes they are published in geospatial formats already (typically a GPX or KML) file, other times I need to trace the route myself to get the geospatial data. Unsurprisingly, with route data coming from so many sources, ogr2ogr regularly features in my route prep process.

Today I found a nice shortcut to prepare route data, which is to leverage an option to ogr2ogr’s CSV format export that exports data in well-known text (WKT). WKT is particuarly useful to me because it’s a text format which rgeo natively supports - I use rgeo to transform geospatial data to and from Ruby objects, including database persistence in native PostGIS types. Before I discovered this option, I used to have to manually apply some find and replace operations in my editor to convert a format that was close to WKT like GeoJSON or KML to WKT.

The option is passed using -lco, which are format input flags. Here’s an example:

ogr2ogr -f CSV output.csv input[.geojson, .kml, .shp, etc] -lco GEOMETRY=AS_WKT

The output CSV will export one row per feature, with a header column of ‘WKT’. If your input format features have attributes, these will also be included as features.

WKT
"LINESTRING Z (174.235252414479 -41.0947374817005 0,174.234414785847 -41.0946484189218 0,174.2339359575 .... )"

Because this process is part of ogr2ogr, it’s also no longer necessary for me to convert from a complex format like a shapefile to a text format like GeoJSON - because ogr2ogr is converting to CSV anyway, it will accept any input file it supports (and it supports loads) - this can include operations like filtering, stripping or converting attributes, or reprojection, if I need.

A handy find!

Tapping targets with Stimulus

Sat, 10 Aug 2024 00:00:00 +1200

Normally I would use Bootstrap’s stretched link behaviour to implement functionality where clicking anywhere on a containing element triggers a link element. This is useful for cards, menus, and other cases where a ‘primary’ link can benefit from a larger tap target.

Bootstrap’s stretched link works by adding an ::after pseudo element to the link element with the stretched-link class, then using absolute positioning to position the pseudoelement across the container. The ‘container’ requires some definition since it depends on the normal requirements for an absolute-positioned container - Bootstrap’s docs on identifying the containing block are useful here.

In my case, I couldn’t quite get this to work. I was implementing a card with a dropdown menu of actions, of which one action was the primary one. I wanted to support functionality to allow users to tap anywhere on the card to trigger the primary action, unless the dropdown menu was triggered.

To resolve this, I created what I thought would be a small Stimulus controller to listen for click events on a containing element, and trigger a click on the element:

import { Controller } from '@hotwired/stimulus';

export default class TappableTargetController extends Controller {
  public static readonly targets = ['receiver'];
  private readonly receiverTarget!: HTMLElement;

  public connect(): void {
    this.element.classList.add('cursor-pointer');
    this.element.addEventListener('click', (evt) => {
      this.receiverTarget.click();
    });
  }
}

This seemed to work great! The element had the correct cursor style, and when clicked, triggered the primary action. Then I tried to open the dropdown menu to check other actions - oops - this click was also captured, so the dropdown menu was no longer expandable!

I used Claude 3.5 to create a more complicated version to handle this case:

import { Controller } from '@hotwired/stimulus';

export default class TappableTargetController extends Controller {
  public static readonly targets = ['receiver'];
  private readonly receiverTarget!: HTMLElement;

  public connect(): void {
    this.element.classList.add('cursor-pointer');
    this.element.addEventListener('click', this.handleClick.bind(this));
  }

  private handleClick(event: MouseEvent): void {
    const target = event.target as HTMLElement;

    // Check if the clicked element is the controller element itself
    if (target === this.element) {
      this.receiverTarget.click();
    } else {
      // Check if the clicked element is a button or anchor tag
      const isInteractableElement = target.matches('button, a, [role="button"]');

      // If it's not an interactable element, trigger the receiver's click
      if (!isInteractableElement) {
        this.receiverTarget.click();
      }
      // If it is an interactable element, let the default behavior occur
    }
  }
}

This nearly works, but in my case, the dropdown menu displays an overflow icon using an SVG. The variant above seemed that it would work, but the event is triggered on the SVG, not the button, so is not counted as a click on an interactable element. I knew what I’d need to do - I’d need to consider the target’s ancestors - but again, one of those fiddly things that I reached for Claude for the next iteration:

import { Controller } from '@hotwired/stimulus';

export default class TappableTargetController extends Controller {
  public static readonly targets = ['receiver'];
  private readonly receiverTarget!: HTMLElement;

  public connect(): void {
    this.element.classList.add('cursor-pointer');
    this.element.addEventListener(
      'click',
      this._handleClick.bind(this) as EventListenerOrEventListenerObject
    );
  }

  private _handleClick(event: MouseEvent): void {
    const target = event.target as HTMLElement;

    // Check if the clicked element or any of its ancestors up to the controller element is interactable
    if (this._isOrHasInteractableAncestor(target)) {
      // Let the default behavior occur for interactable elements
      return;
    }

    // If we've reached here, it means the click was not on an interactable element
    this.receiverTarget.click();
  }

  private _isOrHasInteractableAncestor(element: HTMLElement): boolean {
    let currentElement: HTMLElement | null = element;

    while (currentElement && currentElement !== this.element) {
      if (this._isInteractableElement(currentElement)) {
        return true;
      }
      currentElement = currentElement.parentElement;
    }

    return false;
  }

  private _isInteractableElement(element: HTMLElement): boolean {
    return element.matches('button, a, [role="button"], [tabindex="0"]');
  }
}

This version works, but doesn’t handle an important user experience consideration - it should be possible to use a standard keyboard shortcut to open the link in a new tab. This is usually done by holding the Command (Control) key down while clicking the link. The final controller iteration handles this case:

import { Controller } from '@hotwired/stimulus';

export default class TappableTargetController extends Controller {
  public static targets = ['receiver'];
  private readonly receiverTarget!: HTMLElement;

  public connect(): void {
    this.element.classList.add('cursor-pointer');
    this.element.addEventListener(
      'click',
      this._handleClick.bind(this) as EventListenerOrEventListenerObject
    );
  }

  private _handleClick(event: MouseEvent): void {
    const target = event.target as HTMLElement;

    // Check if the clicked element or any of its ancestors up to the controller element is interactable
    if (this._isOrHasInteractableAncestor(target)) {
      // Let the default behavior occur for interactable elements
      return;
    }

    // Prevent the default action
    event.preventDefault();

    // If Ctrl key (or Cmd key on Mac) is pressed, open in new tab
    if (event.ctrlKey || event.metaKey) {
      this.openInNewTab();
    } else {
      // Otherwise, perform the regular click action
      this.receiverTarget.click();
    }
  }

  private _isOrHasInteractableAncestor(element: HTMLElement): boolean {
    let currentElement: HTMLElement | null = element;

    while (currentElement && currentElement !== this.element) {
      if (this._isInteractableElement(currentElement)) {
        return true;
      }
      currentElement = currentElement.parentElement;
    }

    return false;
  }

  private _isInteractableElement(element: HTMLElement): boolean {
    return element.matches('button, a, [role="button"], [tabindex="0"]');
  }

  private _openInNewTab(): void {
    // Assuming receiverTarget is an anchor element
    if (this.receiverTarget instanceof HTMLAnchorElement) {
      window.open(this.receiverTarget.href, '_blank');
    } else {
      // If it's not an anchor element, log a warning
      console.warn(
        'Receiver target is not an anchor element. Cannot open in new tab.'
      );
    }
  }
}

This version worked great - I could hold down Command to open links in a new tab, and it worked exactly like a native link.

That’s exactly what got me thinking though - while I had emulated one user experience enhancement, there were tonnes of others that users might be relying on, and it’s not really practical for me to emulate every one of these enhancements.

Ultimately, I decided not to use this code for this exact reason - I have too many concerns about not supporting behaviour that users will expect to ‘just work’, and might even harm the accessibility of links that I know will be important to users.

Instead, I reverted to using the stretched-link class, but changed it to be on my dropdown menu toggle, instead of the primary action. I then gave the primary action a bit more prominence in the dropdown menu. This introduces an extra click to trigger the action, but also retains the accessibility and usability of how this link can be accessed and interacted with - and that’s important to me.

I still have an option to experiment with transparent (but still visible to screen reader) links, so I have a fallback to support a primary action, but for now, I’m ready to test triggering the dropdown, and will iterate on this if required.

iOS Action Sheet with Strada

Sat, 03 Aug 2024 00:00:00 +1200

Virtualtrails uses Strada to bridge native iOS code to the web application, including notifications, native Apple + Google Signin and Apple Healthkit integration. I added a feature yesterday to replace a Bootstrap dropdown with a native iOS sheet, just because it’s a more intuitive experience for users.

Strada has two parts - the web side (which is in the form of a Stimulus controller with some guards built in so it only runs when Strada’s native side is available), and the iOS side (which is in the form of a ‘component’ which receives ‘messages’ from the web side, and can reply to them).

The Strada Bridge Component is pretty straightforward:

import { BridgeComponent, Message } from '@hotwired/strada';
import { Turbo } from '@hotwired/turbo-rails';

interface PresentDropdownNavigationSheetLink {
  text: string;
  url: string;
  style: string;
}

interface PresentDropdownNavigationSheetData {
  selectedLink?: PresentDropdownNavigationSheetLink;
  links: PresentDropdownNavigationSheetLink[];
}

interface PresentDropdownNavigationSheetMessage extends Message {
  data: PresentDropdownNavigationSheetData;
}

export default class DropdownNavigationSheetController extends BridgeComponent {
  public static component = 'dropdown-navigation-sheet';
  public static values = { links: Array };
  private readonly linksValue: PresentDropdownNavigationSheetData[] | undefined;

  public present(e: Event): boolean {
    e.stopImmediatePropagation();

    this.send(
      'present',
      { links: this.linksValue },
      ({ data }: PresentDropdownNavigationSheetMessage) => {
        if (data.selectedLink) {
          Turbo.visit(data.selectedLink.url);
        }
      }
    );

    return false;
  }
}

When the ‘present’ action is triggered from the element, it sends a ‘present’ message to the iOS component, with the links to present in the action sheet. Each link has text, a URL (most of the time a path in my case), and a style (which maps to the iOS action sheet item styles). iOS will respond with a message when the user selects an action, with the selectedLink property filled in. I then use Turbo to ‘visit’ this URL - this is essentially window.location, but with extra powers around other native integration (e.g. open a URL modally).

The iOS code is super simple, it just responds to ‘present’ messages by opening an action sheet, and replying with the selected item:

import SwiftUI
import Strada

final class DropdownNavigationSheetComponent: BridgeComponent {
    override class var name: String { "dropdown-navigation-sheet" }

    override func onReceive(message: Message) {
           guard let event = Event(rawValue: message.event) else {
               return
           }

           switch event {
           case .present:
               handlePresentEvent(message: message)
           }
       }

    private var viewController: UIViewController? {
           delegate.destination as? UIViewController
   }


    private func handlePresentEvent(message: Message) {
        let alertController = UIAlertController(title: nil, message: nil, preferredStyle: .actionSheet)
        guard var data: PresentMessageData = message.data() else { return }
        let links = data.links

        for (index, link) in links.enumerated() {
            let action = UIAlertAction(title: link.text, style: link.actionStyle()) { [weak self] _ in
                let newData = PresentMessageData(links: links, selectedLink: links[index])
                self!.reply(with: message.replacing(data: newData))
            }
            alertController.addAction(action)
        }

        let cancelAction = UIAlertAction(title: "Cancel", style: .cancel, handler: nil)
        alertController.addAction(cancelAction)

        viewController!.present(alertController, animated: true, completion: nil)
    }


}

private extension DropdownNavigationSheetComponent {
    enum Event: String {
        case present
    }

    struct DropdownNavigationSheetLink: Encodable, Decodable {
        let url: String?
        let text: String
        var style: String = "default"

        func actionStyle() -> UIAlertAction.Style {
            switch self.style {
            case "destructive":
                    .destructive
            case "cancel":
                    .cancel
            default:
                    .default
            }
        }
    }


    struct PresentMessageData: Encodable, Decodable {
        let links: [DropdownNavigationSheetLink]
        let selectedLink: DropdownNavigationSheetLink?
    }
}

I use some ! assertions in here to make sure things fall over when properties I expect to be defined are not. Honestly I could probably change these to be ? and it can just do nothing.

The markup that invokes this is also pretty simple. I build the links collection in ERB which is pretty gross, but also contained to this menu partial, and I can quite easily extract it to an object. This is one of the times I kind of want to use viewcomponents, but I also then would be adding a dependency for this one thing. The reason I pass the links as a value is just to not encode anything in the Stimulus controller, since I’ll use different links based on user access and permissions, and in different places.

<% links = [
    policy(progress_update).show? ? { text: "Share", url: share_path, style: "default" } : nil,
    policy(progress_update).edit? ? { text: "Edit", url:  edit_path, style: "default" } : nil
  ].compact %>

  <% button_attrs.merge!(data: {
    controller: "bridge--dropdown-navigation-sheet",
    action: "bridge--dropdown-navigation-sheet#present",
    bridge__dropdown_navigation_sheet_links_value: links
  }) %>

  # button_attrs then gets used to make a button, e.g.
  button_tag("More options", **button_attrs)

Implementing an around callback with Minitest using Ruby

Sun, 23 Jun 2024 00:00:00 +1200

I use both RSpec and Minitest, but prefer Minitest. It’s not quite as feature-rich as RSpec, but I find that I prefer the tests I write in Minitest - they are more Ruby-ish, and aren’t as reliant on RSpec’s DSL to work.

I don’t use around callbacks in RSpec very often, but it is something I recently found myself wanting to do in Minitest. Since I couldn’t find a built-in way to achieve this, I fell back to just using Ruby - in this case, proxying the test method.

Just to be clear, in RSpec, an around callback is passed the example as a block arg. Typically, you’ll take an action before running the example, then run the example, then take an action after running. This is often used for setting a variable for the duration of a test, instrumentation, that kind of thing:

describe Thing do
  around do |example|
    puts "Running before example"
    example.run
    puts "Running after example"
  end
end

The thing with Minitest/Test::Unit is - it’s just Ruby. To achieve something similar with Ruby, we can just wrap Minitest’s test method to take the action we want before and after calling the actual test method:

class ThingTest < Minitest::Test
   def self.wrapped_test(description)
    test(description) do
      puts "Running before test"
      yield
      puts "Running after test"
    end
  end

  wrapped_test "does foo" do
    assert_equals 1, 1
  end
end

Because this is just Ruby, wrapped_test can be moved to a module, or a base test class to be reused by other tests. It’s also entirely possible to create multiple test wrappers, and because they’re declarative, they’ll have nice descriptive names of what they are wrapping. You can even wrap wrappers - it’s Just Ruby!

Auto-resizing images for .ico files using ImageMagick

Wed, 19 Jun 2024 00:00:00 +1200

There’s a bit of a mystery about how relevant favicon.ico files are in 2024. Like it not though, they are the lowest common denominator, so they still have some use.

Today I learned about a handy option that can be passed to ImageMagick that will automatically resize a source into a range of common dimensions to be packed into an ico file.

Given a source file, say favicon.svg, which is, say, a vector image, the following command generates a favicon.ico:

convert -density 256x256 -background transparent favicon.svg -define icon:auto-resize -colors 256 favicon.ico

Source: https://gist.github.com/azam/3b6995a29b9f079282f3

This will also work with a non-vector image source, like a PNG, in which case I believe the density argument is not needed.

The favicon by default resizes to 256, 192, 128, 96, 64, 48, 40, 32, 24 and 16px variants:

❯ identify favicon.ico
favicon.ico[0] PNG 256x256 256x256+0+0 8-bit sRGB 38282B 0.000u 0:00.002
favicon.ico[1] ICO 192x192 192x192+0+0 8-bit sRGB 0.000u 0:00.002
favicon.ico[2] ICO 128x128 128x128+0+0 8-bit sRGB 0.000u 0:00.001
favicon.ico[3] ICO 96x96 96x96+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[4] ICO 64x64 64x64+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[5] ICO 48x48 48x48+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[6] ICO 40x40 40x40+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[7] ICO 32x32 32x32+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[8] ICO 24x24 24x24+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[9] ICO 16x16 16x16+0+0 8-bit sRGB 337400B 0.000u 0:00.000

The -define option does accept arguments though, so you can resize to a different range or subset:

convert -density 256x256 -background transparent favicon.svg -define icon:auto-resize=32,16 -colors 256 favicon.ico

identify favicon.ico
▶ identify favicon.ico
favicon.ico[0] ICO 32x32 32x32+0+0 8-bit sRGB 0.000u 0:00.000
favicon.ico[1] ICO 16x16 16x16+0+0 8-bit sRGB 5430B 0.000u 0:00.000

This can be useful if you don’t need all the predefined sizes, since the resulting file will be much smaller.

Generating responsive utility classes with Bootstrap

Fri, 14 Jun 2024 00:00:00 +1200

I’m a big fan of Bootstrap, I use it a lot. I’ve used it for many years, and with the recent support for theming with CSS variables, and for their utility API, I’ve found I can have the best of both worlds - a rich and capable library of components, along with an extensive class-based approach for making tweaks to elements and components.

One of the neat things about the utility API is that it’s generated from a SASS map. This means that the map can be manipulated before generation to affect which utilities are generated, and how they are generated.

There are lots of things that can be done with this manipulation, but in this case I’m going to talk about adding a single option to a utility: responsive: true. This will generate a variant of each utility class for each breakpoint (which are themselves configurable).

As an example, Bootstrap defines a range of text color utility classes - text-danger, text-dark, text-light, etc - all the colours defined in $utilities-text-colors, plus some options for muted, emphasis, etc. The utility definition is in the utilities map:

$utilities: (
  // ...
  "color": (
      property: color,
      class: text,
      local-vars: (
        "text-opacity": 1
      ),
      values: map-merge(
        $utilities-text-colors,
        (
          "muted": var(--#{$prefix}secondary-color), // deprecated
          "black-50": rgba($black, .5), // deprecated
          "white-50": rgba($white, .5), // deprecated
          "body-secondary": var(--#{$prefix}secondary-color),
          "body-tertiary": var(--#{$prefix}tertiary-color),
          "body-emphasis": var(--#{$prefix}emphasis-color),
          "reset": inherit,
        )
      )
    ),
    // ...
);

Source

You’ll notice that this map is missing the responsive option. This can be added by manipulating the $utilities map. This needs to be done after importing bootstrap/scss/utilities (so that the map is actually defined), but before importing bootstrap/scss/utilities/api (so that the map changes are reflected in the generated classes):

@import "bootstrap/scss/utilities";

$utilities: map-merge(
  $utilities, (
    "color": map-merge(
      map-get($utilities, "color"),
      ( responsive: true )
    )
  )
);

@import "bootstrap/scss/utilities/api";

With this option added, we can now apply normal responsive modifiers to our text- color classes. As an example: .text-danger.text-lg-success will color the element these classes is applied to to red (#dc3545 by default) up to large screens (992px by default), then to green (#198754 by default).

This is a somewhat convoluted example, but the takeway is more significant - in the same way that the responsive option can be added, any other parts of the map can be added, or removed, or changed - including class names, values, and other options like print: true (generates modifier classes for in print mode, for example d-print-none will hide an element with print media).

How to rearrange menu bar icons in Mac OS

Mon, 10 Jun 2024 00:00:00 +1200

This is a tip I re-discover every couple of years, then forget until I find my menu bar getting a bit cluttered up again.

Holding down the Command key allows menu icons to be dragged backwards and forwards along the menu bar to order them. This is available for all items other than the control center and date/time, including system items like network, bluetooth and sound.

Logging the DOM state of an element with testing-library

Thu, 06 Jun 2024 00:00:00 +1200

I’m sure this is all over testing-library documentation, but I haven’t had to do this before, so I think it’s worth a quick post about.

If I’m writing a testing-library assertion, every now and then the test won’t pass, because the element I expected to be present, was not. Usually, when this happens, a snippet of the DOM will be output, but will be truncated - sometimes the element that has not matched is in the truncated version, and sometimes it is not.

So that I can see the full DOM state, I’ve come across screen.debug(). This is, I suspect, what testing-library calls when a DOM matcher test fails, but it has some useful options which can help debug a test.

First of all, the element to log can be passed. This is useful if you have a selector you can match, because it allows just that element to be logged, without any of the noise before, after, or around it:

screen.debug(screen.getByTestId("test-container"));

Obviously you can pass any DOM query function you want there, including if you need to await an element with findBy*, or even direct DOM lookups with document.querySelector (though my eslint rules in this case don’t permit me to do this).

If the DOM contents are still too long to see the problem, you can also pass options to debug with the element - maxLength is handy, since it lets you override the length at which the contents will be truncated.

screen.debug(screen.getByTestId("test-container"), { maxLength: 10000 });

Less useful for easy debugging, you can also pass extra options to dictate how the output should be rendered - for example, turning off syntax highlighting.

Stretching interactive elements

Wed, 05 Jun 2024 00:00:00 +1200

Stretched links have been a feature of Bootstrap for some time, but it’s quite a useful technique to understand, because the functionality it offers - expanding the tap target of an element, works for any element that can be interacted with with a tap or mouse gesture, not just links. Some common examples include:

Buttons
Inputs
Labels

The way that stretch links work is that they define a ::after pseudo element, and then use absolute positioning to have that pseudoelement fill it’s relatively positioned container (and the relative positioning is a requirement for this technique), using the following CSS:

.stretched-link::after {
	position: absolute;
	bottom: 0;
	content: "";
	left: 0;
	right: 0;
	top: 0;
	z-index: 1;
}

The neat thing about using this technique is that whatever element the stretch is applied to retains it’s normal position and style. Because the pseudo element is part of that element though, any interaction bubbles - so clicks, double-clicks, drags, hovers, etc, all propagate. This makes this technique really useful for:

Cards (where the card has an unsurprising primary action)
List items
Droppable file inputs (to make the drop target bigger)

Passion projects

Mon, 27 May 2024 00:00:00 +1200

Throughout my career, I’ve always had at least one project on the go outside work - call this a personal, learning, or passion project - it all amounts to the same.

I’ve got a lot of benefit out of these projects - sometimes, I approach them deliberately to try out a new technique or tool. Sometimes, I just enjoy the process of starting a new app from scratch. Sometimes, I’ve got an idea to satisfy a problem I, or someone I know, is facing.

This post is mostly dwelling on how often I see these sorts of projects being undervalued - the point of these projects, in my mind, is that they don’t have to succeed. They don’t even need to make sense, but they also need to have a bit more thought and structure than a single-file code snippet or codepen. What they provide is a kind of sandbox - both from the point of view of providing a space to play around, but also from a stakes point of view - there’s no client getting invoiced here, there’s no startup running out of cash unless this project turns it around, or a board of directors to be presented to to secure ongoing funding. That’s a really fun and relaxing environment to work in.

This doesn’t mean that development or effort isn’t fast paced, but it doesn’t have to be. I’ve personally noticed a real difference in how I approach projects, when I have space to think. This means that if I face a problem, or something doesn’t work the way I expect - there’s no pressure. I can put that problem down for a little while, even a few days or weeks. I can come back to it from time to time, and try something out, see if it works or not. If something really doesn’t work out the way I want it to, I can walk away completely, or I can spend a couple of hours tidying it up enough to go on my Github, MIT license it, and leave it there, in case it’s useful to someone else. There’s no obigation to support it, keep it alive, or up to date.

This lack of urgency also helps pacing in other ways. I’ve found the working patterns for my passion projects tend to be really different when you already have professional and personal commitments. For me, this means that my ratio of planning to doing is very weighted towards planning - I’m spending less time actually at my desk, but much more time not at my desk thinking about the next move. I’ve found that this pattern of work has really helped me in my professional softwave development job as well, since it’s taught me the value of picking the right approach the first time, and how the right decision can continue to pay dividends throughout the life of a project. Sometimes spending 90% of the time thinking through the best approach means that the 10% of actually executing on that approach is more productive as if I’d spend 100% of the time fumbling for a working solution.

I’ve got a number of ‘completed’ projects that I personally use most days. I find the end state for these projects which are largely feature-complete (or at least feature complete for my needs) really interesting, because it turns out that the process of taking care of these projects is also really different from professional work. From my point of view, professional long term support tends to be focussed on security - patching packages, and upgrading frameworks, libraries and languages as they reach end of life. Personal projects can have this pressure, but it depends on priorities. I see these types of projects being more like taking care of a garden - sure, there will be a degree of chores, like mowing the lawn or weeding, but there will also be a degree of small, incremental tidy ups - pruning, and tidying up loose ends or rough patches you’ve noticed and now can’t unsee. Every now and then, there will even be a larger piece of work - maybe redoing part of the garden by creating a new feature or undertaking a refactor of some core code.

Regardless, I”ve got a lot of value out of creating projects with no particular end goal in mind. It’s a completely different working situation than professional work, and the lack of structure provides a great deal of space to explore technical, architectural, planning and project management skill development.

To check out passion projects I’ve done before, check out my repository history on Github.

Procs can be passed to Rails' collection form helpers

Sat, 25 May 2024 00:00:00 +1200

Just a quick little way to reformat or otherwise transform a value to be used as the label in a select, radio button, or checkbox collection group in a Rails form:

Given I have an enum defined like so:

class User
  enum :distance_unit, { mi: "mi", km: "km" }
end

And translations defined like so:

en:
  distance_units:
    km: Kilometer (Metric)
    mi: Mile (Imperial)

Then I can pass a short proc as the ‘label’ argument to a collection_* form helper, which allows the value to be transformed:

<%= f.collection_radio_buttons :distance_unit,
                               User.distance_units.keys, :to_s, ->(val) { t("distance_units.#{val}") } %>

Note: This also works to transform the ID argument as well, however normally the model or object already has a method which can be passed by symbol to identify the value.

Avoiding script injection when rendering ERB

Thu, 23 May 2024 00:00:00 +1200

Bad (but not obviously so):

data = { greeting: "Hello, world!" }
ERB.new("
  <h1>Show greeting:</h1>
  <p><%= greeting %></p>
").result_with_hash(data)

# => <h1>Show greeting:</h1> <p>Hello, world!</p>

The question is - what happens if greeting is something different…like <script>alert('greeting!')</script>? Perhaps, if you’re used to Rails, not what you might expect:

data = { greeting: "<script>alert('greeting!')</script>" }
ERB.new("
  <h1>Show greeting:</h1>
  <p><%= greeting %></p>
").result_with_hash(data)

# => <h1>Show greeting:</h1> <p><script>alert('greeting!')</script></p>

ERB is a templating language, but the content it expects is agnostic - it could be HTML, script, CSS, text…anything. Because of this, it doesn’t perform any sanitisation or escaping for you.

One option here is to wrap everything in methods like CGI.escapeHTML, but in a Rails environment, it’s much better (and safer!) to instead rely on Rails’ existing rendering mechanisms!

Good:

data = { greeting: "<script>alert('greeting!')</script>" }
ApplicationController.renderer.render(
  inline: "
    <h1>Show greeting:</h1>
    <p><%= greeting %></p>
  ",
  locals: data
)
# => <h1>Show greeting:</h1> <p>&lt;script&gt;alert(&#39;greeting!&#39;)&lt;/script&gt;</p

Streamlining daily blog posts

Wed, 22 May 2024 00:00:00 +1200

For quite a while, I managed to find something interesting to write up each work day. Unfortunately, life got in the way, and as is easy to do, this habit stopped.

I’m hoping to get back into writing, and I’m trying to do this by making it easier to write up a post without leaving my flow as much. To help with this, I set up this shell script:

#!/bin/bash

blog-post() {
    # Check if title argument is provided
    if [ -z "$1" ]; then
        echo "Please provide a title for the blog post."
        exit 1
    fi

    # Save the title
    title="$1"

    # Change directory to your blog directory
    cd ~/Projects/github.com/joshmcarthur/joshmcarthur.github.com

    # Create a new blog post markdown file
    make post title="$title"

    # Generate the file name
    file_name="_posts/$(date +"%Y-%m-%d")-$title.md"

    # Open the new post in VSCode with --wait argument
    code --wait "$file_name"

    # Commit and push changes after VSCode is closed
    commit_and_push "$file_name"
}

# Function to commit and push changes to GitHub
commit_and_push() {
    file_name="$1"

    # Add all changes
    git add "$file_name"

    # Extract the post title from the file name
    post_title=$(basename "$file_name" | sed 's/^[0-9]*-//' | sed 's/\.md$//')

    # Commit changes with post title and current date
    git commit -m "Updating $post_title - $(date)"

    # Push to GitHub
    git push origin master
}

# Call create_post function with title argument passed from command line
blog-post "$@"

This script will switch to my blog, and create a new post file, then open VSCode with --wait, which means it waits for the editor to be closed before returning to the calling script. Once this happens, it will commit and push the post automatically. Any corrections or updates, I’m happy to do as a subsequent commit.

I’ve added this to my dotfiles bin/ folder, and chmod +x’d it so I can call it from anywhere. It’s ready to go!

Adding elevation to DOC track dataset

Sun, 14 Aug 2022 00:00:00 +1200

I don’t know if this is interesting to anyone, but on Friday and for an extra hour or two in the weekend, I was playing around with calculating an elevation profile for walking tracks in NZ.

I started with a dataset from data.govt.nz from the Department of Conservation (which for AU is responsible for walking tracks and facilities at a national level - basically all the national parks, and big walks - local councils take care of smaller parks, tracks etc). That dataset has just over 3,000 records, including a track name, distance in meters, and a path comprised of X,Y points.

For the elevation, I grabbed a dataset from LINZ (Land Information NZ), who maintain topographical map coverage of NZ. The topomaps have lots of detail on them, but I was interested in the contours. Contours are shapes that indicate elevation on a map. Contour lines that are closer together indicate steeper ground, and farther apart, a gentler climb. For the Topo50 map series in NZ, a contour line indicates an altitude gain or loss of 20m. Every 100m, a contour line is displayed as a bolder line with a label indicating the altitude. In the dataset I downloaded from LINZ, contours are represented as polygons. This dataset was reasonably small - 1.8gb compressed. I investigated a digital elevation model, or DEM. This is basically a giant dataset of points, associated with an altitude. Usually, this data is collected from air or spare using LiDAR, which is basically bouncing a laser beam off the ground and measuring the distance. In NZ, LiDAR coverage is still limited to around the main centres. I am going to investigate this more, since there are a couple of 8-meter resolution DEM datasets that I might be able to use - these are derived from the contour lines anyway though, rather than something as precise as LiDAR.

I set up a new PostGIS database, and imported the two datasets, which I had downloaded as shapefiles, using GDAL, in particular, ogr2ogr. GDAL is to geospatial data what ImageMagick is to image data, and is great and converting between formats. Once I had imported this data, I had two tables, doc_tracks, and nz_contours_topo_150k.

I then did a bit of googling, and immediately found a query that did almost literally exactly what I wanted - took an existing line shape, and existing elevation data, and matched one with the other to create a new line shape with elevation (Z axis) added:

WITH
    sequenced_points AS (
        SELECT description, status, length_over_ground, object_type, equipment,
               mln_seq,
               pts.path[1] AS elem_seq,
               pts.path[2] AS pnt_seq,
               pts.geom AS geom
        FROM (
            SELECT descriptio as description, status, shape_leng as length_over_ground, object_typ as object_type, equipment,
                   ROW_NUMBER() OVER() AS mln_seq,
                   (ST_DumpPoints(wkb_geometry)).*
            FROM doc_tracks
        ) AS pts
    )

SELECT DISTINCT
       description, status, length_over_ground, object_type, equipment,
       ST_MakeLine(st_setsrid(ST_MakePoint(ST_X(sequenced_points.geom), ST_Y(sequenced_points.geom), q.elevation), 2193)) OVER(PARTITION BY sequenced_points.mln_seq, sequenced_points.elem_seq ORDER BY sequenced_points.elem_seq) AS geom
FROM sequenced_points
JOIN LATERAL (
    SELECT elevation
    FROM nz_contours_topo_150k
    ORDER BY wkb_geometry <-> sequenced_points.geom
    LIMIT 1
) AS q
ON true;

This query isn’t too hard to parse, but uses a few techniques that are unusual for my normal webapp CRUD queries. It uses a lateral join, common table expression, and partitioning. The original answer did not have a call to ST_SetSRID, which I added, since otherwise the point is constructed without an SRID, which means any mapping software will not know how to display it. The SRID of both datasets I imported is 2193, which is the NZTM projection used throughout NZ. I could have reprojected either, or both datasets to ‘regular’ latitudes and longitudes, WGS84 or 4326, but this would distort the track paths, which I needed to keep quite precise.

This query worked, but was incredibly slow. I also found it hard to run for just one track at a time, because in the CTE, the first thing it does is break the track path into individual points. So putting a LIMIT clause on that just returned part of a track.

I stated looking at how to speed this up. The first thing I did, was add indices to the geospatial columns:

-- Create an index on the contour line polygon geometry
create index nz_contours_topo_150k_geo on nz_contours_topo_150k using gist (wkb_geometry);

-- Create an index on the doc track geometry
create index doc_tracks_geo on doc_tracks using gist (wkb_geometry);

This didn’t really help. The index nz_contours_topo_150k also took a suspiciously short amount of time. Almost like it didn’t do anything at all…

The next thing I did was move the preprocessing of the track lines into a new table. This meant my elevation-adding query got much simpler, since it didn’t need to partition any more:

create table doc_track_points as (
 SELECT tracks.descriptio as description,
        	   tracks.object_typ as track_grade,
               mln_seq,
               tracks.path[1] AS elem_seq,
               tracks.path[2] AS pnt_seq,
               tracks.geom AS geom
        FROM (
            SELECT descriptio, -- <some_other_column>,
                   object_typ,
                   ROW_NUMBER() OVER() AS mln_seq,
                   (ST_DumpPoints(wkb_geometry)).*
            FROM doc_tracks
        ) AS tracks);
create index doc_track_points_seq_idx on doc_track_points (mln_seq, elem_seq);
create index doc_track_points_geom_idx on doc_track_points using gist (geom);

With the track points in their own table, I could filter the rows to just return points for a particular track:

SELECT
       dtp.description, dtp.track_grade,
       st_setsrid(st_makepoint(st_x(dtp.geom), st_y(dtp.geom), q.elevation), 2193) as geom
from doc_track_points dtp
JOIN LATERAL (
    SELECT elevation
    FROM nz_contours_topo_150k
    ORDER BY wkb_geometry <-> dtp.geom
    LIMIT 1
) AS q
ON true
WHERE dtp.description='Upper Fenian/Fenian Gold/Adams Flat Trk';

This track has 167 points along it’s route - not a tonne. It took 6 minutes to add elevation to it. It wouldn’t have been impossible just to have this run and generate all the data, but I still felt it should be faster than this for that number of points. Because I was operating on one point at a time, and adding the elevation to it, I would still need to post-process that table’s data back into an actual line at some point as well.

The next thing I did was start running EXPLAIN ANALYZE on the query. Prefixing a query with EXPLAIN (ANALYZE, COSTS, VERBOSE, BUFFERS, FORMAT JSON) generates a JSON output that can be pasted into a visualisation tool like https://explain.depesz.com/ to visualise where the query is spending it’s time.

Using this tool, I could see that even though the correct indexes were being used, nearly the entire query time was being spent matching contour polygons to track points. This isn’t terribly surprising, since this is where the actual geospatial calculation is happening - everything else is mostly just a straight operator-based query.

Since indexes were already being used, I was a bit stuck here, but I had a suspicion - while there are well-established mathmatical formula for finding the nearest point on a polygon to a point (and in fact this is how Postgis is matching the polygon to the point internally), it’s still a potentially expensive calculation that needs to be done. I wondered whether the query performance would be improved if I “exploded” the contour lines out into their respective points, indexed these, and then matched on individual points, rather than a whole polygon. I particularly suspected that indexing these points would result in an efficiency gain, because PostGIS geospatial indexes are very much optimised for finding the nearest point to another point.

So, I used ST_DumpPoints as I had for getting the points from a track line, to dump out the points comprising a contour polygon, into a table:

create table contour_line_points as (
	select nctk.geom as wkb_geometry,
		   nctk.elevation
	from (
		select elevation,
			   (ST_DumpPoints(wkb_geometry)).*
		from nz_contours_topo_150k
	) as nctk
);

This table took a few minutes to generate - not bad.

I then added an index to this table. This took a very long time - about 13 hours - but I wasn’t too worried. I had expected it to be slow, and essentially once the index was built, I wouldn’t have to rebuild it unless I change the underlying data. I probably could also tune some Postgres settings like maintenance_work_mem to try and speed this up.

create index contour_line_points_geom on contour_line_points using gist (wkb_geometry);

Now I ran the query for resolving the track points above, but swapped out the table name - contour_line_points instead of nz_contours_topo_150k. I could leave the operator the same, because the operator already knows how to handle different types of geospatial data - I’ve just switched it from matching a polygon to a point, to a point to a point.

Now when I ran the same query for that single track with 167 points, it took around 250ms. A great improvement! It was such a good improvement that I rolled my query back to the original version. I preferred this version because it didn’t require me to pre-process the track lines into points then have to post-process them back into a line - I could just take in the 2D line, and return the 3D line with elevation added. I ran it, and off it went. It still took a couple of hours to run, so I prefixed the query with CREATE TABLE doc_tracks_elevation AS. This meant the result of the query was written to a table so that I could quickly query data from the returned dataset (using CREATE MATERIALIZED VIEW AS here would have been better, since I could refresh the view easily, but I didn’t think of it at the time). The query for generating this table I ended up with was:

create table doc_tracks_elevation as
WITH
    sequenced_points AS (
        SELECT description, status, length_over_ground, object_type, equipment,
               mln_seq,
               pts.path[1] AS elem_seq,
               pts.path[2] AS pnt_seq,
               pts.geom AS geom
        FROM (
            SELECT descriptio as description, status, shape_leng as length_over_ground, object_typ as object_type, equipment,
                   ROW_NUMBER() OVER() AS mln_seq,
                   (ST_DumpPoints(wkb_geometry)).*
            FROM doc_tracks
        ) AS pts
    )

SELECT DISTINCT
       description, status, length_over_ground, object_type, equipment,
       ST_MakeLine(st_setsrid(ST_MakePoint(ST_X(sequenced_points.geom), ST_Y(sequenced_points.geom), q.elevation), 2193)) OVER(PARTITION BY sequenced_points.mln_seq, sequenced_points.elem_seq ORDER BY sequenced_points.elem_seq) AS geom
FROM sequenced_points
JOIN LATERAL (
    SELECT elevation
    FROM contour_line_points
    ORDER BY wkb_geometry <-> sequenced_points.geom
    LIMIT 1
) AS q
ON true;

A little while later (about an hour), and I have 4,850 tracks with elevation. Why is the number higher than the original? Well, that’s a good question. I believe it is because some tracks, for whatever reason, have more than one line associated with them. This could be because of a river or tidal crossing, where DOC have represented the track as two segments rather than one continuous line. Because of the way the ST_DumpPoints and partitition works, it returns a distinct row per line, rather than a multiline. At the moment, this is fine for me to continue to start to visualise this data. In the future, I will post-process these line shapes back into a multiline by matching against the track description and/or objectid.

Selecting a single track’s information is as fast as I would expect it to be, since I don’t need to recalculate each time I want the elevation of a track:

select * from doc_tracks_elevation where description='Upper Fenian/Fenian Gold/Adams Flat Trk';
=> Time: 20.296 ms

Now that I have calculated the basic elevation data, I’ve got some work to do to start visualising this data. I’ve also identified some optimisations in my current process I’d like to put in place.

For visualisation, I plan to generate a JSON file for each track, as well as a manifest that can be used for an autocomplete search. The idea would be that someone can search for a track, which then fetches the JSON containing the 3D track route. I would use the elevation points from this route to show a graph of the elevation over distance, as well as the route itself. I may experiment with web mapping libraries that support 3D, like Mapbox, to show the track against a 3D terrain layer. With the Z axis defined, I can also generate the actual travel distance for each track, rather than just distance over ground that comes from the route.

For the data processing stage, I would like to switch to using a DEM - probably the 8m derived DEM from contours. While exploding the contour points has yielded sufficient results, it’s not very high resolution. As far as I know, ST_DumpPoints just dumps the ‘corners’ of polygons, and not the intermediate points. That means that I might not be finding the correct elevation for a track point - it all depends on the orientation of nearby contours. Using a DEM with a 8m grid means that I can return more accurate elevation. I also would like to investigate which tracks have multiple lines, and ensure that I can post-process these back into their original form. I would also like to look at augmenting the track path data with some statistics - things like gain/drop, as well as the actual track distance as I mentioned above. Finally, to assist the visualiation, I would like to add marker points to the data every kilometer across the ground, and kilometer based on track distance. This would help to break the track path into segments to allow users to have a reference point for a particular area of climb or descent.

Rails supports transforming aria hashes into dasherized attributes

Thu, 09 Jun 2022 00:00:00 +1200

TIL that aria- attributes get the same treatment in Rails tag helpers as data-. That is, you can do:

link_to "X", y_path, aria: { pressed: true } -> aria-pressed="true" OR
link_to "X", y_path, "aria-pressed" => true -> aria-pressed="true"

I can’t find any multi-dash aria properties on MDN, but I imagine Rails will follow the same formatting rules of turning underscores to dashes, e.g.

link_to "X", y_path, aria: { active_status: "madeup" } -> aria-active-status="madeup"

This is a handy thing to know, since this lets long tag definitions be nicely wrapped and indented across multiple lines, as well as naturally grouping related attributes.

Create a trix editor with a hidden toolbar

Thu, 02 Jun 2022 00:00:00 +1200

Trix is fairly new to me, but now that it’s closely integrated with Rails, it’s become by go-to for rich text editing.

I’ve been working on a project where I wanted a text field with some basic rich text capabilities, but I wanted the input to look like a normal textarea. This would allow standard text entry, but allow users to use keyboard shortcuts to specify bold, italic and underlined text.

I came across this issue, which let me to a test fixture which gave the pointer I needed to do this. If we create our own <trix-toolbar> element, and hide it, then Trix will use this toolbar, but it will remain hidden.

<trix-toolbar id="hidden_trix_toolbar" style="display:none;"></trix-toolbar>
<trix-editor toolbar="hidden_trix_toolbar" input="text_input"></trix-editor>
<input type="hidden" id="text_input" />

Asserting a partial was rendered in an RSpec view spec

Mon, 02 May 2022 00:00:00 +1200

View specs are great. While I always keep in mind they are a unit test and do not test the full integration of a feature, I find that they are fantastic for asserting around all the edge cases of a feature - things like the expected validation messages showing up, or buttons appearing or disappearing based on different scenarios. I used to use system tests for this kind of thing, and find that these get really slow and fragile over time, while view specs stay blazing fast.

Often, I’ll break a template up into smaller pieces. Rails has a mechanism for this called partials. I’ll often break up things that are either their own “thing” (like a “card” display for an object, or “share links” for an object), or things that have some logic wrapped around them that I want to keep contained (for example, a button with a permission check wrapped around it).

When I’m testing such templates with a view spec, I will often have a view spec for the main template, and then another view spec for any more complex partials. In this scenario, I want to assert in the main template that the partial was rendered, but I don’t mind too much what is rendered, since I have another view spec specifically for that partial that checks that.

In this case, I’ve found that RSpec spies are a great way to do this. Spies are set up ahead of the test run, record interactions with themselves, and then allow the calls that they received to be asserted against. RSpec spies also have a handy method called and_call_original. This records the call, but then invokes the original method. I can use this to record that a particular call to render was received, but then still actually render the partial.

Here’s an example:

require "spec_helper"

RSpec.describe "widgets/show.html.erb", type: :view do
  it "renders the 'edit_button' partial" do
    assign(:widget, Widget.new)
    allow(view).to receive(:render).and_call_original
    render

    expect(view).to have_received(:render).with("widgets/edit_button", widget: widget)
  end
end

This test catches if the correct partial is not rendered, or if unexpected locals are passed to the partial, but doesn’t require me to assert in this spec exactly what the partial might render - I can put that in the partial spec.

I’ve found this is quite a handy technique and helps to make it quick and easy to test templates, since it allows the test to be super focussed on what the current template under test is doing without worrying about any conditions the parent template might otherwise need to take into account. It also allows for some nice dependency injection into the partial spec, since the locals can be adjusted in the spec (for example, to pass a different ‘user’ local, or a feature flag stub or something like that). Changing these settings just for one partial in a wider template would not otherwise be possible without splitting up the template into partials, and the template spec into partial specs.

This technique is also a candidate for refactoring into a custom matcher for view specs. I won’t cover that in this post, but since allow(view).to receive(:render).and_call_original doesn’t actually effect how templates are rendered, you could quite easily run this before all view specs, and then support a matcher like expect(view).to have_rendered_template() - which is very similar to what the rails-controller-testing gem does - but in a view spec rather than a controller spec (though I suspect this gem probably would work with a view spec since a view spec defines an anonymous controller under the hood).

CLI tool entr for running a command when a file changes

Thu, 17 Mar 2022 00:00:00 +1300

entr is a tool I’ve been using a bunch recently - mostly for watching source files, and running tests when I change the source file. entr can be piped one or more files, which means it’s really easy to use - just use ls, find, or any other tool that outputs filenames, and then pipe it to entr:

ls app/models/widget.rb | entr bundle exec rspec spec/models/widget_spec.rb

ls app/models/widget.rb app/presenters/widget_presenter.rb | entr bundle exec rspec spec/system/widgets/list_spec.rb

find . -name "*.rb" | entr rubocop

entr also has a handy shortcut, /_ to use a placeholder if you’re just watching a single file which cuts down on typing:

ls app/models/widget.rb | entr bundle exec rspec /_

There are plenty more options to check out in the entr man page

entr is available in Linux distros via yum, apt, etc. On Mac OS, it’s available in Homebrew: brew install entr.

Assert window.location properties with Jest and jsdom

Wed, 19 Jan 2022 00:00:00 +1300

I have a React component that is part of a wider Ruby on Rails application. It fetches a record, and if the record does not exist, it redirects to the server-rendered 404 page.

Unfortunately, while the user will see the ‘not found’ page, the HTTP status code will be 200 OK by default, because this page is static. Best practise would be to define a location block in your web server (Nginx, Apache, IIS etc) to respond with a 404 status when this static file is requested.

Having made this component, I’d like to add a Jest assertion that when the record is absent, the component does not render but instead tells the window to change it’s location.

Unfortunately, a naieve assertion does not work the way I expected:

it("navigates to /404 when the record is absent", () => {
  render(<MyComponent record={undefined} />);
  expect(window.location.pathname).toBe("/404");
});

Jest tells us that we can’t use window.location in our component, because jsdom can’t navigate:

Error: Not implemented: navigation (except hash changes)

That’s fair enough! Adding navigation would be a big increase in scope for jsdom, and I can’t blame anyone for not supporting this.

The trick to getting this to work is to redefine the location property on the window object in our test, so that we are using and asserting against an object that we control:

it("navigates to /404 when the record is absent", () => {
  // jsdom doesn't allow us to 'navigate' from a component, because how would that work?
  // Since we don't need to navigate, but just assert that navigation _would_ have occurred,
  // we can replace window.location with a URL, which has a pathname property we can assert
  // against
  Object.defineProperty(window, "location", {
    value: new URL("http://example.com"),
    configurable: true,
  });

  render(<MyComponent record={undefined} />);
  expect(window.location.pathname).toBe("/404");
});

You might notice that we’re using a URL object there. Why is that? Well, it turns out that window.location shares many properties with URL. Not all, and that’s important to know - especially for things like events and functions like reload(), but if you’re trying to assert that a property of the location changed, this is perfect. We replace the window.location property with a new URL object, change a property of that URL within our component, then assert that the property changed to the value we expect it to.

You might also notice that we’re passing an option to defineProperty - configurable: true. Adding this option means that other tests can change the value of this property later on the test. It’s also important to note that we’re replacing window.location from when this test runs onwards. To me, this is OK, since window.location raises errors in jsdom and doesn’t work the way we expect anyway, but if you want to avoid side effects, you can assign the original window.location to a variable, then assign the property back at the end of your test - or put all this in a before/after hook in your test file or suite.

References

https://stackoverflow.com/questions/54021037/how-to-mock-window-location-href-with-jest-vuejs
https://www.benmvp.com/blog/mocking-window-location-methods-jest-jsdom/
https://remarkablemark.org/blog/2018/11/17/mock-window-location/

Mid King Biv Trip Report

Thu, 06 Jan 2022 00:00:00 +1300

This trip report is really more a cautionary tale of overconfidence and inexperience leading to a challenging Tararua overnight trip to Mid King Biv.

Started: 11am from Holdsworth Carpark

Finished: 6:30pm the next day at Holdsworth Carpark

Travel Times:

Day 1:

2 hours to Atiwhakatu Hut (break)
1 hour from Atiwhakatu Hut to the swingbridge over Atiwhakatu Stream
20 minutes to Baldy turnoff
1 hr 10 minutes to bushline
40 minutes to Baldy summit
1 hour to South King
2 hours to Mid King Biv (< 1 km)

Total: 9 hours

Day 2:

~4 hours from Mid King Biv to Barton Track (South Mitre Stream bridge)
~3.5 hours from Barton Track (South Mitre Stream bridge) to Atiwhakatu Hut
~2.5 hours from Atiwhakatu Hut to Holdsworth carpark

Total: 10 hours

Day 1: Holdsworth carpark to Mid-King Biv via Baldy

This trip had been planned carefully to fit in around family commitments betweeen Christmas and New Years, so having found a couple of days spare, me and my tramping buddy for this trip were locked in and ready to go. The original plan was to travel up to Atiwhakatu Hut and either stay at Jumbo or carry on past to McGregor Biv, returning the next day over the Broken Axe Pinnacles, down Baldy, back to Atiwhakatu, and out.

The original plan, as scribbled over a topomap layer by me - we planned to travel this circuit left to right.

We met in Carterton at 9am, to sort out gear, supermarket shopping and last-minute coffees and food. The weather forecast promised showers for the first day, with a slight improvement the next. Facing a high liklihood of it being a fairly wet and miserable day, we revised our plans, opting to flip the route around to instead aim for Mid King Biv, stay overnight there, and try and get all the way across the Pinnacles and past Jumbo to head out the next day. Our plans were somewhat influenced also by very full hut bookings for Powell, Jumbo, and Atiwhakatu, which meant that we were unlikely to have much luck finding a bunk in any of these huts.

By 10am, we had finally organised ourselves and hit the road to Holdsworth, arriving at 10:45 and signing in on the intentions book at Holdsworth Lodge by 11am. We made steady time into Atiwhakatu, arriving after 2 hours for a lunch stop. The hut was looking suspiciously empty for the time of day considering it was supposed to be fully booked, with just one group in, but we scoffed down some wraps and leftover Christmas ham before continuing up the valley. From this point on, this was new track for me. The weather was fairly grey, with drizzle or showers most of the time.

The track from Atiwhakatu Hut to Mitre Flats Hut is reasonably well travelled, but obviously doesn’t receive the love and attention the Atiwhakatu Hut track receives (the track between Holdsworth and Atiwhakatu Hut was significantly rebuilt a few years ago to make more accessible for family groups). It sidles along beside Atiwhakatu Stream, dodging up and down the bank a couple of times to avoid slips and bluffs. After an hour, we reached the swingbridge over the stream.

After crossing the stream, the climbing began! The travel so far had been wet, but as flat as it ever gets in the Tararuas. From this point on, we had some solid climing to do from about 550m to Baldy at 1325m, and then on to South King at 1531m. The climb up to the bushline at Baldy wasn’t too bad - pretty similar to most other Tararua climbs - straight up the spur, with the contour lines on the topomap determining whether it was clambering or just puffing up steadily with the occasional breather. In this time, we travelled up through Beech, then mossy mountain beech, before this to thinned out and we arrived at the bushline just slightly after the posted time of 1 hour. As we’d approached the bushline, we’d moved properly into the cloud, so it was fairly steady drizzle and mist now. The posted time to Baldy at the intersection we’d left behind near Atiwhakatu stream is only to the bushline, and from here it was tops travel - no times, just distance to tick off. Travelling up through the tussock and mist, it took us another 40 minutes to reach the summit of Baldy - or at least a piece of DOC caution tape tacked to a wooden batten, what I assumed to be the summit.

Something that I noticed on this trip, particularly around the tops, is that the route is generally not poled in this area. Previous tops travel I’ve done in poor visibility around Tabletop/Kime Hut, and even Powell/Holdsworth has felt much safer since either warratahs or blue plastic poles mark the route there, and the poles are spaced so as to be reasonably visible even in poor conditions. On this route, it was largely a case of following the worn trail and the topomap, with an occasional cairn to let us know we were on the right trail. This generally worked out for us, though it requires more attention than following poles, and we did have a bit of trouble just after the Baldy summit where some erosion made it hard to pick out the route. In clearer conditions, this would be easier since the ridgeline would be visible, but in the mist it was quite hard to gauge whether we were following the crest of the ridge and not dropping off.

Erosion, large slips and bluffs between Baldy and South King provided good motivation for sticking to the top of the ridge.

With a couple of craggy bits to negotiate, it took us about 45 minutes to reach the South King Tarn where the route from Baldy we were on joins the route coming North from Jumbo via the Broken Axe Pinnacles, our planned route for the following day. By this time it was after 5pm, so while we still had some day to work with, we were in a bit more of a hurry to reach the biv and dry out a bit, so turned to the right to carry on up to the top of South King proper, reaching the top after another 15 minutes or so.

It was here that we made our major navigational blunder, attempting to follow a GPS track that I had found while researching for the trip. The source of my research, a Wilderness magazine post seemed to indicate that the route to the Biv dropped directly off South King to reach what looked on the topomap like a spot where the bushline was slightly higher. This route was tempting, since it saved us a bit more travel along the ridgeline and got us heading straight to the hut, hopefully into the shelter of the beech forest faster than other options. If we’d stopped for a little longer and made use of the cell reception we’d had since halfway up Baldy, we would have found that while this source does suggest a route off South King, several other sources instead recommend the spur from Mid King itself, just a bit further north. Unfortunately, we made the mistake of seeing the hut was only about a kilometer away, and went for it, dropping off South King towards Mid King Biv.

At first, travel was pretty easy. After a day of drizzle, the tussock was slippery, with the occasional spiky spaniard grass eliciting pained exclamations, but we were losing the altitude we needed to reach the hut quickly. We then started passing the occasional leatherwood shrub, which gradually became less and less occasional, until we were having to pick a route between them.

Leatherwood has a particular reputation around the Tararuas as an extremely hardy, thick leaved shrub, growing close to the ground, but becoming larger and larger closer to the bushline, around 2-3m. It has a particularly annoying springiness to it - it is difficult to break branches off leatherwood or even to push through, since any resistance is pushed right back at you.

The descent on the tussock had been steep, but manageable, but now we faced some challenging decisions. The leatherwood was now all around us, and from what we could see, was getting bigger and thicker, making it hard to see where we were putting our feet. While the mist restricted our visibility, we could see on our left hand side that the ground was dropping away sharply. In front of us, it was looking like it might do the same, and we edged forward to see if we could see a safe route down.

At this point, we had the first of several decision points about whether to carry on or climb back up to the main ridge and start again. Looking back, given the fact we had any doubt about our route, climbing back up would have been the correct move - but at the time, we were wet, and tired. The climb back up through slippery tussock to meet a ridge where we still wouldn’t definitely know where to go was daunting. We chose to carry on, assuming we’d be able to continue picking a route through. We looked on the map and reminded ourselves that the hut was not far away, and that the green on the map surely represented easy-travelled, sheltered beech forest.

Having chosen to continue down the slope, we found a spot that was the best of a bad lot. There really weren’t any safe options that we could see, and looking back, this was a really bad call for us to make, but we found a spot that looked like it could be clambered down, around 3 meters of maybe 75 degree slope, and set off. I went first, and managed 2 footholds before I ran out halfway down. By then of course, I was committed, and then second foothold sort of changed it’s mind about being a foothold anyway - I slid down the rest of the slope on my front, landing quite heavily. Fortunately for me, I landed flat on my feet - on an angle, I think I probably would have been looking at least at a sprain.

The only good thing about having done that drop is that with a different perspective, I could look for better route down for us to rejoin at the bottom of the slope. This meant that we didn’t both need to do as much of a slide to reach the bottom.

After this, the travel continued to become more and more challenging. We felt increasingly more trapped in our situation now, feeling that it was unlikely we’d get back up that slope. We continued to do our best to follow the topomap towards the hut, but constantly had to stop to try and scout the best route through the leatherwood, which had become much taller and larger as we’d dropped down another contour line or two. This travelling was particulaly tiring. We knew we were close to the hut, but not imminently so. The leatherwood grew very densely, concealing drops and holes. In places, the we tried to go under or through the leatherwood, causing cuts and scratches. In other places, the leatherwood was so dense that we would try and crawl over the top of it. Despite the topomap showing a single stream that led past the biv, after the wet day we’d had, we continued coming across other small streams, any of which could have been the one to follow. We crisscrossed the larger 2 of 3 streams several times trying to find clearer ground, and eventually ended up dropping into the largest stream for the final couple of hundred meters, splashing down the stream, ducking under or clambering over the leatherwood which at times completely tunneled in the stream, and a couple of times when the water was too obstructed, climbing back into the leatherwood to work around the obstacle. We reached the biv at 8pm, not long before dark, having approached it from the back via the stream. Fortunately, this biv, like many ex-NZFS huts, is painted bright orange.

This part of the day was both incredibly challenging and humbling. It took us over 2 hours to travel around a kilometer to the biv, and absolutely pushed our endurance and fitness to the limit. I don’t feel that we were ever in life-threatening danger (we carried a PLB for that scenario), but we were certainly beginning to reach the point where we would have had some difficult decisions to make around whether we continued to try and reach the shelter of the biv, or whether we should shelter where we were, and try again in the morning. The root cause of us ending up in this situation was essentially a failure to properly research and understand the best route to the hut, to make rushed decisions influenced by desire to get to shelter and hot food, and to not understand the characteristics of the Tararuas. What we thought was a higher bushline in green on the map, was of course leatherwood and other mountain shrubs. Other routes, such as the spur from Mid King, show clearer travel further down towards the biv, but instead we headed right for the green on the map assuming it would be sheltering forest, and ended up stuck in a basin of leatherwood as a result. We had several opportunities where we could have turned around, returned to the ridge, and probably found the actual route down, but since we’d already had a long day, as soon as we started dropping altitude, we felt committed to continuing - sunk cost strikes again.

We were able to use the 1 bar of cell reception at the biv to advise our contacts that we had arrived safely, but tired. Having expended significant energy getting to where we had, we discussed and agreed that to travel over the Broken Axe Pinnacles with another long, exposed day before reaching Jumbo Hut would be pushing our luck and stamina. We would instead drop down to the valley near Mitre Flats Hut, and cross a saddle back to Atiwhakatu Hut to complete a loop. We had a quick dehy meal, a hot drink and crawled into the 2-person ‘dogbox’ biv to get some sleep.

Day 2: Mid King Biv to Holdsworth Carpark

In contrast to the misty and rainy day we’d had previously, the weather was looking much better in the morning, with the sun shining through the trees, giving everything a very gold and green look. We knew we had a long day ahead of us - we were anticipating it taking 2 hours for us to drop to the swingbridge near Mitre Flats Hut over the South Mitre stream, then 3 hours to get from there to Atiwhakatu Hut, and another 2 hours or so out - 7 hours total. We had a quick breakfast and coffee, and hit the track. I was cursing our navigation error from the previous day as we followed a clear route marked with orange triangles for 10 minutes to the bushline. From here, the triangles were replaced with pink tape, but the route was still clearly obvious, and it looked as though if we’d dropped off the spur from Mid King instead of South King, we would have had a much easier time.

As we reached the bushline, we looked up towards the Kings, and across towards the Pinanncles, and had a quick checkin. Should we go over the ridge route to Jumbo after all? In the end, we opted to continue dropping to Mitre Flats, for a few reasons:

We’d already been in touch with our contacts to advise we were going back through the valleys. Changing our minds would require us to stop, plan out our route a little, and get in touch with them again.
Even though it was quite sunny where we were, the cloud was still covering the main ridge. Neither of us had travelled that route before, and I knew that the Broken Axe Pinnacles in particular are narrow and can be a bit technical, especially in poor visibility.
A good night’s sleep had restored us somewhat, but we were still feeling pretty battered from the previous day. It seemed best given some poor decisions the previous day to take the less risky option of travelling down and out through the valley.
We considered seeing if we could arrange a pick up at The Pines, where the most direct route from Mitre Flats Hut emerges. This would require a bit of coordination with our contacts in 1-2 bar cell coverage, and would still require us to get to Mitre Flats and then out, which is signposted as 4 hours, plus another hours drive to reach Holdsworth carpark. We decided that the overall travel time for this option meant that it wasn’t partciularly compelling since we were tired, but not injured, and getting ourselves back to the cars under our own power meant we could travel to our own schedule. If either of us had been even mildly injured, we would have taken this option, since it would have been the fastest way out.

The track from Mid King to Mitre Flats isn’t on any maps, even the Topo50 series. The hut intentions book made it clear that it’s quite a well-travelled route though, with many people coming up from Mitre Flats on a day trip. My primary concern for this route was that we wouldn’t be able to find the track, but it turned out to be fairly obvious where to go. We did miss the first few meters and overshot, but seeing a cairn on the other side of some leatherwood, were able to deploy our new scrub-bashing skills to join the track. The track is taped, but very intermittently, so we had to pay attention to where we were heading to make sure we were still following the track. This was easier at the start, while the beech forest was still quite stunted, and the path through the moss was quite obvious. As we got further down into the bush, things got more complicated, with a lot of dead ends and spots where treefall had concealed the line of the track. It was a relief to spot a piece of tape or a cairn every now and then to let us know we hadn’t gotten badly lost.

When we did lose the track, we had the advantage of being on a spur, meaning we could traverse across the top of the spur until we found the most promising-looking worn route, which we would follow until we found a cairn or tape. The spur route effectively heads straight down the spur, dropping from about 1100m to just under 500m by South Mitre Stream.

The aim is to meet the South Mitre Stream where Baldy Creek joins it. We mostly managed this aside from a diversion where we left the spur too soon, and ended up down at Baldy Creek about a kilometer from South Mitre Stream. We fossicked around a bit and contemplated going down the stream, but in the end decided to climb back up the 50 or 60 meters to rejoin the spur. After we eventually reached the point where stream and creek intersected, we started heading down beside the stream towards the swingbridge on the Barton Track. This travel wasn’t too difficult since it was reasonably flat and open beside the stream, but we didn’t find a particularly clear route at any point, and lost it completely just 100m or so from the swingbridge. I suspect it climbed up to join the Barton Track after the bridge, but since we could see the bridge, we dropped into the stream and crossed the stream to avoid a small bluff the bridge was built onto, and re-crossed the stream via the bridge.

Overall, we expected this section of track to take us 2 hours, and it had taken us 4 hours. Being used to front-country type travel, I think this is probably the longest stretch of non-triangle-marked track I’ve done, and it was surprisingly time-consuming and stressful to make sure that we were following the spur and not dropping to one side or the other too early, and to regain the track when we left it, which we did many times (usually only by 10 meters or so). Probably a more experienced tramper wouldn’t find this route difficult, since the bush is quite open and easy to travel through, but I hadn’t realised how dependent I was on following from one marker to another. By the time we joined the Barton Track and stopped for a break beside the bridge, I wasn’t feeling great - we were well behind schedule on what we already knew was going to be a fairly long 7 hour day, and we still had a lot of ground to cover.

We had considered stopping in at Mitre Flats Hut for a break, which was only a couple of hundred meters away in a clearing. Given we were “behind schedule”, we skipped this option, and started trudging. One of the downsides of the Barton Track which runs between Mitre Flats and Atiwhakatu Huts is that it goes up and down - a lot. While we’d dropped on the spur down to 400m, we had to immediately climb back up to 600m, and traverse approximately along this contour (I say “approximately” because in reality the track was always going up and down by 20m or so, never flat), before dropping 100m to cross a stream, then climbing 200m to meet the Baldy turnoff we’d left yesterday just short of 700m. Reaching the Baldy turnoff, psycologically, was a big boost - it meant a lot to have closed the loop and now be retracing our footsteps over ground we’d covered - we knew what was coming, and more importantly, knew that Atiwhakatu Hut was only about 1.25 hours from the turnoff. We made slow progress between the Mitre Flats swingbridge and Baldy turnoff, about 2.5 hours (signposted time is 1.75 hours). Largely this was due to being pretty tired, and needing a lot of mini-breaks and breathers. After reaching the Baldy turnoff, we gained a little speed and stumbled into Atiwhakatu Hut for lunch about 3:45pm.

Evidence of how tired I was from the previous day's challenges - in a 10 hour day, I took a single photo

At Atiwhakatu Hut, we took a little time for break, and treated ourselves to the last of our dry clothes we’d been saving for the last stretch. I’d considered a longer stop at this hut - perhaps an hour or two - to rest up a bit before continuing, but after having a bit of food, decided to carry on and get it done. We left just before 4:30, and immediately noticed the easier travel on the well-graded and gravelled Atiwhakatu Track, which served as a relatively gentle 2-and-a-bit hour cooldown out to the carpark, which we reached at 6:30pm, just in time to get a message out before my 7pm due-out time I’d arranged earlier in the day at the Baldy turnoff where we found a little cell coverage. Having already split the gear up back to their respective owners while stopped at Atiwhakatu, we were pretty relieved to each jump in our own cars and head back home, after a humbling trip, and one that to me has served as a reminder about how quickly risk factors can add up to quite a difficult situation and trip.

The approximate routes we followed. Purple:The route we followed off South King to Mid King Biv. Red:A guess of the route we should have taken from Mid King to Mid King Biv. Green:The approximate route from Mid King Biv to meet Barton Track near Mitre Flats Hut, including approximately where we mistakently dropped off the spur to Baldy Creek.

Retrospective

In retrospect, there are a number of factors that I will consider in the future:

The weather forecast was for showers through the day. This meant that we were not going to stay dry. Given we were already looking at a trip that was going to be reasonably challenging for our skill level, we could have considered an alternative destination or route.
While Atiwhakatu Hut and Jumbo were both booked, we had camping stuff with us, including a tent. We had mostly ruled out staying at Jumbo because of the bookings, when actually we could have probably either camped or found a bunk, and this trip would have still allowed us to venture onto the tops, but in a less remote area more in-line with our skill level. The quietness of Atiwhakatu Hut on the first day could have prompted us to change our trip plans and head to Jumob, but we already felt committed to what we had decided.
When we reached South King, I had a couple of different webpages downloaded to my phone to refer to, but I picked the first of these and stuck with it since it included a route to follow on a topomap. As it turns out, this route wasn’t the best way to go - I wonder if it was drawn from memory after the trip, or if that group just didn’t mind travelling through the leatherwood as much as we did. If I’d stopped and properly considered all of the information I had on hand rather than rushing to reach the biv, there’s a high chance we would have continued to Mid King and probably found a clearer route down.
Navigating our own way along an unmarked route through the bush takes much longer than just following a track, even with similar distances. The fact that this section took twice as long as we thought it would was disspiriting and exhausting, and made the rest of the day much harder to get through. If we’d been able to accurately forecast our travel time for the day, and given the reasonably good weather conditions when we left the biv, we probably would have chosen to go back up and along the ridge to Jumbo, or even back down Baldy. We still would not have had good visibility, but tops travel time is easier to assess, and we would have reached Atiwhakatu Hut after around 3.5 hours via Baldy, or Jumbo Hut after perhaps 4 or 5 hours.

Overall, I learned a lot from this trip about where my comfort level lies, but also the endurance I have when necessary. We’ve not been put off future trips, but have already agreed that given our skill and comfort levels, our next trip in this area will be to head to Jumbo Hut and proceed out to the tops from there with a hut at our backs, rather than overextending ourselves to reach a remote shelter.

Quickly access Ruby Mail object parts

Sun, 12 Sep 2021 00:00:00 +1200

Often, mailer specs will start with a simple assertion like expect(mail.body).to include "Some text in the expected email here". This works for a while, but quickly breaks down when an email grows more complex, such as having special characters causing it to be encoded as 7bit or QuotedPrintable, having attachments, or supporting different formats (like plaintext & HTML).

I’ve always tried to do various horrible things relating to traversing the parts of the message object, and extracting the part that has a mimetype of text/html. I should have known that the Mail library would have a shortcut for this!

Given a mail object, you can call .html_part, and .text_part to access the HTML and text parts of the message respectively. Under the hood, these helpers use find_first_mime_type, which means you can find any other format you may support by passing the mimetype to this helper. Note - this helper filters out attachments - if you want to check what attachments a message has, you should use .attachments for that.

With these helpers, the test can handle all sorts of formats and attachments while still supporting a nice stable assertion - expect(mail.html_part.decoded).to include "Some text in the expected email here".

Access ActionView context in an RSpec test

Thu, 02 Sep 2021 00:00:00 +1200

I recently created a Presenter object to wrap around an ActiveStorage::Blob that I wanted to decorate with some presentation methods for things like a human file size and content type.

The initialisation signature for my presenter looks like this:

class MyPresenter < SimpleDelegator
  def initialize(obj, view_context)
    super(obj)
    @view_context = view_context
  end

  def human_file_size
    @view_context.number_to_human_size(byte_size, precision: 1)
  end

  # more presentation methods...
end

I went to test this, and got a bit stuck! I needed to provide a view context to this class that I could call view helpers on. I could pass in a double, but would then have to stub out every single view helper that the presenter used. This would mean my tests would be testing my stubs, not the actual behaviour. I could use a double.as_null_object, but that assumes that all the helpers are OK to return nil - that’s an extremely uncommon thing for a view helper to do!

I stopped and thought for a minute - I know that there are several types of RSpec tests where you can access a view context - particularly controller and view specs. I decided to begin my investigation with view specs, since this was most closely aligned with what I was trying to test.

I started with the source of rspec-rails, particularly lib/rspec/rails/view_rendering.rb. It sort of looked relevant, but it looked like it was using a lot of methods coming from elsewhere - things like controller, and lookup_context.

Next, I looked to see where this module was used, and there I found lib/rspec/rails/example/view_example_group.rb. This looked even more relevant - I could see things like controllers, and helpers being set up. Great! All of these were using a method named view, though, and while the method documentation for this looked relevant - “The instance of ActionView::Base that is used to render the template.” - the actual method body just returned the value of _view - and this wasn’t instantiated inside this module.

Looking at the top of this module, I could see an include from Rails itself - include ActionView::TestCase::Behavior. This was a real giveaway, because all of the other top-level includes were for other modules from within rspec-rails.

I switched over to the Rails repository, and headed into lib/action_view to find this class, and I found it at actionview/lib/action_view/test_case.rb.

Within this class, I found the view method. This method was aliased to _view, which was what RSpec was using.

In setup_with_controller, I could see that the @controller variable was set to an instance of ActionView::TestCase::TestController. Neat! This means that the set up of a view context is entirely contained to this class, and I can do exactly what the view spec module is doing - include ActionView::TestCase::Behaviour, and then access the view method in my spec to use the view context.

As an example:

RSpec.describe MyPresenter, type: :presenter do
  include ActionView::TestCase::Behavior

  describe "#file_size_human" do
    it "returns the expected value" do
      format = ActiveStorage::Blob.new(byte_size: 1500)
      expect(described_class.new(format, view).file_size_human).to eq "1.5 KB"
    end
  end
end

Access parent node with Capybara

Wed, 02 Jun 2021 00:00:00 +1200

Sometimes I’ll need to find an element in relation to another. An example of this is a link with a single icon element inside it:

<a href="/example"><img src="icon.png" alt="Example Page" /></a>

Because Capybara element finders can be chained (e.g. you can do find().find().find() to traverse down elements, the ability of XPath to traverse the DOM tree can be used to easily find the parent node:

page.find("img[alt='Example Page']").find(:xpath, "..")
# => Returns the 'a' element

Any XPath can be passed here, for example if you wanted to find a sibling of the parent or traverse further up the DOM tree. Handy!

— via Stackoverflow

ActiveRecord::Base.update actually calls model setter methods

Sun, 16 May 2021 00:00:00 +1200

When an update method like @my_record.update(test_attr_1: "foo", test_attr_2: "bar") is called, my assumption was that those attributes immediately got passed to some ActiveRecord magic to update the record. Not true! Each attribute key calls the setter on the object - in this case, @my_record.test_attr_1=(new_value) and @my_record.test_attr_2=(new_value) would be called.

This is neat because you can ‘redirect’ that attribute assignment to actually do something else, like set a different column. Here’s what an example that shows how to allow a boolean to be passed into an attribute called “enabled” that instead sets a “enabled_at” timestamp:

  def enabled=(flag)
    self.enabled_at = flag ? Time.zone.now : nil
  end

  def enabled?
    enabled_at.present?
  end

Usage:

# Updating
thing = Thing.last
thing.update(enabled: true) # Sets enabled_at to Time.zone.now and saves
thing.enabled? # => true
thing.update(enabled: false) # Sets enabled_at to nil and saves
thing.enabled? # => false

# Instantiating
thing = Thing.new(enabled: false)
thing.enabled? # => false

# Assign before save
thing = Thing.last
thing.assign_attributes(enabled: true) # Enabled, but not saved yet!
thing.save! # OK, saved now

# And of course this works the same as any other model attribute
# when it comes to form/template stuff:

# In ThingController#create:
params.require(:thing).permit(:name, :description, :enabled)

# Template:
<%= f.check_box :enabled %>

You could even do more than one assignment - for example, a publish boolean might manage more than one timestamp:

def publish=(should_publish)
  if should_publish
    self.published_at = Time.zone.now
    self.unpublished_at = nil
  else
    self.published_at = nil
    self.unpublished_at = Time.zone.now
  end
end

In the scenario of publishing, this method is useful because if a record is unpublished, knowing when this happened is relevant - just setting published_at to nil wouldn’t indicate this.

Note: Anything more complex than this should be in its own class. A service object is probably what you are looking for.

Handy Ruby shorthand for testing class ancestors

Thu, 29 Apr 2021 00:00:00 +1200

Today I was working on a method that had to take a string, and resolve it to a class. This method needed to perform some validation on the model class, to make sure it was a type we expected.

My first iteration of this didn’t work - I expected that I could use Object#kind_of?, however this only works for testing whether an instance is a kind of class or module, rather than the class itself. This check looked like:

model.kind_of?(ParentThing)

My next iteration did a direct superclass comparison:

model.superclass == ParentThing

This worked, but was more brittle than I wanted, since it would only work when ParentThing was the direct ancestor of the class.

I checked in on my company Slack, where Eoin reminded me of the ancestors method, which returned an array of the ancestors of the class or module the method was called on - essentially, the superclass chain. From this, I composed the test:

model.ancestors.include?(ParentThing)

This worked, was flexible enough to allow ParentThing to not be the direct ancestor, so I went to commit it, when Rubocop conveniently told me of a recommended replacement:

Performance/AncestorsInclude: Use <= instead of ancestors.include?

This led me to the Rubocop documentation, then to the Fast Ruby reference, indicating that not only does that shorthand exist, it is also faster. Neat!

The <= method, along with a few other handy-looking examples, are documented in the Ruby documentation.

mod <= other → true, false, or nil

Returns true if mod is a subclass of other or is the same as other. Returns nil if there’s no relationship between the two. (Think of the relationship in terms of the class definition: “class A < B” implies “A < B”.)

My final check implementation became a lot simpler:

model <= ParentThing

Go offline with Capybara, Selenium and Chromedriver

Thu, 29 Apr 2021 00:00:00 +1200

There’s some preamble here. Jump to solution

Recently, I’ve been working on a client project that needs to function offline. This means that I have a subset of tests where I need to change the connectivity status of the browser driving the tests to simulate being offline to make assertions that the application is behaving correctly.

I already knew that the devtools had the capability to simulate either being completely offline, or to impose latency and throughput on the connection to simulate poor connectivity and/or older connection standards like 2G.

Because of this, I suspected that there would be an API to change the network connection status, I just had to track it down. After a bit of a journey through Selenium’s different layers (and languages - often I’m stumble across what I’m looking for in a different language binding for Selenium like .NET or Python, and then have to go and find the equivalent bit of code in Ruby), I found the solution:

#
# Sets network conditions
#
# @param [Hash] conditions
# @option conditions [Integer] :latency
# @option conditions [Integer] :throughput
# @option conditions [Boolean] :offline
#

def network_conditions=(conditions)
  @bridge.network_conditions = conditions
end

— Source

This method calls the setter on @bridge, which invokes the command through chromedriver to actually set the conditions:

# File 'selenium/webdriver/chrome/bridge.rb', line 45

def network_conditions=(conditions)
  execute :set_network_conditions, {}, {network_conditions: conditions}
end

This method is defined as an ‘extension’ (basically, a module) on the Capybara page’s driver’s browser (this is essentially a pointer to the running browser application). It functions as a setter method in Ruby, so we can just pass a hash to set the variables we want. It’s worth pointing out that I did have to provide all of these hash keys - it doesn’t appear to be possible to provide just one.

Here’s how I set my test browser to offline:

page.driver.browser.network_conditions = { offline: true, latency: 0, throughput: 0 }

And online again:

page.driver.browser.network_conditions = { offline: false, latency: 0, throughput: 0 }

You could also adjust latency and throughput if you wished, so while the application was technically online, it perhaps had very high latecy and very low throughput. This would be useful for testing some kind of ‘slow network connection’ message, for example.

Access browser console messages from a Capybara test

Tue, 27 Apr 2021 00:00:00 +1200

A very handy API I’ve stumbled across today:

page.driver.browser.manage.logs.get(:browser)

This returns an array of Selenium::LogEntry instances which have the expected properties - level, timestamp, and message. You can use standard Enumerable methods to filter the log entries down to the information relevant to you.

For example, all error messages:

page.driver.browser.manage.logs.get(:browser).select { |le| le.level == "SEVERE" }.map(&:message)
=> ["http://127.0.0.1:40539/madeup - Failed to load resource: net::ERR_CONNECTION_REFUSED", ...]

Use Foundation data attributes in a React functional component

Tue, 06 Apr 2021 00:00:00 +1200

I’ve been porting some previously server-rendered Rails templates to instead sit within a React application. This particular application uses Foundation Sites for base styles, and uses some of Foundation’s components which use Javascript to function - mostly basic stuff like data-dropdown, data-accordion, data-tabs. Foundation (still) uses jQuery for setting up these elements, and by default, initializes based on the ready event. This is at odds with React, since components are dynamically added and removed from the DOM.

To get this working, I first moved the markup that was in a template into a functional component:

// app/frontend/components/Dropdown.jsx
import React from "react";

export default () => (
  <div>
    <button class="button" type="button" data-toggle="example-dropdown">Toggle Dropdown</button>
    <div class="dropdown-pane" id="example-dropdown" data-dropdown data-auto-focus="true">
      Example dropdown.
    </div>
  </div>
);

CodeSandbox

The component renders, but the dropdown doesn’t work! This is because the component has been mounted to the page after Foundation has initialised.

To resolve this, I used two React hooks useRef and useEffect. useRef is used to establish a reference to the DOM node, and useEffect is used to run a function when the component has rendered.

Here’s how the hooks are used in the working component:

// app/frontend/components/Dropdown.jsx
import React, { useRef, useEffect } from "react";
import jQuery from "jquery";
import "foundation-sites";

export default () => {
  const foundationRef = useRef(null);
  useEffect(() => jQuery(foundationRef.current).foundation(), [foundationRef]);

  return (
    <div ref={foundationRef}>
      <button class="button" type="button" data-toggle="example-dropdown">
        Toggle Dropdown
      </button>
      <div
        class="dropdown-pane"
        id="example-dropdown"
        data-dropdown
        data-auto-focus="true"
      >
        Example dropdown.
      </div>
    </div>
  );
};

CodeSandbox

The first argument to useRef is set to null, as there is no additional value to use until the component is mounted. useEffect accepts the ref as a dependency, as it only needs to render when the ref changes (e.g. when the component is mounted), not every re-render.

Foundation JS is getting pretty dated these days - it’s a heavy dependency, and still relies on jQuery to work correctly. For projects already using the framework though, it provides a useful mixture of styles and scripts, so it’s worthwhile being able to continue to leverage these javascript-backed behaviours in React components.

What happens when I run 'rake db:migrate' in Ruby on Rails?

Sat, 03 Apr 2021 00:00:00 +1300

This post is written stepping through the most recent stable branch of Rails on Github: 6-1-stable. It may not remain up to date.

Reading time: about 20 minutes

Skip table of contents

How is the ‘db:migrate’ rake task defined?
How is my migration run when I run bundle exec rake db:migrate?

How is the ‘db:migrate’ rake task defined?

The Rails gem includes ‘activerecord’ as a gem dependency:

 s.add_dependency "activerecord",  version

— rails/rails.gemspec

version is set by reading a file named ‘RAILS_VERSION’ in the ‘rails’ gem directory. In this branch, this is 6.1.3.1. This means that the core gem versions like ActiveRecord track the Rails gem version exactly.

The activerecord gem adds it’s own ‘lib/` directory to Ruby’s load path:

  s.require_path = "lib"

— rails/activerecord/activerecord.gemspec

The ‘activerecord’ gem ‘railtie’ file is `require`-d by Rails

This can happen in two ways: either by config/application.rb requiring ‘rails/all’, or by config/application.rb specifically loading the Railtie:

require "active_record/railtie"

&mdash: rails/railties/lib/rails/generators/app/templates/config/application.rb.tt)

ActiveRecord Railtie is evaluated

A ‘Railtie’ appears in most Rails-related gems. It’s like an initializer, but for a library to set itself up within Rails, rather than within your application. Railties are used to do things like set up default configuration, declare initiaizers, and define Rake tasks:

rake_tasks do
  namespace :db do
    task :load_config do
      if defined?(ENGINE_ROOT) && engine = Rails::Engine.find(ENGINE_ROOT)
        if engine.paths["db/migrate"].existent
          ActiveRecord::Tasks::DatabaseTasks.migrations_paths += engine.paths["db/migrate"].to_a
        end
      end
    end
  end

  load "active_record/railties/databases.rake"
end

— rails/activerecord/lib/active_record/railtie.rb

See that ENGINE_ROOT bit? That checks to see if ActiveRecord is being run within an ‘engine’. A Rails engine is a bit like an application-within-an-application, and it can have it’s own migrations. That’s what this code is doing - if it’s running in an engine, it checks to see if that engine has a db/migrate folder. If it does, it adds the engine’s migrations to the list of all the places ActiveRecord’s database tasks (like db:migrate) should look for migration files.

For the rest of this guide, we’ll assume we’re not in an engine, so this block doesn’t affect us. The load line is the important bit.

ActiveRecord Database Rake tasks are defined

‘databases.rake’ has a special ‘rake’ extension. ‘rake’ files are specially evaluated in the context of the rake tool, that defines such methods as ‘namespace’ (group tasks), ‘desc’ (describe what a task does), and ‘task’ (describe a sequence of actions to take).

I’m not actually sure what Rake does to automatically be able to load .rake files with the DSL methods already loaded. Maybe another post sometime?

Within this lengthy file (ActiveRecord makes use of a lot of Rake tasks to manage databases), the db:migrate task is defined:

desc "Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)."
task migrate: :load_config do

— rails/activerecord/lib/active_record/railties/databases.rake

The first thing that will run here is the load_config task (that key: value syntax is how Rake task prerequisites are defined).

load_config is pretty simple as well:

task load_config: :environment do
  if ActiveRecord::Base.configurations.empty?
    ActiveRecord::Base.configurations = ActiveRecord::Tasks::DatabaseTasks.database_configuration
  end

  ActiveRecord::Migrator.migrations_paths = ActiveRecord::Tasks::DatabaseTasks.migrations_paths
end

If there aren’t any configurations, it finds them (ActiveRecord::Tasks::DatabaseTasks.database_configuration is worth checking out - this is where your database.yml file is loaded into ActiveRecord/Rails!)

It also passes along the migrations_paths that we saw earlier when engines were being set up to the ActiveRecord::Migrator. This class will come up again in the migration process, but for now, this is just setting up the migrator to know where to load migrations for.

Speculatively, the reason this is done is because it would be technically possible to create more than one instance of a ‘migrator’, pointing at different sets of migrations. Passing this configuration in decouples the class that actually ‘runs’ the migrations from the class that controls the registry of migration file locations.

This task depends in turn on the environment task. This isn’t actually part of ActiveRecord, but of Rails.

Basically, it “loads” the application, all of it’s dependencies, initializers, etc. - all that slow stuff. No need to go into further detail here, other than to note that without depending on this task, things like Rails.env, Rails.root, and lots of other useful things aren’t available.

And now back to the ‘migrate’ task:

desc "Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)."
task migrate: :load_config do
  original_db_config = ActiveRecord::Base.connection_db_config
  ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env).each do |db_config|
    ActiveRecord::Base.establish_connection(db_config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
  end
  db_namespace["_dump"].invoke
ensure
  ActiveRecord::Base.establish_connection(original_db_config)
end

Looks like some connection shenanigans going on there. The current database connection config is stashed in a variable, and restored using an ensure block (this code will always run, even if an exception is raised). A list of configurations is looked up matching ActiveRecord::Tasks::DatabaseTasks.env, and for each of this environments, ActiveRecord::Tasks::DatabaseTasks.migrate is run.

I’ve never used this, but since I was curious, ActiveRecord::Tasks::DatabaseTasks is quite simple:

def env
  @env ||= Rails.env
end

— rails/activerecord/lib/active_record/tasks/database_tasks.rb

In other words, unless someone or something sets it otherwise, the environment that the migrations are run in will be the same as the rest of your application (Rails.env, set from environment variable, command line option, etc). But migrations can be run in a different environment from the rest of your application. This could be used if you had a different database user who had additional privileges to change the database schema from the user that your application normally runs with.

How is my migration run when I run `bundle exec rake db:migrate`?

ActiveRecord::Tasks::DatabaseTasks.migrate

This method is the first sign of migrations actually being ‘run’:

def migrate
  check_target_version

  scope = ENV["SCOPE"]
  verbose_was, Migration.verbose = Migration.verbose, verbose?

  Base.connection.migration_context.migrate(target_version) do |migration|
    scope.blank? || scope == migration.scope
  end

  ActiveRecord::Base.clear_cache!
ensure
  Migration.verbose = verbose_was
end

First, the target version is checked. Usually, this version isn’t actually provided - in this case, it’s assumed the runner of the migrations wants the “latest” version. It is possible to pass an environment variable named “VERSION” though - and when it is provided, it must match the expected format.

Next, the scope is grabbed out an environment variable called ‘SCOPE’, if present, and so is the ‘verbose’ setting for migrations (unsurprisingly, using an environment variabled named ‘VERBOSE’).

There’s then a block of code that is the next step to run migrations - we’ll loop back to that, but assuming the migrations all go smoothly, the ActiveRecord cache is cleared with ActiveRecord::Base.clear_cache!. This method clears a bunch of instance variables that are used to save database schema lookups at runtime.

If it looks like this method is just looking up a bunch of runtime configuration from environment variables then calling something else - that’s because it is. This block of code is simply decoupling environment-variable-based config from the actual migration process. It’s clear from the code block above that the next step lies in ActiveRecord::Base.connection.migration_context.migrate.

`ActiveRecordBase.connection.migration_context`

This method is called on a connection - that’s something a bit different from what’s been seen previously. Remember how in the db:migrate rake task definition, connections were switched around based on the migration environment? That’s where this connection comes into play. From here on in, the migration code is going to need to know which database to connect to, and how - for this reason, migration_context is defined on ActiveRecord::Base.connection (which is an implementation of an AbstractAdapter)

it can call self, and have an actual connection to the database.

migration_context actually just builds another class - MigrationContext. It passes two arguments to the initialize method of this class:

migrations_paths - this has been seen before - it can be overridden in the connection config, it looks like (I suspect you can specify migrations_paths in config/database.yml for this to take effect), but most of the time it will fall back to the ActiveRecord::Migrator.migrations_paths method that was set in the last method.
schema_migration - this has a slightly complex definition, but most of the time ends up being either a ActiveRecord::SchemaMigration class or subclass.

ActiveRecord::SchemaMigration is an interesting piece in it’s own right. It’s actually a model - a model that’s backed by a database table you may have seen before called ‘schema_migrations’. You also may not have seen it before, because most of the time it never needs to be touched. It holds the ‘state’ of which migrations have run. It has a very simple schema

a single string timestamp column which is also the primary key.

ActiveRecord::MigrationContext

This class looks complicated, but is mostly a state machine to control the direction to migrate in. There are a bunch of helper methods to migrate the database, back (arbitrarily or to a particular version), forward (arbitrarily or to a particular version), one step backwards from the current version, or one step forwards from the current version.

Because we know this migration context is being called from database_tasks.rb, we’re only interested in the migrate method:

  def migrate(target_version = nil, &block)
    case
    when target_version.nil?
      up(target_version, &block)
    when current_version == 0 && target_version == 0
      []
    when current_version > target_version
      down(target_version, &block)
    else
      up(target_version, &block)
    end
  end

— rails/activerecord/lib/active_record/migration.rb.

target_version came from that VERSION environment variable, but most of the time, if you’re running bundle exec rake db:migrate in a normal situation, it’ll be nil, so we call up(nil). If a specific version was passed, and it’s less than the current migration version (which is looked up from that ‘schema_migrations’ table), then down(target_version) is called.

The up method calls the next part of the chain, the Migrator:

def up(target_version = nil)
  selected_migrations = if block_given?
    migrations.select { |m| yield m }
  else
    migrations
  end

  Migrator.new(:up, selected_migrations, schema_migration, target_version).migrate
end

— rails/activerecord/lib/active_record/migration.rb

This method looks for migrations to run based on the block, if one is provided - in ActiveRecord::Tasks::DatabaseTasks.migrate one is provided:

migrate(target_version) do |migration|
    scope.blank? || scope == migration.scope
end

In other words, if ENV["SCOPE"] was provided to the Rake task, only migrations that match that scope will be run. Migration ‘scoping’ is based on the filename of the migration, according to the regular expression stored in ActiveRecord::Migration::MigrationFilenameRegexp:

/\A([0-9]+)_([_a-z0-9]*)\.?([_a-z0-9]*)?\.rb\z/

In other words, migrations are expected to be named with two or three parts:

a version number (integer 0-9). This is usually a timestamp as seconds since epoch, but can just be an incrementing number. It has to be present, otherwise a IllegalMigrationNameError will be raised
the migration name - alphanumeric characters plus underscores. This is the part you normally parse out yourself when scanning the db/migrations/ directory - something like add_full_name_to_users.
the migration scope - after a full-stop character, but before “.rb”. The scope is used to group migrations. Often, this feature is used by Rails engines to identify the origin of a migration. By default, migrations are unscoped.

This is all useful to know, but for the purposes of continuing our journey, most of the time, scope is blank, so we run pass all migrations along to the Migrator class in our up method.

ActiveRecord::Migrator

This class acts as the ‘controller’ for a set of migrations. The primary responsibility it has is to prepare a list of migrations to run, and to then actually run these migrations (with some advisory lock decoration when this is is supported).

When the class is initialized, it sets up the schema migration database table (a reminder - ActiveRecord::SchemaMigration is an internal database table that is used to store the version numbers of migrations that have been run), and the internal metadata key-value table that ActiveRecord uses to store metadata, like the environment that migrations were last run in.

This class supports running a single migration (provided by ‘target version’ - e.g. that ENV['VERSION'] environment variable again). It uses the run method for this In the case of doing a normal bundle exec rake db:migrate though, it performs a different action, called migrate (unsurprisingly). Migrate wraps an advisory lock around running the migrations. An advisory lock is a feature of some of some database engines supported by Rails, which is just that - advisory. The migrations check for an advisory lock, and if one exists in the database, a ConcurrentMigrationError is raised, halting the migrations. If a lock doesn’t already exist, a new advisory lock is created, preventing any other migrations being run until the advisory lock is released.

With the advisory lock issued, or if advisory locks are not enabled, migrate_without_advisory_lock is called, getting a step closer to running migrations.

def migrate_without_lock
  if invalid_target?
    raise UnknownMigrationVersionError.new(@target_version)
  end

  result = runnable.each(&method(:execute_migration_in_transaction))
  record_environment
  result
end

invalid_target? only runs if an actual target_version is provided - it ensures that the version that was provided is not zero, and that a migration actually exists that matches that version.

The actual ‘migration’ is pretty simple - it calls ‘execute_migration_in_transaction’ with each migration. The runnable definition is worth looking into, since it’s actually the thing that decides what will be run:

def runnable
  runnable = migrations[start..finish]
  if up?
    runnable.reject { |m| ran?(m) }
  else
    # skip the last migration if we're headed down, but not ALL the way down
    runnable.pop if target
    runnable.find_all { |m| ran?(m) }
  end
end

The migrations method is also relevant:

def migrations
  down? ? @migrations.reverse : @migrations.sort_by(&:version)
end

So, if we are migration ‘up’ (which we are, since we’re running db:migrate and not db:rollback), migrations are ordered by their version (e.g. in ascending order). Any migrations that have already been run are skipped - migrations have been run if they exist in the schema_migrations table. start is determined by the current migration version (again, this is the ‘maximum’ value number fom the schema_migrations table), or zero if there are no version numbers in the table (this is the case, for example, with a just-created database). finish is determined by either the index of the migration file if a target version has been provided (which is not the case, if rake db:migrate is being run without any environment variable arguments), or the last index of the migrations list (e.g. size - 1).

execute_migration_in_transaction is the final step in the Migrator before the actual migration is ‘called’ (we’ll get to what this means in a second):

def execute_migration_in_transaction(migration)
  return if down? && !migrated.include?(migration.version.to_i)
  return if up?   &&  migrated.include?(migration.version.to_i)

  Base.logger.info "Migrating to #{migration.name} (#{migration.version})" if Base.logger

  ddl_transaction(migration) do
    migration.migrate(@direction)
    record_version_state_after_migrating(migration.version)
  end
rescue => e
  msg = +"An error has occurred, "
  msg << "this and " if use_transaction?(migration)
  msg << "all later migrations canceled:\n\n#{e}"
  raise StandardError, msg, e.backtrace
end

The first two lines guard against the migration already being run. Next, the migrator opens a database transaction (ddl_transaction calls use_transaction?, which wraps the block in a transaction unless the migration itself has signalled that it should not be run in a transaction (which is necessary for some specific database operations, and can be specified in a migration using the disable_ddl_transaction! method, or if transactions are disabled or unavailable for the database connection).

After the transaction is begun, the migrate method is called on the migration, then record_version_state_after_migrating is called, which updates the schema_migrations table. If anything goes wrong during the migration, a descriptive error is raised. You might recognise this error message from your own migrations: “An error has occurred, this and all later migrations canceled”.

Your migration! (ActiveRecord::Migration)

At this point, the migrator has figured out which migrations need to be run, and is running them one by one in a transaction. It’s now up to your migration. Let’s say the migration you’ve written performs the following action:

class CreateBlogs < ActiveRecord::Migration[6.1]
  def change
    create_table :blogs do |t|
      t.string :name, unique: true
      t.timestamps
    end
  end
end

That square-bracket syntax is a bit unusual. It’s the way that Rails has chosen to be able to support older migrations written for previous versions of Rails. The ActiveRecord::Migration class implements the self.[] method, which looks up the correct concrete migration class to use from the ActiveRecord::Migration::Compatibility class. This class contains a number of subclasses named after Rails major/minor versions, where each subclass overrides the current (e.g. Rails 6.1) migration methods to apply the previous versions’ default options. In this way, migrations back to Rails 4.2 can be supported, even though ActiveRecord 6.1 is installed. Neat!

We saw that the ActiveRecord::Migration::Migrator calls the migrate method on the migration class. What does this look like?

Turns out it’s pretty simple:

def migrate(direction)
  new.migrate direction
end

— rails/activerecord/lib/active_record/migration.rb

OK, so this creates a new instance of the migration class, and then calls migrate on that. Here’s the instance method:

# Execute this migration in the named direction
def migrate(direction)
  return unless respond_to?(direction)

  case direction
  when :up   then announce "migrating"
  when :down then announce "reverting"
  end

  time = nil
  ActiveRecord::Base.connection_pool.with_connection do |conn|
    time = Benchmark.measure do
      exec_migration(conn, direction)
    end
  end

  case direction
  when :up   then announce "migrated (%.4fs)" % time.real; write
  when :down then announce "reverted (%.4fs)" % time.real; write
  end
end

So, if the direction is :up (which it is for our purposes, since we’re running bundle exec rake db:migrate), it announces that the migration is beginning, then calls exec_migration (the next step!) using a connection checked out from the connection pool and wrapped in Benchmark.measure to measure the amount of time taken to run the block (i.e. the actual migration operations). Once that’s done, the connection pool block completes, returning the connection to the pool, then the total time taken to run the migration is announced. With these announce methods, the migration will run, and the migration output you are probably used to will be shown:

== 20210404082749 CreateBlogs: migrating ======================================
-- create_table(:blogs)
   -> 0.0091s
== 20210404082749 CreateBlogs: migrated (0.0092s) =============================

So, migrate instruments the actual migration operation, and provides some informative messaging output. What about the actual migration? For that, we need to jump into one more method - exec_migration:

def exec_migration(conn, direction)
  @connection = conn
  if respond_to?(:change)
    if direction == :down
      revert { change }
    else
      change
    end
  else
    public_send(direction)
  end
ensure
  @connection = nil
end

Some of these methods are maybe starting to look familiar, and may even match what is in your migration! If the migration has a change method, and the direction is not :down, then the change method of our migration is called, using the passed-in connection from the connection pool. If the migration does not implement change (it is optional after all, but if you don’t implement change, then you should ideally implement both up and down to make sure your migration can run in both directions), then it calls either the up or down method, depending on the direction - this will be the up method in our case, since we’re migrating and not rolling back.

Now, whatever is in our own migration’s change method will be run, altering the database!

Rolling back a change method wraps the change method in a block passed to revert. This method is called on the database connection, which reverses any operations called on the connection within the block - create_table becomes drop_table, add_column becomes remove_column, etc.

Updating db/schema.rb

If you’ve run a migration before, you’ll probably have noticed that running a migration (or rolling back a migration) tends to also update db/schema.rb. Where does this change come from? To answer this question, we need to go all the way back to the db:migrate Rake task definition, and take a closer look at the last line:

desc "Migrate the database (options: VERSION=x, VERBOSE=false, SCOPE=blog)."
task migrate: :load_config do
  original_db_config = ActiveRecord::Base.connection_db_config
  ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env).each do |db_config|
    ActiveRecord::Base.establish_connection(db_config)
    ActiveRecord::Tasks::DatabaseTasks.migrate
  end
  db_namespace["_dump"].invoke
ensure
  ActiveRecord::Base.establish_connection(original_db_config)
end

So, migrations are run, and then another Rake task is called - db_namespace["_dump"]:

task :_dump do
  if ActiveRecord::Base.dump_schema_after_migration
    db_namespace["schema:dump"].invoke
  end
  # Allow this task to be called as many times as required. An example is the
  # migrate:redo task, which calls other two internally that depend on this one.
  db_namespace["_dump"].reenable
end

This is just a proxy for rake db:schema:dump then - if dump_schema_after_migration is truthy, it calls the actual task, then re-enables itself to allow it to be called again (by default, Rake tasks can only be invoked once, hence why it must be re-enabled).

The rake db:schema:dump task looks very similar to the migration task:

task dump: :load_config do
  ActiveRecord::Base.configurations.configs_for(env_name: ActiveRecord::Tasks::DatabaseTasks.env).each do |db_config|
    ActiveRecord::Base.establish_connection(db_config)
    ActiveRecord::Tasks::DatabaseTasks.dump_schema(db_config)
  end

  db_namespace["schema:dump"].reenable
end

It looks up which connection configurations should be used for database tasks, establishes a connection, and calls dump_schema on DatabaseTasks.

Going all the way down the rabbithole of how the schema gets dumped is probably another step-through in it’s own right. Let’s stop here by noting that ActiveRecord::Tasks::DatabaseTasks.dump_schema uses ActiveRecord::SchemaDumper to produce a formatted Ruby file containing the necessary ActiveRecord methods to reconstruct the database.

This is the case if the schema format is set to :ruby, which is the default. The schema can also be stored as SQL, which allows the exact database schema to be dumped to a file named db/structure.sql, at the expense of database engine compatibility (e.g. a dumped PostgreSQL schema will only be restorable to databases using the PostgreSQL database engine, not MySQL, SQLite or others). When preparing structure.sql, a database engine-specific tool is used to dump the schema, rather than anything in ActiveRecord. For PostgreSQL, psql --no-owner --schema-only is used.

Once our migration is run, and schema.rb is updated with the database schema changes - that’s it! Our database is up to date, and ready for us to use the new or modified data structure our migration has handled.

Preview Devise mailers

Wed, 10 Mar 2021 00:00:00 +1300

I use ActionMailer::Previews whenever I am implementing a transactional email. They’re a super handy way of working with email development, and I’ve even enabled them in UAT environments before so clients and designers can check the email easily.

In many apps, nearly all the transactional email is actually sent by Devise, so I wanted to set up a ActionMailer preview for the emails Devise sent. Someone on StackOverflow have already put together the answer, but I made some small changes based on the specific application:

There are a couple of notes I have:

I had to comment out the confirmation_instructions, since I don’t have the :confirmable strategy active on this app. It fails with a non-specific error message about being unable to find the route (e.g. it gets all the way to rendering the email template before it fails).
There’s probably a way of introspecting the Devise strategies to figure out dynamically which preview methods to generate, if you wanted to make this reusable across different apps. You can see all the mailer methods available for Devise in IRB with Devise::Mailer.methods - ApplicationMailer.methods, or in pry with show-source Devise::Mailer.
You can have as many preview methods per mailer method as you want - e.g. if you had different content or links that were generated, you can have different preview methods that build different data (e.g. if an admin user had a different subject or email content, you could have one preview for the admin user email, and one for the normal user email).
This class looks up the first user and uses that. I usually use FactoryBot.build_stubbed here, because I want a record that looks like it is persisted, but I don’t actually need it in the database to render the email.
You could use Devise.friendly_token instead of “faketoken” if you wanted to have something that looked a bit more realistic.

Docker: You must use Bundler 2 or greater with this lockfile

Wed, 27 Jan 2021 00:00:00 +1300

Ever seen the error “You must use Bundler 2 or greater with this lockfile.” trying to build a Docker image from a Dockerfile that uses an older Ruby version? Even though you are gem installing the correct version of Bundler?

It turns out that the Ruby Docker images set an environment variable called BUNDLER_VERSION, which Bundler will always try to use to install dependencies when you run bundle install, even if a newer version has been installed. Even if you uninstall the old version, it will still not use the newer version. The solution is to prefix your RUN bundle install instruction with unset BUNDLER_VERSION. The full command that I am using to install the version of Bundler my Gemfile.lock was bundled with and then install my dependencies is:

RUN unset BUNDLER_VERSION &&\
    gem install bundler -v "$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n
    1)" &&\
    echo $(bundle --version) &&\
    bundle install

Note: I know that this affects the ruby:2.5 image (old, I know). While the problem might not cause an error in newer versions, the behaviour is the same

BUNDLER_VERSION is set, and the version specified by this environment variable will be used in preference to any other installed version. For this reason, it’s worth checking this, even on newer Ruby images, since dependencies may be installed with a different version of Bundler than you were expecting.

Follow-up: this works when building the image, but not when using bundle after that (it reverts to trying to use the old version). I tried a range of things to get this environment variable to persist, but export BUNDLER_VERSION doesn’t persist in the container, and putting the environment variable export in ~/.bashrc or /etc/profile only gets run when a shell is run - not when a standalone command (like bundle exec rails s --binding=0.0.0.0) is run. I ended up not being able to find a solution for this in the Dockerfile. Instead, I added a docker-entrypoint.sh script in the root of the project with the following contents:

#!/bin/bash
set -Eeuo pipefail
# Use the version of Bundler specified in Gemfile.lock, not the one packaged
# with the original Docker image
export BUNDLER_VERSION="$(grep -A 1 "BUNDLED WITH" Gemfile.lock | tail -n 1)"
exec $@

And declaring this entrypoint in my Dockerfile: ENTRYPOINT ["docker-entrypoint.sh"]

This means that when my command is run, it will be passed to the entrypoint - e.g. docker-entrypoint.sh bundle exec rails s --binding=0.0.0.0, and will have the BUNDLER_VERSION environment variable exportd from within that script.

Strip newlines from terminal output

Sun, 24 May 2020 00:00:00 +1200

Every so often I will need to take some kind of key file - like a PGP key, SSH, OpenSSL, that kind of thing, and be able to paste it into a one-line text entry somewhere. Maybe a .env file, or shell script, or some piece of infrastructure that just accepts a text field as the input (like CI configuration).

I’ve always had to google for how to remove newlines from such a file, and I’ve seen a range of tricks using sed, awk, all of that sort of thing.

I’ve just been able to find an easier way, which I’m documenting here, because I want to remember it for next time.

To remove newlines from a key-type file:

cat ~/.ssh/id_rsa.pub | tr -d '\n'

cat - stream the file into the pipe
tr - “ tr - translate or delete characters”
-d - delete the character, don’t try and replace it
'\n' - the character to delete (must be in single quotes to avoid your shell trying to interpret it).

Since I have pbcopy available on my laptop, I can easily copy this to my clipboard:

cat ~/.ssh/id_rsa.pub | tr -d '\n' | pbcopy

(I’m on Linux, but have an alias for pbcopy so it behaves the same way as it would on Mac OS).

Or base 64 encode it first:

cat ~/.ssh/id_rsa | base64 | tr -d '\n' | pbcopy

Inspect zip file contents

Wed, 13 May 2020 00:00:00 +1200

If you’ve got a large zipfile, or just a badly named one and you’d like to quickly take a peek at what’s inside it, there’s a handy flag that can be passed to the unzip utility (apt install unzip):

unzip -vl path_to_zip.zip

This yields an output like:

root@224cb547fae0:/usr/src/app# unzip -vl spec/fixtures/example.zip
Archive:  spec/fixtures/example.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      12  Defl:N       14 -17% 2019-01-08 03:33 d0985364  Example File.txt
       0  Stored        0   0% 2019-01-08 03:33 00000000  __MACOSX/
     570  Defl:N      417  27% 2019-01-08 03:33 c696190f  __MACOSX/._Example File.txt
--------          -------  ---                            -------
     582              431  26%                            3 files

This shows the file size, the compression algorithm, and the mtime, all without bothering with a UI or extracting to a tempdir. Handy!

See recipients of a GPG encrypted message

Mon, 27 Apr 2020 00:00:00 +1200

gpg --list-only $infile

e.g.

gpg --list-only test.txt.gpg

This command outputs recipient info, including recipient key ID and details if known:

With an empty keyring (no public keys imported):

gpg: encrypted with RSA key, ID 70F28839

With a populated keyring (public key of recipient imported):

gpg: encrypted with 4096-bit RSA key, ID 70F28839, created 2018-05-29
      "Josh McArthur <jo....@gmail.com>"

Cool utilities I've discovered this weekend

Mon, 13 Apr 2020 00:00:00 +1200

`bat`

bat is cat with syntax highlighting, but is still compatible with pipes, etc. Really neat. I like the filename header.

Github: https://github.com/sharkdp/bat

`jira`

A CLI for Jira. Maybe it’ll be faster than using the web interace???

Regardless, it supports a fancy config loading system that looks like I can configure it as part of my tmux setup to pop me into the correct project for the directory I’m in.

https://github.com/go-jira/jira

`entr`

A really nice tool in the UNIX philosphy of building simple, composible command line utilities. entr can be piped to to run a command when the stream on the left hand side changes. This is similar functionality to guard, livereload, etc, but is useful precisely because it is generic.

It also has a handy option for commands that don’t support signals or a modified stream &emdash; -r will cause entr to terminate the process and start it again, rather than trying to ‘hot reload’.

The website has heaps of examples with documentation on the additional options: http://eradman.com/entrproject/

TLDR pages

TLDR pages aim to extend on man pages with practical examples of common tasks. The website, https://tldr.sh/ is usable as-is, but there are a bunch of command-line tools available. I’m trying to Node.js version, but there are various options depending on the runtimes you have installed.

https://tldr.sh/

UNIX find case insensitive

Tue, 29 Oct 2019 00:00:00 +1300

I use the find all the time to (unsurprisingly), find files or directories I’m looking for. I prefer it to any UI built into my OS (currently Ubuntu, but all OSes seem to have this kind of thing - Alfred/Spotlight/Start Menu etc), because I usually know roughly where I’m looking, and I can be declarative about what I’m looking for (e.g. file that starts with x and has the extension .blah).

Today I learned that the find option I use all the time, -name, has a case-insensitive version. I can’t believe I didn’t try and guess this earlier, but that option would be -iname. This option has identical behaviour to the standard, but doesn’t match on case. This means that wildcards like * can still be used, but the terms included in the search string can be in any case - upper, lower, or mixed.

I came across this because I was attempting to export my highlights from my Kindle. After some research, I discovered that highlights are stored on a Kindle in documents/my-clippings.txt. Great! Except..I have many, many files in that folder. I first used find . -type f -name '*clippings*' to find it…that didnt’ match anything. I was then able to use -iname '*clipping*' where I did find it - named “My Clippings.txt”.

Because I used the wildcard in my term, I also found an additional file I can now looking into - “My Clippings.sdr”.

My favourite HTTP client

Thu, 17 Oct 2019 00:00:00 +1300

It’s 2019. I’m writing code that needs to communicate with other APIs all the time. The code I post below reflects a snapshot of my current template API client. If I need to commuicate with a third-party in Ruby, I’m using this class.

require "ostruct"

class MyApiClient
  class Error < StandardError; end

  def initialize(configuration=nil, stubs=nil)
    @configuration = configuration || Rails.application.config_for(:my_api)
    @connection = Faraday.new(url: @configuration[:host], headers: headers) do |conn|
      conn.request  :json
      conn.response :json, parser_options: { object_class: OpenStruct }
      conn.response :raise_error
      conn.response :logger, Rails.logger, bodies: true if @configuration[:debug]

      stubs ? conn.adapter(:test, stubs) : conn.adapter(Faraday.default_adapter)
    end
  end

  %i[get post patch put delete].each do |verb|
    define_method(:"raw_#{verb}") do |*args|
      @connection.public_send(verb, *args)
    rescue Faraday::Error
      raise Error
    end

    define_method(verb) do |*args|
      public_send("raw_#{verb}", *args).body
    end
  end

  def headers
    { user_agent: user_agent } # ... other default headers here
  end

  def user_agent
    @configuration.fetch(:user_agent, "My Application #{Rails.application.config.version}")
  end
end

Let’s deconstruct this.

We require “ostruct” to get the OpenStruct class. OpenStruct is a handy little class in the Ruby standard library that accepts a hash, and allows attributes to be accessed either via hash keys (my_object[:attribute]), or via method calls (my_object.attribute).
Make the class MyApiClient. Note we’re not subclassing anything here. This class uses a bit of stuff from Rails, but it’s all optional.
Define our own error class, a subclass of StandardError. We’ll use this class further down to wrap all exceptions coming from our HTTP library. Wrapping exceptions is beneficial since it allows consumers of this API client to not need to know too much about how the request is being made - just that something went wrong. If something goes wrong here, a MyApiClient::Error will be raised. The consumer can handle this with an error message or retry and/or inspect the cause attribute of the error to access the specific error from the library if more context is required.
Start our initialize method. This method accepts two arguments - configuration and stubs. We’ll get into what these do in the next couple of points.
Set up @configuration. This defaults to whatever is passed in - the API client just expects something that has hash-style accessors - so a Hash would do the job here, but also an ActiveRecord model, an OpenStruct, or anything else that implements [](key). If nothing is passed in, we fall back to getting Rails to fetch our configuration using config_for. I have blogged before about config_for, so won’t go into detail - basically Rails looks for a YAML file in config/YOUR KEY.yml - so config/my_api.yml in this case, parses it to a Hash, and then grabs whatever values are under the key named by Rails.environment - so development, test, production etc. It’ll also run this file through ERB when it reads it, so you can reference environment variables or any other config service using <%= %> tags.
Set up the base HTTP connection. This is an HTTP API client, not an HTTP API library, so we want to lean on others’ hard work here. You can actually use any HTTP library you’d like here - as you’ll see further down, all that our class really needs of @connection are methods representing the HTTP verbs - get, post, etc. Some libraries even just have a request(verb, ...args) method you could use instead. It really doesn’t matter too much as long as your helper methods we’re about to define know what to expect back from whatever library you’re using. In this case, I’m using the Faraday gem, with middleware. The set up I have with Faraday is useful, but not super specific to this client class, so I’ve talked more about it in the Faraday section below.
Next we define some methods - two methods for each HTTP verb. The first method is named raw_#{verb} - raw_get, raw_post, and so on. The purpose of this method is to take arguments for a request that the HTTP library is expecting, and make a request. It should return the raw response object that the library returns. In this method, we rescue errors bubbling up from our HTTP library - in this case, Faraday::Error, and re-raise our own error. Calling raise inside a rescue block with a new exception class like this will automatically assign the original error to the cause attribute of the re-raised error. The second method we define is just named after the verb - get, post, etc. This method is intended as a friendlier version of the raw_request method, and the idea behind this one is to provide a shortcut to just getting the response body data - 99% of the time, this is what you want, and so long as your HTTP library can raise errors when it runs into bad HTTP status codes (e.g. 400..600), you don’t need to worry about checking the response status - just handling any errors. For the HTTP library used here, faraday, we have the raise_error middleware making sure that errors are raised when something goes wrong, and the json response parser that will turn our response body into an object.
We’re nearly done - just a couple of configuration methods to go. headers should return the default headers to be applied to all requests. In this case, we just add a User-Agent header. It’s courtesy when consuming an API to make sure that your requests are identifiable, and the User-Agent header is perfect for this. You can put any other headers you’d like here in, such as Authorization, X-Api_Key, etc.
The user_agent method just builds a UA string for us to use - here, we’re using the common name of our application, and the version of our application. This will return something like “My Application abc123”, which allows both the name and release of our application to be identified, and if necessary filtered or rate limited. Without adding a user-agent, the HTTP library will usually use it’s own name as the user agent string, which means that all of your requests will be indistingushable from all the other “faraday”, “HTTParty”, “curl”, “wget”, etc. requests that others are making.

And that’s it! Testing is also pretty simple. I usually prefer integration testing something like this by mocking an HTTP request/response - usually the HTTP library will support something like this. You can also test your dynamic method definitions by calling them and asserting that the same method with expected args is called on the @connection, with the default headers mixed in.

Faraday

You’ll notice above that Faraday makes up a fair amount of the meat of functionality of this class. I wanted to break it into it’s own section because, as I mentioned above, it doesn’t really matter what HTTP library is used, so long as it can be passed data to make a request with, and passes some kind of response object back. Examples of other HTTP libraries you might consider instead of Faraday are:

excon - a bit more bare metal, but a much smaller dependency - this would be suitable if you had a moderately complex HTTP endpoint to communicate with from a gem where you maybe didn’t want to have a dependency as large as Faraday.
HTTParty - quite a flexible library but sometimes a bit too abstract for my liking. It also has an annoying post-install message whenever bundle install is run - including if it’s depended on by another gem. The README for HTTParty has it’s own example of how to make a super-slim API client class, so if you’re looking for something specific to HTTParty, be sure to check that out.
Net::HTTP. Oh-so-tempting since it’s part of the Ruby standard library, but it really is a low-level API. Net::HTTP may be worth considering if you are the author of a gem and don’t want to add extra dependencies, but otherwise it’s probably best to use a library to avoid code that is perhaps more verbose than it needs to be.

Faraday is modelled on rack, which is the de-facto interface between HTTP requests/responses, and your Ruby server. You could think of Faraday as the ‘frontend’ version of Rack.

It has a similar system of almost immediately bundling the request/response into an ‘env’ (environment) object, and then fulfilling the request and transforming the response by applying a middleware pipeline to it.

The actual out-of-the-box behaviour of Faraday is quite capable of making an HTTP or HTTPS request, complete with params, headers, and all of the other stuff that makes up the core of HTTP. To really unlock some neat behaviour though, it’s worth checking out the middleware that can be applied to your Faraday connection object. Middleware have been split out of the main Faraday project, so you only need to have that extra gem dependency if you need it. The gem is called faraday_middleware.

The middleware I use in my API client is relatively small. I’ll step through each one and describe what I use it for/what it does:

conn.request :json - tells FaradayMiddleware::EncodeJson to automatically transform any params or request body I pass in to JSON. This means that I can pass in a whole big hash (or in fact anything that responds to to_json), and Faraday will automatically transform it into JSON before sending the request.
conn.response :json, parser_options: { object_class: OpenStruct } - tells Faraday to automatically transform the response body from a string, back into JSON (obviously this requires the response actually be valid JSON!). I am passing some special parser_options here to tell JSON to decode using OpenStruct as the object class. Normally, JSON.parse will return a Hash, which is fine, but means that all the attributes need to be accessed using [] with String keys. Using OpenStruct as the object class means that (as I mentioned above), attributes can be accessed using hash-key lookup syntax, or method syntax. I’ve also written a blog post that describes this technique with a few examples at https://www.joshmcarthur.com/til/2018/12/03/ruby-deserialize-json-to-an-openstruct.html.
conn.response :raise_error - this middleware is actually part of the main Faraday project, so you don’t need the middleware gem for this one. It inspects the HTTP status code returned in the request, and will raise a variant of Faraday::Error if the request did not succeed. These variants can be specific for common statuses, like Faraday::BadRequestError or Faraday::ResourceNotFound, or a bit more generic, like Faraday::ClientError and Faraday::ServerError. The rresponse status, headers, and body are attached to the error for later inspection. In the API client, we’re using this middleware to make sure that Faraday::Errors are raised when a HTTP request fails - we’re then rescuing the error, wrapping it in our own error class, and re-raising it.
conn.response :logger, Rails.logger, bodies: true if @configuration[:debug]. By default, Faraday won’t really log much that is useful. This is by design, since logging a full request/response takes up a number of log lines. This middleware is conditionally added to the connection if the configuration we pass in (which, remember, can either come from a hash passed to the API client, or by included in our config/api_client.yml as debug: true) includes a :debug key that is truthy. If debug mode is set, we direct the Faraday logging middleware at our Rails.logger (you could direct it to it’s own log file or any other Logger if you wanted to, but Rails.logger means that all our app logs go into a single stream), and also tell it to log the request body with the bodies: true option. Without this option, Faraday will only log the request URL and some minimal response info, which isn’t as useful for debugging as seeing the full request/response.
conn.adapter - set either to :test, or Faraday.default_adapter. This setting is conditional on whether stubs have been passed in to the API client. If they have, we assume that we’re testing the API, so we tell Faraday to use a fake adapter named ‘test’. This adapter will look up a request in the stubs object when the client is called, and if a stub exists that matches the request (matching on path and/or params and/or headers), it will return the stubbed response. If stubs have not been passed in, we’re operating in ‘real’ mode, and set the adapter to Faraday.default_adapter. This defaults to net/http, but there’s all sorts of adapters you can use instead.

For more information on testing and using adapters with Faraday, you’ll find both the testing guide and the adapters guide useful.

That’s it. Hopefully this has been a useful and interesting deep dive into a nice understandable and configurable HTTP API client. If you’ve spotted any mistakes or points that need clarifying, please feel free to contribute a patch to my website repo!

See full Rails application logs on Heroku

Tue, 15 Oct 2019 00:00:00 +1300

I recently had an issue on an application that was deployed to Heroku. I could see from the frontend that I was getting a 500 status code in response to my request, but when I took a look at the Heroku logs (using heroku logs), I saw a single line containing my request, but no other details:

2019-10-15T02:47:56.555551+00:00 heroku[router]: at=info method=POST path="/widgets"
host=www.widgets.example.com request_id=c4819835-c2e6-480b-8bd3-4d70725f8bbf fwd="43.255.160.224"
dyno=web.1 connect=0ms service=45ms status=500 bytes=1891 protocol=https

This is interesting, because Heroku definitely used to log whatever your Rails app spat out to STDOUT, but it seems that by default they now suppress some messages - even if your application is configured with a verbose log level, like my application was set to: config.log_level = :debug.

The solution to this problem came from this StackOverflow answer - Heroku has a separate LOG_LEVEL environment variable that can be set to control the verbosity of your process logs. A simple heroku config:set LOG_LEVEL=debug, and I was seeing the stacktrace in my log output:

2019-10-15T02:54:43.377469+00:00 app[web.1]: I, [2019-10-15T02:47:56.520678 #4]  INFO -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf] Started POST "/widgets" for 43.255.160.224 at 2019-10-15 02:47:56 +0000
2019-10-15T02:54:43.377471+00:00 app[web.1]: I, [2019-10-15T02:47:56.521691 #4]  INFO -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf] Processing by WidgetsController#create as JS
2019-10-15T02:54:43.377473+00:00 app[web.1]: I, [2019-10-15T02:47:56.521749 #4]  INFO -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   Parameters: {"authenticity_token"=>"Psdasdas=", "widget"=>{"name"=>"Test Widget"}, "commit"=>"Save"}
2019-10-15T02:54:43.377475+00:00 app[web.1]: D, [2019-10-15T02:47:56.526155 #4] DEBUG -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   [1m[36mUser Load (1.2ms)[0m  [1m[34mSELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" ASC LIMIT $2[0m  [["id", 1], ["LIMIT", 1]]
2019-10-15T02:54:43.377478+00:00 app[web.1]: D, [2019-10-15T02:47:56.532972 #4] DEBUG -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   [1m[36mActiveStorage::Blob Load (1.3ms)[0m  [1m[34mSELECT "active_storage_blobs".* FROM "active_storage_blobs" WHERE "active_storage_blobs"."id" = $1 LIMIT $2[0m  [["id", 4], ["LIMIT", 1]]
2019-10-15T02:54:43.377480+00:00 app[web.1]: D, [2019-10-15T02:47:56.548897 #4] DEBUG -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   [1m[35m (1.3ms)[0m  [1m[35mBEGIN[0m
2019-10-15T02:54:43.377482+00:00 app[web.1]: D, [2019-10-15T02:47:56.551282 #4] DEBUG -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   [1m[Widget Create (2.1ms)[0m  [1m[32mINSERT INTO "widgets" ("name", "updated_at", "created_at", "user_id") VALUES ($1, $2, $3, $4) RETURNING "id"[0m  [["name", "Test Widget"], ["updated_at", "2019-10-15 02:47:56.546980"], ["created_at", "2019-10-15 02:47:56.546980"], ["user_id", 1]]
2019-10-15T02:54:43.377484+00:00 app[web.1]: D, [2019-10-15T02:47:56.552428 #4] DEBUG -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   [1m[35m (1.0ms)[0m  [1m[31mROLLBACK[0m
2019-10-15T02:54:43.377486+00:00 app[web.1]: I, [2019-10-15T02:47:56.552765 #4]  INFO -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf] Completed 500 Internal Server Error in 31ms (ActiveRecord: 11.7ms | Allocations: 5871)
2019-10-15T02:54:43.377489+00:00 app[web.1]: F, [2019-10-15T02:47:56.553564 #4] FATAL -- : [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   
2019-10-15T02:54:43.377491+00:00 app[web.1]: [c4819835-c2e6-480b-8bd3-4d70725f8bbf] ActiveRecord::RecordNotUnique (PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "widgets_pkey"
2019-10-15T02:54:43.377494+00:00 app[web.1]: DETAIL:  Key (id)=(17) already exists.
2019-10-15T02:54:43.377496+00:00 app[web.1]: ):
2019-10-15T02:54:43.377498+00:00 app[web.1]: [c4819835-c2e6-480b-8bd3-4d70725f8bbf]   
2019-10-15T02:54:43.377500+00:00 app[web.1]: [c4819835-c2e6-480b-8bd3-4d70725f8bbf] app/controllers/widgets_controller.rb:21:in `create'

Nice! Once you’ve got the error tracked down and resolved, you can either remove this config value entirely, or reduce it to a less verbose level, like warn.

Fancy Postgres indexes with ActiveRecord

Tue, 15 Oct 2019 00:00:00 +1300

Another day, another undocumented Rails feature!

This time, it’s that ActiveRecord::Base.connection.add_index supports an undocumented option to pass a string argument as the value for ‘column’.

This string is passed directly to the SQL statement, making it possible to use all sorts of fun things to lock down the constraint. It also means that you can make an index more declaratively if you’re not comfortable using the built-in options for key lengths or ordering.

Here’s a real-life example of the String invocation, where I am making a case-insensitive unique index on folder titles that ignores leading and trailing whitespace:

add_index :folders,
          "user_id, TRIM(BOTH FROM LOWER(title))",
          unique: true,
          name: :user_folders_title_unique_idx

I’m making a unique index on "folders"."title", scoped to "folders"."user_id". All standard so far, except for how the title is declared:

TRIM(BOTH FROM LOWER(title))

These are Postgres functions, and will act on the value that is passsed to it. In this case, TRIM will strip whitespace from the string passed to it. I’m adding BOTH so that both leading and trailing whitespace is trimmed. LOWER, unsurprisingly, converts the string passed to lowercase.

As far as I know, because this is passed directly to the CREATE INDEX call, any thing that could be passed to a raw SQL index creation call can be provided to this argument of add_index.

I have also verified that the index call is correctly serialized in schema.rb. In the case of the above index, it is recorded as:

create_table "folders", force: :cascade do |t|
  # ...
  t.index "user_id, btrim(lower((title)::text))", name: "user_folders_title_unique_idx", unique: true
end

Finally, something important to note: Rails can normally figure out what the index should be called by sticking the table name and column names you want in your index together with underscores. Not so in this case - because you are passing your own index statement, you must provide the name: option to add_index like I have above, so that the index can be identified.

Crossing technical and personal boundaries: New project nerves

Sat, 17 Aug 2019 00:00:00 +1200

A really interesting thought from Chris Toomey from Thoughtbot’s ‘The Bikeshed’ podcast, that really elegantly clarifies my own thinking on imposter syndrome, feelings of inadequacy, and the strive to deliver value to a broad range of contexts.

The work we do is purposefully somewhat challenging. We’re coming into organistions to try and help them, most often at sort of an inflection point - a point at which they’re struggling, either technically or process-wise. So there’s something we have to help with. It’s not always clear, or it may be that people are pointing at one thing but meaning something else and so there’s always some nerves.

Do I worry about not knowing the technical facets as much? The longer I do this, the longer I see that code is much the same across different frameworks and languages. I’ve got better at learning, I can do that more quickly, use existing knowledge. At the end of the day, the hard part, and the thing that we bring into the situation is not so much about the code, but the conversations - getting real pixels in front of users, and then mapping that back into the code. And THAT skillset is the same everywhere, and I know I have that foundation and I lean into that all the harder at the beginning of the project.

(Paraphrased and left out a couple of less-relevant sentences for brevity that related to pairing at Thoughtbot)

I really like the statements made here. I think I personally tend to concentrate too much on what is being brought to the table in technical skill, and not as much in the value that exists just from having an expert(s) guiding an organisation through something that they’re not necessarily well-equpped to handle on theier own.

Ubuntu laptop backup strategy

Mon, 05 Aug 2019 00:00:00 +1200

I recently upgraded my laptop from a 2013 Macbook Pro to a Lenovo Thinkpad T480. It’s great. I switched around my Ctrl key to the Windows key (where the Command key would be on a Mac keyboard), and everything has pretty much been perfect.

One thing that I did need to rethink was my backup strategy, which, on my Macbook had been ad-hoc TimeMachine. Coming from that, the world of backups on Linux was a little overwhelming. Many backup tools are designed for, or at least documented for, continuously-running server environments, not a laptop. Additionally, the backups will often be something like an automated rsync or scp, and not have a particularly nice ‘set and forget’ experience.

I did a bit of research, and found out that there probably wasn’t a UI tool for my shopping list:

Diff backups: I was coming from TimeMachine, so this was pretty important to me. I wanted a backup to just be a diff of the last backup, not a full copy every time. This cuts down on the time required for backing up, as well as upload size.
S3 backup: I don’t run cloud servers myself - it’s just another thing to keep locked down and secure. I already have some manual backups on S3, and I was keen to get my laptop backups headed there as well. S3 has a bunch of backup-relevant tools available, such as encryption-at-rest, versioning, access logging, and lifecycle rules. The lifecycle rules are particularly useful as this allows me to not have to manage much in the way of backup retention within the actual backup tool - I can just tell S3 to archive backups to Glacier after 6 months, and delete them a year after that (or whatever). Also, S3 storage at the scale of ~200GB is peanuts. Maybe a dollar.
GPG Encryption: The same as S3. I use GPG for a few things, and like it. I like how it’s tied to an identity, not a specific password. I suspect the ‘right way’ (if such a thing exists) is to generate a subkey tied to my main key and use this to encrypt my backups. I don’t really understand subkeys, so I just made a whole new key for backups. This isn’t perfect, but does mean that I can manage the key without worrying about what it’s doing to my main key.

Not much to ask for, I thought, but it proved kind of hard to find what I was looking for. In the end, I did find it though - and the joke was on me, since I had seen duplicity mentioned many times early in my research and had discarded it because “it didn’t have a UI” :-(.

duplicity is a really nice backup tool. This is largely because it’s not really a backup tool itself, more of a tool that sticks a bunch of related tools together into a nice smooth pipeline.

Without any extra installation, duplicity could do all of the things on my shopping list. I even got a neat bonus, where I could use the multi:// backend to backup to a local hard drive initially, and then to my primary S3 backup destination. This setup allows me to have a local, space constrained, not-very-reliable backup that is very fast to restore from - while still retaining my primary, reliable, cost-effective and lots-of-retention data store on S3. I was happy with this. It’s not quite a 3-2-1 backup strategy - I have 2 copies of my data, each on a different storage media, with 1 offsite. I could probably get 3 copies by archiving to Glacier earlier, but my lifecycle rules are currently set up to initially keep backups in S3 only.

So, duplicity could do all the things I wanted it to, but all via command line flags. Not very maintainable, I thought. A quick Google search, and I found that at least one other person thought the same!

backupninja is a wrapper for Duplicity (and a bunch of other backup tools, but I’m just using Duplicity). It supports defining a backup ‘pipeline’ in /etc/backup.d to run a series of backups, one after the other. It also supports scheduling backups and has a very usable curses-based interface via the ninjahelper utility.

I used the ninjahelper utility to create a Duplicity backup configuration file in /etc/backup.d/90.dup, and then edited it extensively to achieve the backup behaviour I wanted. I won’t post the full configuration file here, but here are some excerpts relating to the points I wanted to achieve above:

# Backup every day at 12pm. I am usually at work with an idle laptop around this time. 
when = everyday at 12

# These are extra options that I have experimented with:
#   --s3-use-multiprocessing will use S3 multipart uploads concurrently, one upload processs
#     per core. This speeds up the upload to S3.
#   --s3-use-server-side-encryption will tell S3 to encrypt the uploaded diff files using the S3
#     encryption key. I don't use my own KMS key because I mostly trust GPG.
#   --allow-source-mismatch had to be used initially because I had some partial backups. I probably
#     don't need it anymore.
#   --progress was supposed to tell me the progress of the upload when I manually run a backup. It
#     does do this, but the output isn't particularly useful.
#   --gpg-options needed to be specified, because backupninja runs as root to access all the things
#     it needs to backup. Without this, GPG was trying to use /root/.gnupg, and I didn't want to
#     have my keys spread across multiple users.
options = --s3-use-multiprocessing --s3-use-server-side-encryption --allow-source-mismatch
--progress --gpg-options "--homedir=~josh/.gnupg"

The GPG options are pretty simple. I just provided the encryption key and allowed this to be used for both encryption and signing. I had hoped to keep the key password out of the config file, but didn’t have much luck loading it from elsewhere.

[gpg]

# when set to yes, encryptkey variable must be set below; if you want to use
# two different keys for encryption and signing, you must also set the signkey
# variable below.
# default is no, for backwards compatibility with backupninja <= 0.5.
sign = yes

# ID of the GnuPG public key used for data encryption.
# if not set, symmetric encryption is used, and data signing is not possible.
encryptkey = [public key]

# ID of the GnuPG private key used for data signing.
# if not set, encryptkey will be used.
signkey =

## password used to unlock the encryption key
## NB: neither quote this, nor should it contain any quotes,
## an example setting would be:
## password = a_very_complicated_passphrase
password = [key]

## password used to unlock the signature key, used only if
## it differs from the encryption key
## NB: neither quote this, nor should it contain any quotes,
## an example setting would be:
## signpassword = a_very_complicated_passphrase
signpassword =

The multi destination stuff is pretty neat. The actual config file just had a provider defined, when the provider is actually just a path to a JSON file that defines the providers:

desturl = multi:///home/josh/.dotfiles/backup/providers.json

The JSON file contains details on how duplicity should connect to both backends. Once again, I tried to keep secrets out of this file, but didn’t have any luck providing them with any of the standard ways of configuring AWS access key IDs and secrets:

[
	{
		"description": "Local backup. Not to be used for reliable backups, just for restore speed",
		"url": "file:///media/josh/DROPTANK/duplicity/flaps?mode=mirror&onabort=continue"
	},
	{
		"description": "Primary datastore",
    "url": "s3+http://[bucket name]/duplicity/flaps?mode=mirror&onabort=fail",
    "env": [
      { "name": "AWS_ACCESS_KEY_ID", "value": "[AWS access key ID]" },
      { "name": "AWS_SECRET_ACCESS_KEY", "value": "[AWS secret access key]" }
    ]
	}
]

The query params mentioned here do have special meaning as per the duplicity man page:

mode - mirror or stripe. I want to mirror my backups - have a copy in each destination, not striped across multiple destinations.
onabort - this is set to ‘continue’ for the disk backup, because if it fails, it’s probably because the disk just isn’t plugged in (I’m probably not at my desk). For the S3 destination, this is set to abort, because I want to know if this backup fails.

…and that’s all the configuration. With this approach, I have automated daily diff backups to two different sources - one local, one remote, with two different storage medium. Technically once my lifecycle rules kick in, I’ll actually have three sources and mediums since Glacier backups are stored on a different media than normal S3 blobs (Bluray, I think). I use lifecycle rules on S3 to manage retention, which I can basically configure to be indefinite if I want to, and the pricing is super reasonable, since the diff backups don’t take up too much transfer bandwidth or storage space once the initial backup is complete. Both my local and remote backups are encrypted with a mechanism that I can manage myself - I’m not trusting Dropbox or Google Drive or any other cloud storage provider to keep my files secure - but, since I can, I’m taking advantage of S3 encryption anyway.

Hopefully this has been useful to somebody! I depended on a lot of StackOverflow answers, blog posts, man pages and other resources to put my configuration together - I certainly didn’t just figure it out - but I think that this backup strategy may work well for others as well.

I have a couple of ideas for next steps. Fairly imminent is a restore test, which I’ll probably aim to conduct on an EC2 instance for the first restore just to make sure that everything goes smoothly. If that goes well, I will snapshot my disk and try a restore onto my actual laptop.

I would also like to look into S3 region replication. This is something I normally have set up for website uploads in my deveopment job, but I need to price it out for backups and make sure that it won’t blow things out too badly. If I get replication set up, then along with backups stored in S3 and Glacier, I’ll also have an additional level of redundancy, with backups stored in a completely different region (probably Ireland or Frankfurt since I don’t like putting things is US regions and don’t quite trust London either anymore). Singapore is too close to my physical location to feel safe and my primary backups are already in Sydney.

Upgrading Rails apps with rake app:update

Thu, 25 Jul 2019 00:00:00 +1200

It’s been awhile since I’ve done a Rails update. After this long working with the framework, I’ve learned that it’s much, much easier to keep things up to date with tools like bundle-audit so that things never get so bad that I’ve got a whole major version (or more!) to upgrade.

Having said that, support legacy applications is just something that needs to be done sometimes, and today I learned a little trick that I’ve somehow missed in the last 10 years: rake app:update.

The Rails Guides actually have this task documented pretty clearly - section 1.4 in fact - so I’m not sure exactly how I missed this.

rake app:update isn’t magic, but it is helpful. It basically runs the same code path that rails new does when creating an application, but has the same conflict resolution process that normal rails generate processes do when it comes across two files where the newer version is not the same as the older version:

Overwrite /usr/src/app/bin/rails? (enter "h" for help) [Ynaqdhm] Y
       force  bin/rails

The options to resolve conflicts here are:

Y - yes, overwrite
n - no, keep our version
a - apply this action to all following files (e.g. when you get to bin/ you can generally overwrite all of those files).
q - Quit, stop here.
d - display a diff between the two versions.
h - display help, which is basically this list here.
m - open the conflict in a merge editor of your choice (e.g. VSCode, which can be configured for git to use for conflict resolution).

Out of all of these options, I find diff the most useful option, since it gives me an overview of the changes. I currently manually patch changes that I want to include into the file in another tmux pane, and then choose to not overwrite the file. My shell is not currently set up with a defined mergetool like VS Code, but I’m definitely going to give this a go next time, since it is basically a more streamlined process for what I am already doing.

One final note - rake app:update is NOT bulletproof. It’s not intended to be. It’s just a convenience. As with any change in your application’s code or dependencies, a comprehensive test suite is the most important feature. If you are going into any kind of change process, but especially a framework update without at least some key behaviour tests, stop and add them. It’s incredibly difficult otherwise to check after each change that your application remains functional.

Broadcast receiver pattern

Tue, 23 Jul 2019 00:00:00 +1200

I have recently had the chance to get stuck back into some native Android development. One of the things I really enjoy about working with the Android framework is that I get the chance to explore new patterns, and patterns that I have used before for Ruby, but not yet in Java.

One of these new patterns is the idea of a broadcast receiver. Broadcast receivers are a core part of the Android framework, however receivers are triggered outside the cotext of an application activity or fragment, so it can be a little unclear how to trigger something to happen in an activity (such as updating a UI component) when an intent is received.

The best example of this pattern is a code snippet. The following class is one I added to the NZSL Dictionary Android application. It registers and receives intents from the DownloadManager system component - specifically, when a download completes with a success or failure state (DownloadManager.ACTION_DOWNLOAD_COMPLETED), or when the notification that the system shows when the download is completed is clicked (DownloadManager.ACTION_NOTIFICATION_CLICKED).

public class DownloadReceiver {
    private DownloadCallback callback;

    private BroadcastReceiver receiver = new BroadcastReceiver() {
        @Override
        public void onReceive(Context context, Intent intent) {
            if (callback == null) return;
            String action = intent.getAction();

            switch (intent.getAction()) {
                case DownloadManager.ACTION_NOTIFICATION_CLICKED:
                     callback.onDownloadNotificationClicked();
                     break;
                case DownloadManager.ACTION_DOWNLOAD_COMPLETE:
                     callback.onDownloadCompleted();
                     break;
            }
        }
    };

    public void registerContext(Context context, DownloadCallback callback) {
        this.callback = callback;
        IntentFilter filter = new IntentFilter();
        filter.addAction(DownloadManager.ACTION_NOTIFICATION_CLICKED);
        filter.addAction(DownloadManager.ACTION_DOWNLOAD_COMPLETE);
        context.registerReceiver(receiver, filter);
    }

    public void unregisterContext(Context context) {
        callback = null;
        context.unregisterReceiver(receiver);
    }

    public interface DownloadCallback {
        void onDownloadCompleted();
        void onDownloadNotificationClicked();
    }
}

This class exposes two core methods - registerContext and unregisterContext. These methods are intended to be connected to the lifecycle methods of an activity or fragment - specifically, onPause and onResume.

registerContext accepts two arguments - the context to use, and the callback class implementing the DownloadCallback interface. These arguments can normally be set to the same value - the activity or fragment wanting to receive messages when a relevant download action occurs.

unregisterContext simply accepts a the single context to use to disable the broadcast receiver. No AndroidManifest.xml changes are necessary, since the receiver is connected and disconnected as registerContext and unregisterContext are called.

Within the broadcast receiver, it is also possible to do something else with the value, such as caching it in a class-private field so that registerContext can return a value immediately to the subscriber without waiting for the broadcast intent to be triggered. An example of when I have found this useful is in a NetworkManager, to detect whether there is an active internet connection or not.

The main reason I like this pattern is that it abstracts away just about any kind of behaviour that may be received from a broadcast intent. This can be more than just system broadcasts, as other apps can also emit intents that can be listened for and intercepted.

When an intent is received by a class following this pattern, it can be parsed, validated, and otherwise repackaged before being passed to the subscribing class via a consistent interface that is abstracted away from exactly how the data was received. Nice!

Speeding up TimeMachine backups

Thu, 20 Jun 2019 00:00:00 +1200

TimeMachine is an incredibly smooth backup solution for Mac OS, but it can be a little slow, especially for initial backups or incremental backups that have a large amount of changes (such as a VM image being added to your filesystem or the typical I/O killer, many small files).

I recently performed a TimeMachine backup to a brand new 1TB portable hard drive, and was alarmed at the planned time to complete - 19 hours for 130GB. Here are some tweaks I found to speed things up:

1. Prevent Mac from sleeping

Left to it’s own devices, Mac OS will typically sleep after a period of inactivity. This can place TimeMachine backups AND disks in a slower state. While the backup runs, it is worth adjusting Energy Saver settings to never sleep the computer or display:

There is also a handy app called Caffeine that allows toggling on and off of sleep.

2. Disable “Put hard disks to sleep when possible”

Not all disks support this, but keeping both the portable drive and system drive active is obviously beneficial to TimeMachine backup. This toggle can also be found in the Energy Saver preferences:

3. Ensure Spotlight is not indexing the TimeMachine drive

Spotlight is the indexing service built into Mac OS - by default, it crawls and indexes content on your computer to use when you hit the ‘Search’ icon in your menu bar (or hit Cmd-Space). The problem is that indexing is quite I/O intensive, so this can cause a real slowdown. On a newer disk, I believe Spotlight may also begin indexing the Timemachine backup itself before the backup is complete (there is a warning if you try and exclude the Timemachine backup disk once the initial backup is done, but I did not see a warning when I excluded the Timemachine disk before the initial backup).

To exclude your Timemachine disk from indexing, go into System Preferences -> Spotlight -> Privacy, click the Add (‘+’) button, and select your Timemachine disk from the sidebar in finder. You can also drag and drop the drive into this list.

Once you’ve made this change, you may want to restart your computer to make sure that both Spotlight and TimeMachine recognise this change.

4. Temporarily disable low priority service throttling

Mac OS has a concept of low-priority services. TimeMachine is lumped into this category, since in normal operation backups should just run in the background without taking up more resources than absolutely necessary. While you’re doing an initial backup though, you kind of want TimeMachine to have all the resources it needs to do it’s job a bit faster.

Low priority throttling can be disabled using a sysctl flag via a Terminal. To get to a Terminal, hit Cmd-Space, and type in ‘Terminal’ without quotes. The terminal will open - an empty screen with a prompt. The command to run here is sudo sysctl debug.lowpri_throttle_enabled=0. When you run this command, you may be presented with a disclaimer-like warning and you will be prompted for your computer password - type it in (note that the characters will not be shown), and hit enter to apply the setting.

In the same terminal window, you can now run sudo sysctl debug.lowpri_throttle_enabled to see the status of the throttling. It should say debug.lowpri_throttle_enabled: 0. If it does not, go back and try adding a -w to the command - sudo sysctl -w debug.lowpri_throttle_enabled=0, as some older versions of OS X seem to require this for the setting to stick.

HQ version

With this setting enabled, TimeMachine will no longer be throttled and will be free to use all the system resources it can. You’ll most likely now see TimeMachine appear in the list of apps ‘Using Significant Energy’ - this is great, as significant energy also equals significant system resources.

Also note that this setting only lasts until your computer is restarted. To clear the setting, you can either follow the instructions above, but set the setting to 1 instead of 0, or you can simply restart your computer.

With these changes, my initial backup of 130GB completed in 8 hours overnight, instead of the initial TimeMachine had planned, where after several hours it had backed up just 2GB. YMMV, but the tips above should all contribute to a faster backup process. Don’t forget to switch these settings back to their original values when your backup is complete, since the incremental TimeMachine backups are fast and don’t use much resource.

CURLing to a UNIX socket

Thu, 13 Jun 2019 00:00:00 +1200

In deployed environments, I often come across two Ruby application servers - Unicorn, and Puma. These servers are usually configured to listen on a UNIX socket file, usually located in the tmp/sockets directory of a Capistrano install, e.g. /home/deploy/appname/shared/tmp/sockets/appname.sock.

It’s quite often useful to be able to connect to a UNIX socket to see if the application server is behaving correctly, especially if the instance is not accessible via a public IP address. A little-known feature of the curl command line is that it accepts a --unix-socket argument to send requests to. This allows application servers and any other process that listens on a socket and supports HTTP to be tested easily with a command like:

curl --unix-socket=/home/deploy/appname/shared/tmp/sockets/appname.sock
http://localhost/my-application-route-goes-here

All the usual features of cURL are available when the UNIX file is provided, such as making POST/HEAD/OPTIONS/anything else requests, sending form data or files, and using cookies.

I’ve found that this technique is very handy for diagnosing problems that lie between the web/proxy server (usually Nginx), and the application itself.

tmux system clipboard copy & paste on Mac OS

Tue, 04 Jun 2019 00:00:00 +1200

I have recently begun using tmux as part of my everyday development process. It’s going great, but as somebody who uses a mixture of Mac OS applications like VS Code as well as terminal applications like git, vim, tail, cat, etc., the lack of system clipboard integration has really been hurting.

I went out and did some research into how I can set tmux up to copy to the system clipboard, and first found that support for piping copied content was added in Tmux 1.8 via the copy-pipe command, and then found a blog post by Thoughtbot (of course!), that outlined how to set up copy-pipe to use the Mac OS clipboard shell integration command, pbcopy.

Unfortunately, the Thoughtbot post was written in 2013, and since then, the syntax for defining keybindings has changed somewhat - hence this post.

This is the configuration that works for me on Tmux 2.8 that integrates Tmux copy operations with the Mac OS system keyboard:

In ~/.tmux.conf:

# Vim mode
setw -g mode-keys vi

# Setup 'v' to begin selection as in Vim
bind-key -T copy-mode-vi v send-keys -X begin-selection
bind-key -T copy-mode-vi y send-keys -X copy-pipe "reattach-to-user-namespace pbcopy"

# Update default binding of `Enter` to also use copy-pipe
unbind -T copy-mode-vi Enter
bind-key -T copy-mode-vi Enter send-keys -X copy-pipe "reattach-to-user-namespace pbcopy"

The key changes are:

-t vi-copy to -T copy-mode-vi
send-keys -X now needs to be added between the keyboard shortcut and the ‘command’.

I am still trying to figure out how to integrate this copy operation with mouse support, however I have verified that the vim keyboard shortcuts in tmux ‘visual mode’ now copy to the clipboard as I expect.

Pending block examples with RSpec

Mon, 27 May 2019 00:00:00 +1200

I have known about the different ways of marking tests as pending in RSpec for some time. When working on a feature for which there are already tests (e.g. cloning a similar feature where shared examples or a common starting point for tests can be used), it can be extremely convenient to temporarily mark as test as pending like so:

describe "addition" do
  it "adds 1 + 1" do
    pending("#45: Implement addition")
    expect(subject).to eq 2
  end
end

Something that isn’t covered in the documentation is how to do this for several tests at once. You can accomplish this by changing the it methods to xit, but this still requires each test to be updated.

While researching how to do this, I stumbled across an interesting issue comment in rspec-core, which mentioned that:

Currently pending ‘doc string’ is just an alias for it ‘doc string’, :pending => true.

This is interesting, because that means that pending is implemented as an RSpec tag, and I know that tags can be applied to more than just specific examples - they can also be applied to example containers (describe/context/feature etc)., and will merge their tags with each example.

Sure enough, adding a pending tag to an entire example block will cause all examples in that block to be marked as pending. “Pending” means that the example will still be run, but the failure of the example will not cause a test failure. What WILL cause a test failure is if the example runs and does not fail - this test fails because “pending” is meant to signal that the test SHOULD NOT pass

if it does, this is an indicator that the pending tag can be removed from that example.

If you would like to mark a block of tests as pending regardless of their underlying pass/fail status, the skip tag can be used instead. This will not even attempt to run the examples, resulting in a faster test run.

Here’s a fuller example:

describe "Arithmetic" do
  it "adds 1 + 1 and equals 3" do
    expect(1 + 1).to eq 3
  end

  it "adds 1 + 1 and equals 2" do
    expect(1 + 1).to eq 2
  end
end

These examples can be marked as ‘skipped’ by adding the skip tag to the describe block:

describe "Arithmetic", skip: true`

Results in:

Arithmetic
  adds 1 + 1 and equals 3 (PENDING: No reason given)
  adds 1 + 1 and equals 2 (PENDING: No reason given)

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Arithmetic adds 1 + 1 and equals 3
     # No reason given
     # ./rspec.rb:9

  2) Arithmetic adds 1 + 1 and equals 2
     # No reason given
     # ./rspec.rb:12


Finished in 0.00226 seconds (files took 0.44679 seconds to load)
2 examples, 0 failures, 2 pending

These examples can also be marked as ‘pending’ by adding the pending flag to the describe block:

describe "Arithmetic", pending: true

Results in:

Arithmetic
  adds 1 + 1 and equals 3 (PENDING: No reason given)
  adds 1 + 1 and equals 2 (FAILED - 1)

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Arithmetic adds 1 + 1 and equals 3
     # No reason given
     Failure/Error: expect(1 + 1).to eq 3

       expected: 3
            got: 2

       (compared using ==)
     # ./rspec.rb:10:in `block (2 levels) in <top (required)>'

Failures:

  1) Arithmetic adds 1 + 1 and equals 2 FIXED
     Expected pending 'No reason given' to fail. No error was raised.
     # ./rspec.rb:12

Finished in 0.03692 seconds (files took 0.32464 seconds to load)
2 examples, 1 failure, 1 pending

Failed examples:

rspec ./rspec.rb:12 # Arithmetic adds 1 + 1 and equals 2

Note the failed example, because Rspec in this case expected the test to fail, but it actually passed.

In this case, we can just mark the individual failing test as pending:

it "adds 1 + 1 and equals 3", pending: true

Results in:

Arithmetic
  adds 1 + 1 and equals 3 (PENDING: No reason given)
  adds 1 + 1 and equals 2

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) Arithmetic adds 1 + 1 and equals 3
     # No reason given
     Failure/Error: expect(1 + 1).to eq 3

       expected: 3
            got: 2

       (compared using ==)
     # ./rspec.rb:10:in `block (2 levels) in <top (required)>'

Finished in 0.0288 seconds (files took 0.40468 seconds to load)
2 examples, 0 failures, 1 pending

Finally, instead of passing true or false to skip or pending tags, a string explaining the reason for skipping or pending can be provided:

describe "Arithmetic", skip: "Blocked by ticket #45"

describe "Arithmetic", pending: "Blocked by ticket #45"

Results in:

Arithmetic
  adds 1 + 1 and equals 3 (PENDING: Blocked by ticket #45)
  adds 1 + 1 and equals 2 (PENDING: Blocked by ticket #45)

https://v.redd.it video URLs

Mon, 20 May 2019 00:00:00 +1200

A couple of years ago, Reddit added support for video and gif hosting - basically in response to competition from Imgur and Giphy taking visitors offsite.

This service works well, but they have been to some trouble to obfuscate things and make it a little hard to get to the raw video file - I guess to prevent hotlinking to try and direct visitors towards the actual post instead (with accompanying advertisements). Right-clicking to save a video from a post page won’t work either, since the video file isn’t referenced directly. Instead, they seem to be fetching the video and audio streams as a Blob, and using that as the video source. Sneaky!

If you are after that perfect cat video for your permanent archive though, you probably ARE after the video file. Getting it isn’t hard, once the technique is known.

TL;DR - this post details how to get each individual part of a video. If you just want the video, youtube-dl will do the job nicely.

The first thing to know is that the transcoding process that v.redd.it videos go through splits the video from the audio (if any). If a video does have audio, you will consistently find it at https://v.redd.it/{id}/audio. If a video does not have audio, you’ll get a 403 error page that looks suspiciously like that of an S3 bucket.

The second thing to know is the presence of a very handy file - the DASH playlist. This file can consistently be found at https://v.redd.it/{id}/DASHPlaylist.mpd. The playlist is an XML file listing all the ‘representations’ of a video. A representation is either an audio or video feed, and in this way can support different video resolutions and AV formats. As I mentioned above, Reddit-transcoded videos consistently have zero or one audio feeds in mp4a format, and a range of different resolution videos feeds - 240p, 360p, 720p, etc.

This XML file contains a number of RepresentationElement tags, which each of these containing a BaseURL. The content of the BaseURL tag can be appended to the https://v.redd.it/{id} URL to actually load the video stream, for example:

From https://v.redd.it/xcre0omf86j21/DASHPlaylist.mpd, we see that there are 5 video streams available, and 1 audio stream. By looking out at the BaseURL tag (or automating this with something like xmllint, we derive the following video URLs:

https://v.redd.it/xcre0omf86j21/DASH_720
https://v.redd.it/xcre0omf86j21/DASH_1080
https://v.redd.it/xcre0omf86j21/DASH_480
https://v.redd.it/xcre0omf86j21/DASH_360
https://v.redd.it/xcre0omf86j21/DASH_240

And the audio URL:

https://v.redd.it/xcre0omf86j21/audio

And that’s all there is to it! As I said at the start of this post, if you’re just after a video, youtube-dl is the better solution. It’ll do all this fetching and parsing for you, and will even select the highest-resolution video and re-mux the audio track back into the MP4 file.

Prevent unnecessary slowdown in IRB sessions with slow method calls

Tue, 14 May 2019 00:00:00 +1200

One of the convenient things that any interactive Ruby sesion (IRB) does for you is to inspect the return value of the statement it has just evaluated.

This is normally great, but if you have something that takes a LONG time to evaluate and you don’t actually care about the result, it can cause the session to block completely unnecessarily.

The best example of this is loading an ActiveRecord::Relation. If you’ve worked with ActiveRecord before, you’ll know that queries are lazily-executed - the actual SQL statement is not prepared and executed until a method is called that requires the results.

The gotcha here is that inspect-ing an ActiveRecord::Relation by default tries to echo out the models or other values you have requested - even if you yourself haven’t actually requested them. This can cause some pretty big delays in IRB or a Rails console if you have a large dataset your relation is querying.

To prevent this, if you need to grab a relation object (or anything else that takes a short amount of time to build, but a long amount of time to actually load), just assign the relation to a variable, and add on ; true to the end of your statement. When IRB evaluates this line, it will assign the relation object to the variable, but will not need to inspect it, because it’s not the final result of the statement - true is - and calling inspect on a boolean just stringifies it, which takes no time at all.

Here’s an example. Let’s say we have billions of Widget rows in our database. Fetching all of them takes a terrible worst case scenario of 2 minutes.

Why run this:

> Widget.where.not(id: nil)
=> [<Widget #1>, <Widget #2>...]

in 2 minutes, when you can run:

> relation = Widget.where.not(id: nil); true

in no time at all!

Trip Report: Days Bay Main Ridge, Wellington

Tue, 26 Mar 2019 00:00:00 +1300

Start time: 11:20am from Days Bay Pavillion

Finish time: ~ 2:30pm at Cheviot Bay Road + 15 minutes to cycle back to Days Bay

Time: approximately 3 hours with a few breathers and a 15 minute stop for lunch

An unexpected change in our weekend plans last weekend (mostly a sense of laziness that persisted all through Saturday) let us to try out a new area of the East Harbour Regional Park that we have not yet been into.

Between Point Arthur (where the Pencarrow Head trail starts), and the Wainuiomata Hill Road lookout runs the creatively named “Main Ridge”. This ridge sits between the East Harbour bays and Wainuiomata/Coast Road, and consists of a mixture of mature beech forest and regenerating manuka/native scrub.

Since this route is not a loop, we began by dropping a bike at Cheviot Road, inland from Point Howard. This would allow one of us to cycle to the start point at the end of our walk to collect the car. After dropping the bike, we continued around the bays to the Days Bay Pavillion, where we parked the car in the unrestricted car park spaces along the beachfront (the Pavillion carpark is also free to use, but has a 4 hour restriction).

The Kereru Track begins behind and to the right of the duck pond that is next to the Pavillion, following a small stream gently up the hill, with a residential street on the other side. The Korimako Track also leaves from Days Bay and can be joined to from the start of the Kereru Track, but is a rougher track and joins the ridge further along.

The gentle gradient of the track steepens after passing a small dam after around 10 minutes, before heading into more established beech forest and moving into some zig-zags up a spur.

A further 10-20 minutes clambering up zigzags and steps will have you reaching a memorial bench with a great view over the harbour. This is a great spot for a sit down and breather. Past the bench, continue up the spur, which after the initial climb moves into more typical top-of-spur climing - steep, short climbs with gradual climbing in between.

The spur meets the ridge after another 20 minutes or so after the bench, with a sign pointing the way to the right along the ridge towards Eastbourne. Head left, which is sign posted as 15 minutes to the mtop of the Korimako Track, and 45 minutes to the Ferry Road Track.

The ridge travel is typical for the area - generally pretty gentle with a short climb and descent every couple of hundred metres. I did manage to roll my ankle on one of these descents just for some variety, but it was minor and after stretching it out we carried on.

The Korimako and Ferry Road tracks are uneventful in their own right, but provide convenient milestones to measure progress against the map. We found that we were sticking to a pace that was around 2/3 of the signposted time, which is normal for us on Greater Wellington Regional Council tracks.

After the Ferry Road turnoff, the track turns to follow the ridge NNE with a couple more steep climbs to meet the main Lowry intersection. At 373m, Mt Lowry isn’t a huge hill to climb, however coming all the way up from sea level, it feels like quite an accomplishment. The trig itself isn’t at the intersection - it looks like it is around 10 minutes down the Rata Ridge Track. We didn’t bother finding the trig since we had already got to the viewpoint.

From here on in, the track is shared use for mountain bikers and walkers. Mountain bikers can come up the Rata Ridge Track, and head down the Main Ridge Track to meet up with the mountain bike park on the Wainui hill road. We saw several cyclists climbing their way up the hill, but no one coming down. I’m not sure if this is the predominant direction or not.

The main ridge track continues along towards Lowry Bay, now with the edge of Wainuiomata quite close on the right. There was a motorcross track which you might hear close by, however the track is smooth and well-formed with shade provided by beech trees and manuka. If you are heading towards Cheviot Road like we were, you continue along the ridge for what feels like ages, getting closer and closer to the lookout (the road noise can be heard with the hill road running downhill below the ridgeline). About 10 minutes before the lookout, a slightly wider firebreak is met, with tracks down to Point Howard and Cheviot Road dropping off to the left. Follow this track, crossing the stile, and take the extra 50 metres to go and check out the lookout point over the Hutt Valley.

The track from here drops quite steeply down into a stream valley, so back into Nikau-style bush and shade. The track is gravelled so is a good surface but might be a little slippery underfoot. The track descends to come close to a stream running down the valley and turns to the left to gradually descent the valley wall, meeting the stream 5 minutes before the road end.

If you’ve dropped a bike here, the cycle back around to Days Bay is just under 5 kilometers and is flat and smooth. There is shared footpath some of the way, but on the wrong side of the road (if you are heading towards Days Bay), so expect to do a bit of crossing from side to side.

Overall, this walk, like most in the East Harbour forest is nicer than might be expected considering its proximity to the industrial part of the Hutt Valley and Wellington City itself. It is easy to get to, being accessible by ferry from Wellington, bike, or car, and can fits nicely into a morning or afternoon.

View Larger Topographic Map

Entering a REPL from a PDB Python debugger

Fri, 22 Mar 2019 00:00:00 +1300

Something I have grown accustomed to when debugging Ruby code is the ability to jump into an irb session right from a breakpoint. With byebug this pretty much happens immediately. With pry, running irb will have the desired effect.

When I started getting to the point where I needed to debug Python code I was writing, I almost immediately had to find out how to do the same thing with pdb - I wasn’t getting anywhere with print() and the p, pp commands built into pdb.

It turns out that the Python debugging module (pdb) does have a built in command to drop to a repl with the context of the breakpoint - it’s just a slightly differerent terminology to Ruby.

To drop to a REPL from a PDB breakpoint, the interact command is what you’re after.

def hello_world():
  name = "you"
  pdb.set_trace()
  print(f'Hey {name}')

Mocking Python requests library

Wed, 20 Mar 2019 00:00:00 +1300

One of the toughest part of writing tests is how to go about handling dependencies on external dependencies. Aside from databases, probably the most common type of external dependency is a network request - usually HTTP (or HTTPS).

requests is a very, very popular library for performing HTTP requests in Python. It’s popular because of its easy to use API and simple approach to performing requests that can otherwise prove to be quite complex with lots of edge cases. The example from the homepage says it all really:

>>> r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

So, writing the code to perform an HTTP request is nice and simple. How about testing? Well, it really depends on your approach.

One option is to just let the requests happen - maybe to a testing or staging environment. This is obviously the easiest, but also requires such an environment to exist, to be accessible from anywhere, and requires the requests to consistently return the same results with no side effects.

Another option is to create a fake endpoint that you hit instead. In Ruby, I have done this with sinatra - and I guess in Python you would use something like flask. This can work well if the requests and responses are quite complex (maybe lots of specific params, headers and bodies), but also introduces a server taht needs to be maintained and kept up to date. There is a certain time penalty just to create such a server as well, especially how to fake side effects. It’s easier than it sounds to accidentally end up re-inventing the system that was being faked out in the first place!

Finally, there is mocking. Mocking involves intercepting calls at the code layer, rather than the network layer. This is useful because you don’t have the overhead of interacting with an external service, but still have full control over the result - for example, you can mock out your network request to see how your code reacts to all sorts of edge cases - network errors, partial responses, SSL errors - the list goes on. There is what seems to be a popular library available specifically for mocking calls to the requests library called, appropriately, responses. I would recommend this library if you’re after wide-ranged or complex HTTP mocking. The remainder of this post, I’m going to be talking about mocking HTTP requests using plain ol’ @unittest.mock methods that are built right into Python. This is really only suitable for simple requests, or a contained set of more complex requests, but is a lot easier to reason with and maintain, and saves you adding a dependency to your application.

unittest.mock allows you to replace objects and classes that your code under test relies on. You can assert that methods are called with arguments, add side effects to methods being called, and change the value that a method will return. If you go back and read about some of the different things that can go wrong with HTTP requests, you can maybe envisage how some of these mocking capabilities can be used to test how your code reacts to some of these scenarios.

The easiest way to mock out requests is to use the @patch decorator. This decorator accepts the module and method to mock, and will build the mock for you, passing it as an argument into your test.

Heres an example:

from unittest.mock import patch
from unittest import TestCase

class ApiClientTest(TestCase):

  @patch("requests.get")
  def test_standard_request(self, requests_get):
    # ... test goes here

With the mock available, you can change whatever you like:

You can change the status code return value: requests_get.status_code.return_value = 200
You can introduce a side effect: requests_get.side_effect = Exception("Request failure!")

As well as changing the behaviour of the mocked code, you can also add assertions that a method has been called the way you expect. Using the example posted at the top of this post, this assertion could be something like:

class ApiClientTest(TestCase):

  @patch("requests.get")
  def test_request(self, requests_get):
    requests_get.assert_called_with('https://api.github.com/user', auth=('user', 'pass'))
    
    # Call your code which performs a request here, e.g.:
    requests.get('https://api.github.com/user', auth=('user', 'pass'))

Quite often, after making a request, your code may have a dependency on the response. This isn’t a problem at all, but you might want to make a fully-fledged requests.Response class, like so:

from requests import Response
from requests.exceptions import HTTPError
from unittest.mocks import Mock

class ApiClientTest(TestCase):
 
  @patch("requests.get")
  def test_request(self, requests_get):
    successful_response = Mock(Response)
    successful_response.status_code = 406
    requests_get.return_value = successful_response

    self.assertEqual(requests.get("/").status_code, 406)

  @patch("requests.get")
  def test_failed_request(self, requests_get):
    failed_response = Mock(Response)
    failed_response.raise_for_status.side_effect = HTTPError()
    requests_get.return_value = failed_response

    self.assertRaises(HTTPError, requests.get, '/')

These are simplistic examples. Obviously with full acccess to the response, your options are fairly unlimited.

I have used this approach a number of times now and it has worked very well for me. Without adding complexity or an additional dependency, I am able to test both success and failure scenarios of making HTTP requests without adding too much magic or indirection. It’s quite easy to see how the requests are being mocked, and the return values and side effects that we can expect from these requests.

Aliasing Django model property

Wed, 20 Mar 2019 00:00:00 +1300

Not a super common occurrence, but exposing a Django model property under an alternative name is occasionally useful. I have used it before to assist in polymorphism, where several very similar models needed to have the same attribute.

Given a Django model like:

from django.db import models

class Widget(models.Model):
    email = models.CharField(max_length=254)

We can naievy try and alias the attribute by defining an accessor for it:

# ...

class Widget(models.Model):
    def username(self):
      return self.email

This appears to work - except when we go to use it, we find that Widget().username isn’t a property - it’s a callable method. This means that we can’t use it as an alias, since username would need to be accessed using username(), while email would be immediately accessible as email.

Fortunately, Python has a method decorator avilable called @property. This decorator is a way of signalling to the class that a particular method can be used to get or set an attribute on the class. We can use the @property decorator in our Widget class to alias our attribute nicely now:

from django.db import models

class Widget(models.Model):
    email = models.CharField(max_length=254)

    @property
    def username(self):
      return self.email

And we can access the email using either widget.email OR widget.username. Nice!

If you wish this alias to be bound in both directions (getting the attribute as well as setting the value), this is also possilble, you just need to define an additional method:

from djang.db import models

class Widget(models.Model):
  email = models.CharField(max_length=254)

  @property
  def username(self):
      return self.email

  @username.setter
  def set_username(self, username):
     self.email = username

create-react-app Gitlab Page template

Fri, 08 Mar 2019 00:00:00 +1300

I regularly complete little side projects in React - normally convenience shortcuts or data visualisation projects. My preferrred deployment platform for such static sites is Gitlab Pages, as it allows for flexible builds, and I can use custom SSL certificates so that I don’t need to terminate my SSL with Cloudflare.

Here is the template I use to build my React apps for Gitlab:

image: node:lts

pages:
  stage: deploy
  before_script:
  - npm install -g yarn
  - yarn install
  script:
  - yarn build
  - mv public public.old
  - mv build public
  artifacts:
    paths:
    - public
  only:
  - master

It:

Uses the latest LTS node image
Installs yarn, and then crete-react-app’s dependencies
Builds a production release of the create-react-app application (yarn build)
Copies the old public folder out of the way, and then copies the ‘build’ folder produced by the create-react-app build step to ‘public’. This is necessary because Gitlab pages will only serve files from the ‘public’ folder.
Restrict this deployment to run only on pushes to the master branch. This lets me work on features, merge requests, and bugfixes in the relevant branches before merging to master to release.

GWRC Matangi Train Numbering System

Thu, 28 Feb 2019 00:00:00 +1300

Note: This numbering system is known as “TMS” (even though TMS stands for “Traffic Management System”) of which the numbering system is just a small, bespoke part.

I travel on the Greater Wellington Metlink transit system nearly every weekday, and am often curious about what train I happen to be travelling on. I was recently checking out the Wikipedia page for the FP ‘Matangi’ units introduced in 2010, in particular how the numebering system worked. I had always just assumed that the numbers following the classification ‘FP’ and ‘FT’ were somehow sequential, so my curiousity was piqued to investigate the system further.

To break down the number of a particular unit, it needs to be divided into 4 parts:

1. Classification

This is a 2 or 3 character sequence. For the Matangi units, this is either ‘FT’ or ‘FP’ depending on whether the car is powered or not (only a single car in the set is powered - the other is a ‘trailer’).

2. Series

The first number in the sequence following the classification is used to identify the series - 1000 series, 2000 series and so on.

The initial series of the Matangi production units is 4000. A follow up 5000 series was introduced in 2015-2016, with a few minor improvements such as LED lighting (being retrofitted to the 4000 series).

3. Train Number

The series and following two numbers makes up the train number - this is what would be typically used to identity a single unit. According to the Series, the train number for Matangi units consistently starts with either ‘4’ or ‘5’ (which is how the two generations may be differentiated).

4. Check Digit

The final number is a check digit, used to verify the classification and train number. The check digit was certainly the hardest to research, being referred to in nearly all places as just that - “check digit”. I was fortunate to find the nzrailphotos.co.nz TMS Calculator Utility, which I could take a peek at the code for to see how it was calculated.

The check digit is calculated as follows:

Letters are converted to a numeric format. This means converting them to their character codes (According to ASCII, A-Z is 65-90), and then subtracting 64 to zero-index them. To ensure the placement is correct, they are added to the list of numbers to sum zero-padded - e.g. ‘8’ would be added as ‘0’, ‘8’, and ‘20’ would be added as ‘2’, ‘0’.
Numbers that already fall between 0 and 9 (48-57 in ASCII) have 48 subtracted to zero-index them, and are then added directly to the list to sum. Zero-padding isn’t a concern, since the number is always between 0 and 9.
The ‘sum’ of the numbers is now calculated. Each number in the list above is squared, and then multiplied by it’s position in the list (e.g. a list containing 0, 6, 0, 2 would work out being `(0 * 0 + 36 * 1 + 0 * 2 + 4 * 3) = 48.
The remainder of the sum divided by 11 is calculated, and is ready to use as the check digit.

The reason this calculation is so complex is because the check digit ensures that the numbers and classification is correct, and also in the correct order. A train number with a different classification or number will not have the same check digit, and a train number in an incorrect format (too long/short), will also not have the same check digit.

Since the explanation above is quite wordy, here are some examples:

FT4247. Train number is FT424, sum list is 0, 6, 2, 0, 0, 4, 2, 4, sum is 260, check digit is 7. Number verifies.
FP4247. Train number is FP424, sum list is `0, 6, 1, 6, 4, 2, 4, sum is 326, check digit is
1. Number verifies.
FP5148. Train number is FP514, sum list is 0, 6, 1, 6, 0, 5, 1, 4, sum is 389, check digit is 8. Number verifies.

Bash/Zsh shell expansion

Wed, 27 Feb 2019 00:00:00 +1300

Shell expansion is an incredibly handy shortcut. Previously to today, I’ve only ever used glob (*), which will expand to include all directories and files within the glob scope (e.g. * would list all files and directories in the current directory, test/* would list all files and directories in test/).

There are more though! Today I’m covering { range } and { list }, which are convient ways to expand a known range or list of values. In most cases, it can be used to avoid a for loop, the exact syntax for which I always have to look up.

Range expansion

Syntax: {start value...end value}

Example: {0..5}, {5..10}.

This expression expands numbers, and as far as I can tell, character ranges like {a..z}. This is great if you have directories that are incrementally numbered, and you want to target a subset of them. This isn’t a magical directive that gets passed to shell scripts - it gets expanded before that and passed to the script, which means that this expansion will work with any script that accepts multiple arguments - things like echo, cp, mv and friends come to mind, but many scripts accept a list of files to process somehow.

Negative numbers do not appear to be supported, so the minimum start value is ‘0’ - it can of course be a larger number than this. It’s worth pointing out that the range just passed each value to the command - it won’t skip values that don’t map to a file or directory for example.

Examples:

Make 10 new directories:

> mkdir {0..10}
> ls
0	1	10	2	3	4	5	6	7	8	9

Remove the first 5:

> rm {0..5}
> ls 
10	6	7	8	9

Add 5 more, but start from 50:

> mkdir {50..55}
> ls
10	50	51	52	53	54	55	6	7	8	9

List expansion

Syntax: {first value, second value, ..., last value}

Examples: {cats,dogs,birds,cows}, {1,2,3,5,8,13}

This expression expands to each value in the list. In other words, it’s not necessarily as handy as the range expression for passing directly to a command, BUT it can be used as part of a wider filename.

As an example, let’s say that we have a directory structure like:

   |-livestock
   |---cows
   |---deer
   |---sheep
   |-pets
   |---birds
   |---cats
   |---dogs

You can use list expansion as part of a file path, for example to list all the animals:

> ls {livestock,pets}

livestock:
cows	deer	sheep

pets:
birds	cats	dogs

To list all the animals whose name starts with ‘c’:

> ls {livestock,pets}/c*

livestock/cows:

pets/cats:

Paste mode in VIM

Sun, 24 Feb 2019 00:00:00 +1300

A VIM mode I did not know about for quite some time during my VIM usage is “paste” mode. This mode doesn’t do much, but is invaluable if you are moving code around, copy and pasting between files outside vim.

Paste mode can be activated using set :paste and deactivated with set :nopaste. This mode by default simply deactivates autoindentation, which means that the existing format of the text being pasted will be respected.

This is a very simple and handy trick to keep in mind, as it saves auto-indent picking up the wrong spacing and incorrectly applying “fixes” which often need to be manually fixed up.

Splitting large CSV files with awk

Tue, 19 Feb 2019 00:00:00 +1300

I recently learned of a useful technique when processing large text files that have a consistent separator using awk.

awk is a text processing programming language dating all the way back from 1977. While the format string passed to awk technically represents a full programming language, it is most typically used directly from the command line or from shell scripts.

The command to split a file is easily represented as:

awk -F\, '{print>$1}' input.csv

This command contains a field separator specification (in this example it is comma, escaped just in case - but it can be any character), and a very small Awk directive: {print>$1}. This directive takes the current line, and writes it to a file named after the first field. Unlike normal shell programming, > in awk will append to a file if it already exists, unlike the sh > operator, which typically overwrites the file. $1 simply represents the first column based on the field separator. If you need to reference any other column, you can simply use an incremented placeholder - $2, $3 and so on.

Relatively elegant ActiveRecord create callbacks

Wed, 13 Feb 2019 00:00:00 +1300

ActiveRecord callbacks are pretty popular in the Rails community, but they do have pitfalls. Probably the most significant one is that the responsibilities of a model can creep quite a lot once callbacks begin getting defined. Callbacks can also be hard to test, since they are triggered under a range of conditions, and have multiple phases per condition (before/around/after initialize, validate, create, etc).

There is an interesting, little-known ability to pass a block to the create method of an ActiveRecord model. This can be an elegant way of passing a callback method, from something like a service or form object to allow a model to set attributes on itself immediately before saving.

Here’s an example:

# app/services/ticket_creator.rb
class TicketCreator
  def create(summary:)
    Ticket.create!(summary: summary, &:assign_reference)
  end
end

In this example, our ticket creator service passes the attributes to create as usual, but also passes a proc reference to the assign_reference method of the model - in other words, this is a shorthand, which could also be written as:

Ticket.create(attributes) do |ticket|
  ticket.assign_reference
end

In this example, we would expect assign_reference to be implemented inside the Ticket model like so:

# app/models/ticket.rb
class Ticket
  def assign_reference
    TicketReference.new.generate!
  end
end

Personally, I find this approach to be a lot cleaner than a normal callback, since the invocation of the callback method isn’t linked directly into the model. It is invoked on the model, but not by the model. In this case, it is called by a service object, but could just as easily be invoked from a controller, mailer, or any other class really. It is also relatively easy to test, since we can test the result of the service method being called, rather than the core create method.

Javascript based file downloads in IE9

Wed, 13 Feb 2019 00:00:00 +1300

Getting feature parity on Internet Explorer is never very fun, or easy. I recently worked on a project where we needed to support inline downloads - specifically, a generated CSV string, as a file in all greenfields browsers as well as IE9-11.

Fortunately, IE 10 and 11 support the Blob API, which make downloads pretty simple using the ‘msSaveBlob` non-standard API.

IE9 has no such support though - either for Blob building, or for saving said blobs. Fortunately, there is a convenient workaround, using simple document body manipulation and the SaveAs execCommand that can be sent to a frame.

Here’s the snippet:

function legacyBrowserSave(content, filename) {
  const frame = document.createElement("iframe");
  document.body.appendChild(frame);

  frame.contentWindow.document.open("text/html", "replace");
  frame.contentWindow.document.write(content);
  frame.contentWindow.document.close();

  frame.contentWindow.focus();
  frame.contentWindow.document.execCommand("SaveAs", true, filename);
  document.body.removeChild(frame);
  return false;
}

This snippet should be wrapped in a browser or feature detection script, since saving a blob is supported in nearly every other browser.

Just to step through this solution within the function body, line by line:

On line 1, we create a new <iframe> element.

On line 2, we attach this new element to the document body. The iframe will not display as it has no width or height.

On line 4, we “open” the document. The second argument to this function, "replace", indicates that a new history item should not be added (this isn’t an actual navigation after all).

On line 5, we write the file content into the iframe, and then close the document on line 6. (This open, write, close operation is equivalent to doing document.write except it resets the existing content.)

On line 8, we focus the window within the frame. This is necessary for the next operation on line 9, when we pass an execCommand to the frame’s window, with the argument of “SaveAs”. (execCommand options are not standardized across browsers. In this case, SaveAs is specific to Internet Explorer). The true argument provided here is to indicate whether the default UI should be shown or not and has no effect - it’s just a required arg so that we can pass the filename as the third argument.

On line 10, we remove the frame from the document.body, since the user has now been presented with a dialog to save the file, and the frame is no longer required.

The return false is just to prevent the event from bubbling, since we have handled it.

Here’s a demo of this in action:

Source Gist

Trip Report: East Holdsworth Loop

Sun, 10 Feb 2019 00:00:00 +1300

This trip report covers a fantastic day loop that I found to be a little under-documented when researching - especially the East Holdsworth track, which has become one of my favourite Tararua tracks.

Started: 9:15am from Holdsworth Carpark

Finished: 4:50pm at Holdsworth Carpark

Times:

1hr up Atiwhakatu Track
2.75hrs up East Holdsworth Track
20min to Mount Holdsworth Summit
30min from Summit to Powell Hut
2hrs from Powell Hut to Holdsworth Carpark

We made a prompt, but not too early start from Wellington, heading over the Remutaka Hill Road and straight to Holdsworth - not even a coffee stop. We arrived at the carpark, which was significantly less busy than we had expected given the weather forecast, which was for “Fine, Light Winds” - the Tararua Unicorn weather.

We had organised our daypacks, lunches and snacks the night before, so we were able to get away about quarter past 9, stopping briefly at Holdsworth Lodge to fill in our intentions. We crossed the Atiwhakatu Stream and headed up towards the main intersection at the start of the Gentle Annie track, taking our usual 10-15 minutes along the flat, wide track to get warmed up. We carried on straight at the intersection, continuing through Donnelly Flat and up the Atiwhakatu Valley.

The Atiwhakatu Track is popular with dog walkers and mountain runners as well as trampers, and we met several people going both ways along this track. The track is well graded and maintained, and we were able to stick to our normal pace for this track to meet the swingbridge back over the Atiwhakatu Stream after one hour. A brief wobble over the bridge with the dog (who can’t walk across due to the mesh and needs to be carried), and we reached the intersection with the East Holdsworth Track 2 minutes after the swingbridge.

The East Holdsworth track is one we’ve passed several times on our way up or down the valley. We had been looking for a good weather window to do a loop up this track and back down Holdsworth for some time, as we were interested to see what kind of climb needs to be done to reach the Holdsworth-Jumbo ridgeline in just a few hours.

The East Holdsworth track [topomap] starts off surprisingly, suspiciously easy, travelling along flat terraces with short sharp climbs between each level for about 45 minutes. Towards the end of this time, there is less and less flat terrace, and more and more uphill, as the track steepens to climb up towards the bushline. This section takes some time - around 45 minutes to an hour, as it climbs most steeply between 650m and 1000m. After this, there is a slightly less steep section for around 30 minutes before reaching the bushline at 1200m.

We found that some careful monitoring of rest stops and breaks was needed - despite the relatively brief time period the main climb covers, hourly rest stops weren’t sufficient, and we made an early stop for lunch beneath the bushline at around 1100m. Next time we do this track, I would be inclined to break the uphill climb into shorter sections, as I think we probably allowed ourselves to use up more energy in the steepest part of the climb, which made things harder for us nearer to the bushline.

On a clear day, at the bushline you will be rewarded with sweeping views over the Wairarapa Plains, along with views looking North towards Jumbo and Mitre. We found the remaining climb up through the Tussock to the main Holdsworth-Jumbo ridgeline a lot easier, with a northwester coming down off the ridge to cool us off and a better and better view of Holdsworth and Jumbo the further up we got. In fact, with the wind being a little stronger than forecast, we stopped at a rocky platform around 1250m to put on thermal tops, just to keep the wind off.

From the bushline to the ridge took us around 30 minutes, with more of a terraced climb up to the Holdsworth-Jumbo circuit - gentle slopes followed by short/sharp climbs up the rocky tussock-covered ridge. There’s a DoC sign at the intersection showing the way to Holdsworth/Powell, Jumbo, and back the way you came. It’s worth keeping in mind the possibility of using this track as an emergency exit from the main circuit, as the sign at the intersection indicates that the return trip to the carpark takes just 2.5 hours (personally I would expect that to take longer to get down the steep track, but perhaps that time is intended to be a little lower than it actually is to persuade people who are on the fence to follow the quickest route off the exposed ridge).

Once the main track is reached, the going becomes easier - both due to being on top of the hill, and because there is less tussock cover at the top, with most of the walking being along rock and gravel. There are a couple of small knobs to sidle around along the ridge before the final climb up to the trig at the top of Holdsworth. We found we made good progress in this section, reaching the trig 20 minutes after leaving the East Holdsworth intersection.

We stopped for 10 minutes or so at the trig for a quick peek around with the binoculars, spotting the aircraft wreckage of an RNZAF De Havilland Devon that crashed in 1955, and was controversially stolen in 2009 and returned in 2010. This was definitely the clearest day we’ve had at the summit of Holdsworth, and we could see right out to Kapiti Island to the West, the Wairarapa Coastline to the East, and Hector, Maungahuka, Aokaparangi and Kahiwiroa along the main range beside us. Directly ahead was Jumbo, Angle Knob and Shingle Slip Knob, with McGregor and the Broken Axe Pinnacles in the background. Fantastic views.

From Holdsworth, we followed the ridgeline south towards the construction site at Powell Hut. The rebuild of this hut is expected to completed at the end of April 2019, and it looks like they are making good progress on the main building, with roofing going on and a larger deck under construction. While the construction is ongoing, there is NO shelter or access to the hut, and also NO water supply or toilets available to trampers.

There is a rough diversion “track” in place that bypasses the construction site. This “track” is temporary, and as such as fairly rooty and rough. There’s mud underfoot which makes for a bit of a slip and slide, with the diversion taking around 10 minutes and emerging about 10 meters below the altitude of the hut at the base of the final set of stairs. We heard a number of helicopters (or maybe the same helicopter multiple times?) through the day in the direction of Powell, so suspect they were also making the most of the nice weather.

From here, we were following the Gentle Annie track that we’ve done several times before, and by this point were mostly on autopilot and feeling pretty tired. We had a short break at Mountain House to remove a layer (thermals) that were no longer needed, and refill our waterbottles that we had been rationing since Mt Holdsworth. After Mountain House, it was a quick scramble back down the track, reaching the main intersection with the Atiwhakatu Track around 4:30pm, and arriving back at the car at 4:50pm.

Overall, this loop did take a little longer than expected, but for a day trip that covers the range of forest and terrain this particular loop does, I believe it’ unparalleled, in the Tararua Forest Park at least. A large contributor to the variety on display on this loop is the East Holdsworth Track. While we found it steep, it had a great variety of terrain, views and ecosystems on the climb from river level to the tops. We were even fortunate enough to see a Kaka flying overhead and hear another, most likely due to the extensive trapping program going on around Donnely Flat. This track has definitely gone onto our favourites list, and can only assume that we struggled to find detailed trip reports covering this track previously due to the authors’ wish to keep it quiet for themselves!

Internationalised ActionMailer Subjects

Mon, 04 Feb 2019 00:00:00 +1300

I’m a big fan of keeping strings out of my application code. It’s a pattern I first came across with Android string resources, and since then I’ve been using tools just like the i18n gem to abstract content away from code.

Something mentioned directly in the Rails I18n guides that I really like to see used myself is the built-in support for translating mailer method subjects. As the guides mention:

If you don’t pass a subject to the mail method, Action Mailer will try to find it in your translations. The performed lookup will use the pattern ..subject to construct the key.

— https://guides.rubyonrails.org/i18n.html#translations-for-action-mailer-e-mail-subjects

This is a very handy trick, since email subjects are frequently the last remaining piece of content that exists in mailers. Along with translated templates, making mailers that support multiple languages (or even just the default locale language), is easy, with all strings moved to I18n locale files and templates.

By default, ActionMailer will use the mailer name and the action name as the I18n key if mail is called without a subject option.

Here are some examples:

class UsersMailer
  def password_reset
    # Key is: users_mailer.password_reset.subject
    mail to: @user.email
  end
end

class Admin::AlertsMailer
  def alert
    # Key is admin/alerts_mailer.alert.subject
    mail to: @administrator.email
  end
end

Here would be the corresponding locale file for the above examples:

en:
  admin/alerts_mailer:
    alert:
      subject: "New Administrator Alert"
  users_mailer:
    password_reset:
      subject: "New Password Reset Request"

I feel that understanding I18n in Rails is a really undervalued skill. Frequently, understanding the abstractions that I18n offer can lead to clean separation of content and logic, as well as inspiring some abstractions in your own application. I look forward to continuing to learn more about what can be done within the I18n framework.

My understanding of CSS units

Sun, 03 Feb 2019 00:00:00 +1300

I was listening to Episode #107 of the Syntax Podcast this afternoon, which covered CSS units, and was inspired to actually stop and think about how I use units when writing CSS. I felt pretty reassured that generally my approach is in line with what Wes and Scott were discussing. Let’s go!

rem

I use rem by default for most things. Usually this is margin, padding, and font sizing. rem sizing works well for me because I like to establish a baseline sizing at the document root. Usually this is 8px, but occasionally I’ll work with a 4px or 16px baseline. Once I’ve set this global base size, I can use rem to work in multiples of the baseline, keeping everything relative. This makes it simple to adjust the sizing later, since everything is relative to a common base value.

There were a couple of things I learnt about rem from the podcast:

A typical approach to sizing is to set the baseline to a percentage of the normal browser default font size of 16px. The normal number to see floating around the internet for this is 62.5%, which sets the baseline size to 10px. The reason a percentage is used is that this will scale up and down appropriately if the user has selected a larger or smaller font size.
It can be useful to set the baseline size to a base 10 value - for example, 10px (or the percentage equivalent). The reason for this is is that it allows for easy conversion between common pixel values to rem. For example, a 32px margin could be expressed as 3.2rem, while a font size could be expressed as 1.6rem.

Pixels

I use pixels pretty much only for positioning (although an absolute positioning like this is always as a last resort). I also tend to use pixels for defining a base border radius since, unless corners are super rounded, the numbers involved tend to be small and difficult to reason with in rem. It was an interesting observation about how pixels used to be extremely widely used before they fell out of favour - for no particular reason. Pixels continue to be used a lot in graphic design, and for a lot of websites where relative baseline sizing is not required. For me, I find that using pixels for measurements, even with aids such as variables, leads to brittle and hard to work with styles.

Other units

Just like Scott and Wes in the Syntax podcast, I don’t generally find a need for the many other units available in CSS. I have always been curious to hear about use cases for slightly weird units like centimeters and inches, and was interested to find that these units are generally useful for print stylesheets - in particular specialist paper formats such as envelopes.

Overall, I found listening to this particular episode gave me a great opportunity to reflect on the opinions I hold regarding CSS units, and why I have decided on the approach I use.

Biking from Akatawara to Renata Hut (Attempt #2)

Sun, 03 Feb 2019 00:00:00 +1300

After my failed 2017 attempt to reach Renata Hut, I finally mustered up the spare time and inclination to have another shot.

The TL;DR of the last attempt is that due to significant tree fall on the track, it took me too long to make it up to the Hut - I reached my cutoff and turned around.

After that attempt, I reflected on how, because much of the track is four-wheel drive accesible, I could possibly save some time by mountain biking as far up as I could before continuing up by foot. That’s what I tried this time. It didn’t work out, which I’ll go into.

Start time: 9:30am from Akatawara Hill Rd saddle

Finish time: ~ 3:30pm

Time: 6 hours - approximately 3 hours up, 1.5 hours down, 1.5 hour delay + breaks.

I started from the saddle, which is just as dodgy looking as ever. Not-very-reassuringly, the skeleton of a burnt out car which wasn’t there last time was right in the middle of the gravel patch at the summit. I parked as out-of-sight as I could manage and got on my bike and set off up the hill. I pretty quickly discovered that while the track is definitely bikable in terms of width, the terrain and uphill slog was going to make things tough for me. This was 100% my bike fitness - I’m pretty sure someone more capable on a bike would have no particular problems.

I met a single tramper about 15 minutes up the hill, who was on his way out from Renata. It was good to check in with someone, though he was pretty happy to find how close he was to getting out. The track up is generally in good condition and has been cleared since I was last up. The 4WD track is definitely 4WD - not just a gravel road, in places there are deep water-filled ditches to traverse, and some impressively deep ruts.

I found that in general I made good progress up the hill, although I had to jump off and walk a lot more than I thought I would need to. This was mostly due to struggling to hold up enough momentum to pick a stable path up through ruts, and on balance I found it was a lot less tiring to walk a few meters and continue cycling rather than to battle up each tough bit. Another consideration if you’re looking at cycling is also that the gravel is not that of a maintained road. It’s fine most of the time, but there are frequently spots with larger lumps of rock - fist sized rocks took a lot more effort to push over and tended to roll around a lot more.

I was pleased to get to the large slip and washout where I turned around on my last attempt in pretty good time, and as I carried on to Maymorn Junction, I could see I definitely made the right call. While the slip isn’t THAT far from the Junction, it would have definitely pushed me off my timing to carry on, let alone to go all the way to the Hut. As you can see from the Topomap below, I was about as far from the Junction as the Junction was from the Hut.

View Larger Topographic Map

Anyway, I carried on. I was having no trouble with the track, which is 4WD track all the way to Maymore Junction and on down to Waiotaru Hut. I reached a clearing just in time for lunch, and stopped here with a great view of the main range.

I was expecting to get to the junction about now, but figured that it must be coming up soon with the road opening up. There was no signage, and I couldn’t see any triangles, so I carried on. A short distance down the road, I saw some orange triangles, but there were on the wrong side to be to Renata - I checked out the map, and noticed that there was a track coming in to Renata here. I started to feel like I may have passed the Junction, but my GPS wasn’t putting me exactly where I thought I was, and the topomap showed the tracks converging reasonably roughly, so I carried on. The road starting heading downhill, and I heard some 4WD trucks coming up behind me. This put a little pressure on since the road wasn’t so wide, and I wasn’t sure how fast they were coming up, so I carried on further down the hill.

By the time I figured out that I’d clearly missed the Junction, I was a decent distance down what I now knew was the track to Waiotaru Hut.

View Larger Topographic Map

I still wasn’t getting a good GPS fix, but determined where I was as I was beside a stream crossing. I stopped here and let the group of three 4WD vehicles catch up and pass me before turning around to head back up the hill. This took me around an hour to regain the junction. By this point, I was feeling pretty short of energy, and hadn’t expected to have to get back to MORE uphill cycling straight away, so I had to take it pretty slowly and walk a decent chunk of it. By the time I reached the clearing where I stopped for lunch, and having not passed any other likely looking spots I was more inclined to not carry on until I had confirmed where the track to Renata Hut actually was. By now, I was almost at my cutoff, both in terms of time and energy, and knew I wouldn’t be able to make the Hut that day. After a bit of a hunt around, I found the stump of where a DoC sign used to be. Just past that stump, I found an unmarked track heading up a small bank. At the top of that bank, I could see an orange triangle about 50m away - solving the problem of the missing track!

Maymorn Junction with the track to Renata on the left beside the post.

Next time, I definitely know where the track is, and I also know that I can do a mixture of cycling the easy bits and walking the hard bits to get up to the junction in good time and without using up all my energy. It’s definitely one that I will try again, and hopefully it will be third time lucky.

From the junction, I gratefully started downhill. The downhill trip was mostly quick, with some of the undulations along the ridge providing some gradual uphill stretches to overcome. The rutted parts of the track were a lot easier to roll down than struggle up, and even though I didn’t manage to make it down without getting wet feet in the ditch crossings, by that stage I needed a cool down.

Made it back to the car at around 3:30pm, and spent some time on the Summit stretching and resting up, completely exhausted. Definitely the toughest cycle I’ve done in a day, but I really enjoy the solitude and quietness of this part of the Tararua Forest Park.

List filenames only with grep

Sun, 09 Dec 2018 00:00:00 +1300

Just a quick one - I use grep all the time to find matches for text in files. Sometimes I’m not really interested in exactly what matched, I just need a list of files that matched SOMEWHERE. This morning I found that the -l option with this (or --files-with-matches).

From the man page:

 -l, --files-with-matches
          Suppress  normal  output;  instead  print the name of each input
          file from which output would normally have  been  printed.   The
          scanning  will  stop  on  the  first match.  (-l is specified by
          POSIX.)

Which turns the output from:

app/views/widgets/forms/_basics.html.haml:    = f.input :description, as: :text, input_html: { rows: 3, class: "xxlarge tinymce", id: "textarea-widget" }
app/views/widgets/_edit_form_description.html.erb:<%= f.input :description, input_html: { class: "tinymce xxlarge" } %>
app/views/widgets/_edit_form_description.html.erb:<%= f.input :blurb, input_html: { class: "tinymce xxlarge" } %>

Into:

app/views/widgets/forms/_basics.html.haml
app/views/widgets/_edit_form_description.html.erb

Handy!

Ruby: Deserialize JSON to an OpenStruct

Mon, 03 Dec 2018 00:00:00 +1300

Deserializing JSON from API responses, files and the like is a pretty common task. Depending on the complexity of the JSON structure, this can be a bit difficult to work with though, since the default action of JSON.parse is to deserialize to a string-key Hash. This means that after parsing the JSON, there’s a lot of fetching, diging, and the like to get things working, and the code doesn’t usually come out super readable.

An alternative to this is based on an argument that I stumbled across this afternoon - object_class. This option defaults to Hash, but can be set to anything you like. The best example of this is what happens when object_class is set to OpenStruct:

require 'open-uri'
require 'openstruct'
require 'json'

me = JSON.parse(open("https://api.github.com/users/joshmcarthur").read, object_class: OpenStruct)

me.login
=> "joshmcarthur"

me.name
=> "Josh McArthur"

It even works with nested objects!

repo_uri = "https://api.github.com/repos/joshmcarthur/joshmcarthur.github.com"

repo = JSON.parse(open(repo_uri).read, object_class: OpenStruct)

repo.owner.login
=> "joshmcarthur"


repo.full_name
=> "joshmcarthur/joshmcarthur.github.com"

While the examples above are using OpenStruct, it works equally well with a class of your own creation - the only requirement is that your initialize method accepts a Hash structure.

Exercism Strain: Collaborative implementation and benchmarking

Thu, 29 Nov 2018 00:00:00 +1300

This post covers an incremental implementation of the Exercism “strain” exercise that I worked on collaboratively with one of my colleagues at Ackama, Daniel Jauch.

The Strain exercise brief is to implement keep and discard functions within an Elixir module named Strain, without using the Enum.filter module, for example:

iex> Strain.keep([1, 2, 3], fn e -> e < 10 end)
[1, 2, 3]

iex> Strain.keep([1, 2, 3], fn e -> e < 3 end)
[1, 2]

iex> Strain.discard([1, 2, 3], fn e -> e < 3 end)
[3]

First implementation: Use other Enum methods, just not `filter`

Our first stab at this was to assume that since we couldn’t use Enum/filter/2, there were Enum methods that could be used to get an equivalent result. Here is what we came up with:

defmodule Strain do
  def keep_enum(list, fun) do
    list
    |> Enum.map(&(fun.(&1) && &1))
    |> Enum.reject(&(!&1))
  end
end

This worked, with all tests passing, so we submitted that to Exercism. Yay, done! But then we took a look at some of the community solutions, and realised that our initial foray into Enum was a little misguided, since there was an Elixir feature we completely didn’t know about we could use instead.

Second implementation: Comprehensions

Elixir Comprehensions are a bit of syntactic sugar over Enumerables that allow for some handy brevity when processing Enumerable data. Comprehensions support a generator, and a filter. The generator is the first “argument” to the comprehension and is required. The filter is an optional second argument, and can be used to exclude non-matching values from the list. Is all this starting to sound exactly like how keep and discard should work? We thought so as well!

defmodule Strain do
  def keep(list, fun), do: for x <- list, fun.(x), do: x
end

Much, much simpler, and even more readable in my opinion, once we understood what the two arguments passed to the comprehension did. The implementation of discard is exactly the same, just the inverse result from the check function - !fn.(x) instead of fun.(x).

So, as an Exercism exercise, we were happy with this - from a 5 line function down to a single line.

Then we started wondering - if Enum.filter is what we would normally use for this, and we have come up with two different implementations of the exercise - what is the speed difference between them? Only one way to find out!

Third implementation: Creating a mix project

To perform some benchmarking, we found a Hex package called benchee that looked like it would suit our needs nicely. Before we could add this dependency though, we needed to package our current code into a Mix project!

We started from the Exercism exercise folder:

ls *
strain.exs
strain_test.exs
README.md

We made a new Mix project in this folder:

mix new strain_bench
* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating config
* creating config/config.exs
* creating lib
* creating lib/strain_bench.ex
* creating test
* creating test/test_helper.exs
* creating test/strain_bench_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd strain_bench
    mix test

Run "mix help" for more commands.

And then we copied our exercise files into the project:

cp strain.exs lib/strain.exs
cp strain_test.exs test/strain_test.exs

We needed to make a few changes to our files:

We needed to remove or comment out the first three lines of the test, since this contained some Exercism-specific helper code that tried to load the main Elixir file from a particular location
We needed to rename our strain.exs in the lib folder to strain.ex so that Mix would compile this file and it could be tested.

With these two steps done, we could check our tests still passed:

mix test
..............

Finished in 0.1 seconds
1 doctest, 13 tests, 0 failures

Great! Time to benchmark.

Final implementation: Benchmarking

First, we needed to add that dependency. Easily done - the benchee page on Hex.pm has a dependency block to add to our mix.exs

# mix.exs
defp deps do
  [{:benchee, "~> 0.9", only: :dev}]
end

Then we download the dependencies: mix deps.get

After that, we followed a Elixir School guide on using Benchee. First step - create our benchmark.

# benchmark.exs
list = %w(apple zebra banana zombies cherimoya zelot)
checker = &String.starts_with?(&1, "z")

Benchee.run(%{
  "strain.comprehension" => fn -> Strain.keep(list, checker) end,
  "strain.enum" => fn -> Strain.keep_enum(list, checker) end,
  "enum.filter" => fn -> Enum.filter(list, checker) end
})

We had three implementations we wished to test:

strain.comprehension - our one line comprehension function.
strain.enum - our Enum function (we renamed this to keep_enum so we could benchmark it from the same module)
enum.filter - the normal filter function built into the Enum module.

To run these benchmarks, we simply needed to mix run benchmark.exs, which benchmarks each function in the map a number of times and then reports results. During the benchmarking, we saw some warnings explaining that our functions were a bit too fast and our benchmarking results might have deviation across runs. We “fixed” this by copy-pasting our list until it was larger, but we figured the warning could mostly be ignored since we were really just interested in a rough measurement. Here’s what we found.

Results and Summary

Environment:

Operating System: macOS"
CPU Information: Intel(R) Core(TM) i5-3230M CPU @ 2.60GHz
Number of Available Cores: 4
Available memory: 8 GB
Elixir 1.7.3
Erlang 21.0.6

Test 1:

Comparison:
enum.filter                16.71 K
strain.comprehension       13.76 K - 1.21x slower
strain.enum                11.53 K - 1.45x slower

Test 2:

Comparison:
enum.filter                16.03 K
strain.comprehension       10.77 K - 1.49x slower
strain.enum                 4.79 K - 3.34x slower

Test 3:

Comparison:
enum.filter                15.70 K
strain.comprehension       13.43 K - 1.17x slower
strain.enum                 9.57 K - 1.64x slower

Unsurprisingly, Enum.filter/2 is fastest - this makes sense, since Enum functions are widely used and I expect they have been heavily optimised.

Also unsurprisingly, our Enum implementation of keep was the slowest. We expected this, since we were doing two complete iterations of the list - once for the map, and once for the reject. This function also had a lot more deviation in speed over multiple benchmarking runs - probably just because there’s more going on in that function to get delayed.

Our comprehension sat very nicely, slightly slower than Enum.filter, but not by a huge margin - on average, 1.3 times slower, but still consistently faster than keep_enum. I would expect that the benchmarking time on Enum.filter and keep (with comprehension) would scale pretty linearly as well, wheras the keep_enum implementation would be a bit more unpredictable due to it’s additional complexity.

Overall, we came to the conclusion that Enum.filter is the obvious winner when filtering a list. Not terribly surprising to us! We did find that a list comprehension is still worth considering though, since it’s an extremely capable way of performing map/reduce type operations in a succint way. We also wrapped up the exercise with a real appreciation for how we maybe shouldn’t jump straight into Enum every time just because it’s mentioned in the exercise scenario.

List memory usage per Unicorn worker

Mon, 12 Nov 2018 00:00:00 +1300

I have been using this snippet recently to show the amount of memory currently being consumed by each Unicorn worker. This is a convenient way of checking:

The expected number of workers is being run
The memory usage of workers is fairly uniform
The average memory usage of Unicorn workers (if you are considering adding or removing workers)

Here’s the command:

ps -e -www -o pid,rss,command | grep -e '[u]nicorn_rails worker' | awk '{printf "%s %s\t%s mb\n",$3,$4,$2/1024}'

To break this down:

ps -e -www -o pid,rss,command - list each process’s pid, memory usage in bytes, and the command
grep -e '[u]nicorn_rails worker' - filter the command list to those than mention Unicorn workers
awk - this just pretty-prints the list in the format ‘[command] [memory] mb`.

My notes from Devops Days Wellington, Day Two

Mon, 05 Nov 2018 00:00:00 +1300

A little disclaimer: These notes are mostly for me to refer back to in the future, and are based on quick handwritten notes I took at the conference. They shouldn’t be used as a general reference since I may have misquoted, misremembered, or ventured an opinion that isn’t quite right.

My DevOps ticket was funded by my employer, Ackama.

DevOps Experiments: Reflections of a Scaling Startup: Alison Polton-Simon

This talk was a great start to the day, covering an organisational change that many smaller companies would emphasize with - offering support and ops resources to teams and customers in a growing company without affecting development velocity.

Alison’s talk started with an agenda, which I really liked, as it helped to frame the talk and the direction it was going in. The next slide laid out her employer’s values, which I also really appreciated being visible and upfront.

Action Point: The attribution against the key values slide was keyvalues.com - look at this to see if Ackama’s written values can be published publicly.

One of the core question Alison suggested teams, and companies ask of themselves is “What does reliability means to us?”.

Action Point: In my previous day’s notes, I identified an action point to identify and document the strategy and purpose of operations at Ackama. I would extend this discussion and documentation to include defining what reliability is, for Ackama and our customers.

The remainder of the talk covered a journey through a number of team structures to try and alleviate pressure from developers tasked to deliver features. I really emphasied with this points in this part, as it’s something I’m actively experimenting with. It was useful to see where we are along this track, and to also pick up some potential challenges and strategies for overcoming these in the future.

Alison described three team structures:

Tributes

This was the first team structure they tried out past their initial “everyone does ops, rotating pager duty”, just-in-time approach. A tribute was a rotating single engineer whose full time job it was to handle ops tasks, triaging support, supporting customers, and protecting the feature development team. The tribute person rotated every now and then.

Pros:

More sustainable than a whole-team system
Fewer interruptions
Chance to build up context around support and ops tasks
Opportunities to undertake longer-term ops and support tasks

Cons:

Customer service declined
Hard to hire for (needs to be able to do everything)
Hard to invest and grow the team - tribute always either fighting fires or trying to catch up with support.

Collectives

The movement from tributes to collectives was based around one person not being scalable - from hiring, technical, support quality or human perspectives. This team was intended to support specialisation of of particular team members, especially in regards to customer support and ops work. The collective rotated quarterly, allowing knowledge to be accrued both in depth and breadth.

Pros:

Better focus for devs and support/ops people
Accrued platform support knowledge
Crosstraining and opportunities

Cons:

Development teams don’t support what they build
Juggling act to load balance tasks
Context switch cost both per-task and per-quarter
A quarter isn’t actually that long when you incorporate the time required to get up to speed and handover.

Product Teams

The current iteration of team structure sees developers with support capability, developers with ops capability, developers and other cross-functional team members forming into product-focussed teams with capability to work reasonably autonomously. These teams are designed to align with business verticals rather than technical platforms (I imagine this means they could easily involve marketing/sales/BA/other business roles in the teams with ease if this doesn’t happen already). There is no ‘ops team’, since individual product teams own their own tools and systems. Each team has KPIs and goals to meet against which success is measured.

Action Point: Currently I work as part of a team that exists across several work streams, largely for resourcing convenience - it’s easier to allocate a project to a team rather than a specific person. Consider what measurable KPIs could apply to such teams.

Since the customer/developer ops support tasks didn’t go away with the move to product teams, the tribute model was retained, however this role was constrained to one tribute per team. This meant that the scope of work the tribute is expected to complete is reduced, and constrained to projects that the team already works with on a day-to-day basis (or at least has build experience of).

Action Point: Consider how a tribute system could work within the resourcing teams described as above. How much of a cultural/process change would be required to support this?

There were some challenges outlined with product teams:

Teams owned projects without necessarily holding responsibility for supporting projects. This was the main motivation to the tribute being introduced as I understand it.
Cross-organisation consistency and knowledge sharing is hard
Need to prevent a return to the tribute system where one person gets lumped with all the non-feature-dev tasks
Coordinating product teams takes more work than a single development team (this is for a startup or other organisation that would normally have a single dev team).

While wrapping up, Alison also mentioned an engineering consititution, and it’s benefits in working towards consistency, which I thought was a great idea. The engineering consitution is a collection of basic principles which new hires agree to meet. The constitution is a living document hosted on Github and is continually revised.

Action Point: Research existing engineering consitution documents and begin a process to formalise such a set of principles to be published publicly. Most of these principles have already been documented, they just need to be centralised.

Lean QA: Theresa Neate

I found this talk really interesting with lots of potential for my human ops work with stewarding and mentoring. It was one of a few talks this conference covering the need for testing and QA to be baked into the entire software development, release and post-release support cycles.

A recurring theme of this talk was T-Shaped people. I hadn’t come across this term before, but it relates to people who have general skills in a range of areas (the wide top of the ‘T’), and specialised skill in one precise area (the bar of the ‘T’).

Action Point: Consider how I could work with my mentees to figure out their ‘T’ and how they can focus on general or specialist skills.

Theresa also redefined Quality Assurance as Quality Analysis, which I found to be a really good reframing of how I see QA fitting into development processes. Ideally, they are involved throughout the software development process, and should not be seen as gatekeepers with a pass/fail role - instead, they should be seen as generalists with a specialisation in testing who can find potential problems or quality issues in features, and provide assistance and advice to work with whoever necessary to resolve these.

Theresa’s role was described as “Developer advocate role”. She compared herself to Kelsey Hightower (a well-known Kubernetes developer advocate at Google), but internal facing rather than external facing. This role really reasonated with me, since my professional focus in the last few months has been to introduce some consistency in the tools and processes used at my workplace to deliver unsurprising, high-quality code.

Action Point: “The Phoenix Project” was recommended, reminding me that I should circulate this novel yet again within Ackama.

Action Point: Review a presentation from 2008 called “Agile Infrastructure” that was mentioned several times.

A Devops Confessional: Mrinal Mukherjee

Not many notes from this one. I enjoyed the term “DevOops”, and the monkey slides.

There were three devoops situtations that Mrinal identified:

“One-track Devops”

Same process for each new devops project
Take care of the low hanging fruit, get short term wins
Automate infrastructure, cos it’s cool and easily understood
Become bottleneck as the “devops person”
Become only person who can build and release the app
Discover that process and culture is causing issues, not just technology
Business doesn’t see results because only low hanging fruit is done, no actual painful problems.

“MVP-Driven DevOps”

Get stuff done fast
Start saying things like “We’ll fix it in the next sprint” (no future sprint planned)
Acrue technical debt
Make compromises with quality and security
Hard to maintain code
Slow to build new features because of tech debt, unplanned architecture and quality issues

Can be somewhat avoided with:

Peer review (of architecture, approach and consistency, not just bug spotting)
Minimum engineering standard (analogous to the “engineer constitution” described above)

“Judgemental DevOps”

Delays from other parties/teams
Not understanding the context of changes
Blame Driven Development - blame the other teams, other staff
Become stubborn and defensive
Results in conflict - even as minor as body language can be a giveaway

Mitigate with:

Empathy and shared context with other teams, processes and individuals
Understand goals, constraints and priorities
Collaborate on problems to create solutions

Kubernetes/OpenShift Open Session

A somewhat interesting session where a bunch of large kubernetes deployments talked about the tools they used for logging, monitoring, secret management, configuration and networking. Some little takeaways I got:

~50% people in the room used Kubernetes
~2% used Kubernetes via Google
~2% self-hosted in public clouds (AWS, Azure, Google)
~2% V-Sphere
~5% OpenShift
~1% Rancher (possibly the rancher rep?)
Mostly people had a cluster size of 4-5, some with many more
Mostly people ran 10-20 nodes in a cluster
Storing state is normally the hardest bit (databases, static files)
It can be done, overall don’t run DB in containers unless it’s throwaway (e.g. integration testing)
Majority of people using ECR as the container registry. Some using OpenShift’s internal registry.
Logging stacktraces as JSON blogs is good for busy logs since it’s all on one line
kubectl top and kubectl state metrics are good for monitoring
Ansible and ansible tools (ansible vault, etc) were far and away the most common thing used

Monorepo/shared libraries Open Session

This session didn’t end up being super relevant to me, aside from a couple of little snippets around where infrastructure vs orchestration vs app code should go.

Ignite Session: Continuously Testing govt.nz: Allan Geer and Amanda Baker

An interesting lighting talk covering the constraints of CWP/Silverstripe that I’ve noticed is typical:

Low motivation to regularly update
High risk/cost upgrades
Little to no testing

Additionally, being in a government department, approaches to agile can be interesting. Their flavour of agile was described as “micro-waterfall agile”. Sounds about right.

Allan and Amanda introduced test coverage to govt.nz websites by building the creation of Gherkin test cases into story writing sessions with the product owner, devs and testers. They created an initial stocktake of features to describe existing systems and then prioritised to discove the most important scenarios to test. They then made sure their test metrics were visible to the product team to motivate further testing.

Action Point: Document strategy of stocktaking existing application features and prioritising important revenue/customer generating features as part of legacy app discovery processes.

Ignite Session: Configuration Pipeline: Ruling the One Ring: Jules Clements

Another quick lightning talk about release processes and separating “deployment from “orchestration”. I was also reminded of an additional requirement of a deployment pipeline that I don’t often consider, which is to ensure that the output of a pipeline is a uniquelely identifiable artifact that can be repeatedly deployed (i.e. deployed, rolled back, and deployed again). A Docker image has these traits, and that’s probably the artifact that was intended, but it’s an additional note to make sure that deployments using tools like Capistrano also share these traits (for example, Capistrano deploys to release folders, and points to the “current” one using a symlink. It also writes the hash of the git commit that was deployed into the REVISION file in each release.

Action Point: Come up with a good definition of orchestration I can get my head around

Conference Takeaways

Overall, the conference was OK, but not great. The scale and type of work I do means that I need to constantly translate many of the things I’m hearing in talks down to a much smaller scale, and then decide whether they are problems I have, or if they are, if they are actionable. I think talks covering smaller-scale devops would have been interesting. There were a couple of talks that were this kind of thing, but they mostly covered scaling devops teams, not devops processes and technologies.

My main high-level ideas to delve into more:

Develop ops capabilities within each project team. Don’t have an “ops team”
Formally define ‘ops’ - it’s goals, mandates, and measurable KPIs
Define KPIs for a project that can be built into healthchecks and monitoring early on in the project
Focus on value to customers, not MVP
Figure out how to continue testing post-release
Determine and document goals, constraints and priorities within a project charter
Publish organisational core values
Move away from tribute-style teams to product-ownership teams (and figure out what a “product” is at a services company. Is it a client? A client’s application? A collection of similar applications?)
Quality analysis, not Quality assurance
Document and publish engineer constitution

My notes from Devops Days Wellington, Day One

Mon, 05 Nov 2018 00:00:00 +1300

A little disclaimer: These notes are mostly for me to refer back to in the future, and are based on quick handwritten notes I took at the conference. They shouldn’t be used as a general reference since I may have misquoted, misremembered, or ventured an opinion that isn’t quite right.

My DevOps ticket was funded by my employer, Ackama.

Moving from Ops to DevOps: Centro’s Journey to the Promiseland: Jeff Smith

The most useful talk for me out of the whole day. Discussed the frequent conflict devops has with both “dev” and “ops” (e.g. access vs security), and how many organisations attempt to merge two teams with different context and motivation and instead end up with three teams.

Took job offered as “Devops Manager” - condition of hire was to rename job title to “Production Operations Director”, which makes much more sense.

Align teams and tasks to centralize work. Good questions to explore team boundaries:

What frustrates you?
What makes you come to work?
What makes you not want to come to work?

Action Point: I’m going to trial building these types of questions into my stewarding/mentoring meeetings

Jeff talked about how developers didn’t have access to set up alerts for the systems they created. I have experienced similar pains on both sides of the dev/ops boundary.

Action Point: I’m going to consider how I can open up and make it easier for other developers that I work with to understand and create customer-specific monitoring and alerts.

He also talked about how making a dev environment destroyable and recreatable helped to remove some frustration from both sides of devs/ops. I haven’t experienced this specific problem, but am going to consider staging and UAT environments for some customers being destroyable/recreatable in this way.

I was prompted to consider what KPIs should be considered for particular sites, and to what extent these should fit into monitoring and alerts.

He also pointed out that infrastructure-as-code is not really “self documenting” to a sufficient level, and advised also producing architecture diagrams and some documentation about what key decisions were made and why.

Action Point: I am going to look at producing both AWS and regular architecture diagrams for base Terraform configurations and publishing these internally.

The most useful action point I got from the talk was to define the scope of the operations team. What is it’s primary purpose, which should drive all the work it performs.

Action Point: Prepare draft team charter for the operations team outlining above.

As a final takeaway, it was mentioned that for a larger ops team, keeping track of planned and unplanned work is useful for understanding the health of the team. I think this is a fantastic idea that applies to most project teams that have an element of BAU.

Action Point: I’m going to trial this reporting mechanism at the conclusion of my next development sprint and report in the sprint review.

Monitoring that cares (the end of user based monitoring): François Conil

This talk was all about monitoring and alerting in the context of the user. One interesting point was the timezone difference between NZ and US makes it difficult to find times that “aren’t important” for maintenance windows - e.g. a downtime window at 2am in NZ time would be the middle of the day in the US. Not a concern for what I’m working on now, but valid for larger applications.

They also noted that if everything is classed as critical, then nothing is, describing the concept of an actionable alert as:

Something is broken
Customer has noticed
You are the person who can fix it
You need to fix it immediately

Unless all of these critieria are met, then the alert is not actionable and may as well not be delivered.

A recurring idea that I noted for a few talks was to design the KPIs of the application early on in the development cycle, and monitor these as a primary metric, rather than more low-level metrics. Monitoring customer journeys can reveal issues that standard CPU/memory/disk monitoring may not detect.

Transforming the Bank: pipelines not paperwork: Mark Simposon, Carlie Osbourne

This talk was interesting, but was largely around a successful business process/change management project. I always enjoy hearing about life in larger enterprises. Occasionally I get to hear about some interesting technical challenges, more frequently about the overhead introduced around having multiple levels of decision makers and stakeholders.

An interesting note was that ANZ chose a project that was significant enough to the organistion to prove the concept - internet banking in their case. They started with a 6 week development cycle, then a 4 week release cycle that was frequently extended. Thye noted that the dev team had a sense of disconnect with waiting so long before they would see their code in production. They also had a high deploy cost, since each release would be significant - they staffed this at 25-30 people and usually had 4-5 hours of downtime.

Most of their change seems to be designed around a Jenkins pipeline that was designed to take their code right through from a commit against master to production (running tests, UAT deploy, etc along the way). This allowed them to drop to a weekly release instead of ~10 weeks.

For testing, they cut their selenium tests and replaced with Cypress. I’ve done some brief looking into Cypress, but I’d be curious to know how many Selenium tests they gave up to rewrite, and what their coverage ended up being - seems like an odd cost/benefit tradeoff.

The other refactoring they performed was to their change management process, getting stakeholders to sign off on the release process, rather than each release, so they could re-use the signed off release process each time without needing the overhead of getting sign off.

Mark suggested playing down discussion of ‘MVP’, and instead focus on value to the customer. Release small features often, listen to customer feedback, and interate.

Action Point: I’m interested to talk through this idea with some other developers and scrum masters at Ackama, since I think this is a very important distinction in how I might frame discussions around a feature or application.

ANZ’s release process is now a 4-5 day cycle time to get code into production, followed by a 8 week incremental release process for canary deploys.

Fighting fires with DevOps: Ryan McCarvill

This talk was way different from how I thought it would be, but because of the structure of the talk, a bunch of my notes were invalidated. The talk covered hardware/software packages deployed to fire appliances for Fire and Emergency NZ, and how a capable but complicated system was compromised down to a really interesting but minimalist solution due to unforeseen constraints (temperature in this case).

The initial hardware/software involved using a custom orchestrator to pull and run images from Azure Container Registry onto a hardened Linux box on the appliance, paired with a multi-network comms unit. A tablet and separate nav screen was used for the presentation of data. The box was provisioned using Ansible, and both the Ansible scripts and docker image sources were controlled out of git. This all worked in theory, but in practise the custom hardware boxes would overheat.

With an impending deadline and all the hardware budget consumed, they ended up drastically simplifying, moving a lot of the systems that were designed to run directly on the truck into the cloud, and writing two small systems to run directly on the comms module, which was a locked-down linux distribution that really just had Java available. One of these systems was a simple static file server, which would serve off an external USB device, and the other was a search tool that ended up being written in Rust after all other programming languages available fell over for one reason or another.

A super interesting talk anyway, and having done a similar project, I got a bunch of takeaways for next time. I’m still super interested in how an Elixir umbrella project would function on a low-resource hardware unit such as the one discussed in the talk. Especially since a tool like distillery may be able to package up the runtime and application in such a way that it would just work.

Overall, it was a super interesting story of reverting back to using nice simple, old style technology to solve a new problem.

Open Space Session: Devops at NZ scale

I attended this session, but it pretty quickly went off-topic without an external facilitator, turning into a discussion about how devops functions in different teams, and even a whip-around of job offerings. I didn’t speak up in this one as it took me the entire session to figure out the nature of the discussion I wanted to have. I eventually figured out that I attended the session hoping to have a discussion about “Devops at small scale” (small == NZ really). I work for a services company, so unlike our (generally) larger project company buddies, we’re usually just setting off our customers on their journey, and doing so to the best of our abilities. Devops work is inherently constrained and limited at this stage, so I was interested to get an answer here.

Open Space Session: Devops for startups

The above session led nicely into this one, since my colleagues and I had just come from the previous session, and found the topic to be pretty much what we were after. The takeaways I got from this discussion was to start small and with a consistent kind of approach in mind. It’s always better to start with infrastructure-as-code rather than retrofitting this, and doesn’t take much longer, if it takes longer at all. Generally it’s a good idea to dockerize applications from scratch, but this does require balancing as to whether it will be maintained or is suitable for the platform in question.

A Trip Down CI/CD Lane: Everett Toews

This was just a five-minute talk but I still found it interesting. An overview of CI/CD platforms and their pros and cons.

Creating Shared Contexts: Jeff Smith

Another 5 minute talk, with SO much packed into it. Jeff was back talking about establishing contexts that are shared between teams, and the importance of making sure that context was being communicated, not just a problem, or even worse, a pre-envisaged solution (that the other person may think is the only option). Jeff encouraged thinking about problems, not solutions.

Action Point: Getting teams to agree on goals, constraints and priorities is super important for teams to function (both within and without the team). I’m going to work to incorprate these into future sales pipelines through to project charters and retrospectives.

2018: A Build Engineer’s Odeyssey: Peter Sellars

Another 5 minute overview of a bunch of different tools that a Node.js pipeline engineer might be faced with coming into a role as a dedicated “devops person”. It was mostly material that had already been covered/I’d already picked up (it was intended to be a high level overview, I think). The main takeaway I got that was slightly unexpected was npm ci, which is probably what I want to be using within Rails and Phoenix projects when I just want my locked dependencies (e.g. during Docker builds).

Testing in DevOps for Engineeers: Katrine Clokie

This talk was great, but by now I was getting pretty tired. I really wanted to concentrate since I’ve been deep diving into testing recently, and it contained some really useful lessons around an entire project team’s culture revolving around testing. Katrina intercepted really well with François’s talk about monitoring, coming at the same idea from a testing perspective around driving testing, development and ops processes based on customer journeys and application purpose rather than just “standard metrics”. Something to consider for my projects:

“What questions are you asking of your product that you know it’s good, it’s ready?”

Action Point: Pick a reference project and consider what healthchecks, KPIs, and possibly a sample of intergration or system tests that could be run to continuously test application quality after release.

Action Point: Check out “A Practical Guide to Testing in DevOps” by Clokie.

Open all files as tabs in in vim

Sat, 03 Nov 2018 00:00:00 +1300

Just a quick trick I learned yesterday: I often glob files into vim from my terminal to view them quickly - for example, given the following files exist:

> ls 
file1.txt
file2.txt
file3.txt

When I run: vim *.txt

Then I expect file1.txt, the first file, to be opened

When I run the command :tab all within vim

I then see a tab for each file I have opened:

This is a very nice shortcut that I expect to use often!

RSpec's `have_attributes` matcher

Tue, 23 Oct 2018 00:00:00 +1300

I’ve been working on rewriting some legacy feature tests that have been unmaintained for several years as system tests. One of the most convenient methods I have found while creating these tests is have_attributes.

The RSpec documentation I linked to above is a great overview of the matcher. I have found that it’s useful to matching the result of a service object, processor or even a controller action when we expect the resulting model, or models to have particular attributes set, but do not need to match on the entire object.

Here’s a simple example of have_attributes in action:

class WidgetCreator 
  def create
    Widget.create(status: :enqueued)
  end
end

RSpec.describe WidgetCreator do
  describe ".create" do
    subject { WidgetCreator.new.create }

    it "creates the widget" do
      expect { subject }.to change(Widget, :count).by(1) 
    end

    it "marks the widget as enqueued" do
      expect(subject).to have_attributes(status: :enqueued)
    end
  end
end

have_attributes accepts a single attribute key-value pair, or multiple. If multiple attributes are provided, then all keys and values must match. This matcher also seems to function against any other object which exposes attributes using simple setter and getter methods - like ActiveModel::Model, Struct and OpenStruct. This is based on the implementation using __send__ to read each attribute key from the actual object before comparing the value.

Mitre Flats: Tararua Forest Park

Wed, 03 Oct 2018 00:00:00 +1300

This trip was a bit opportunistic, making the most of a fine weekend in late winter to start getting back into the Forest Park before the official start of the summer season!

Mitre Flats is one of the more popular spots to visit from the Wairarapa side of the Tararua range, and has been on our to-do list for some time. Mitre Flats is a good location to start from if heading towards one of the nearby high points:

There’s also a tricky-looking loop track to do that could be done in a long weekend, heading into Mitre, up and across the Broken Axe pinnacles (a narrow rocky ridge from the looks of it), and down to Jumbo Hut, returning up the Atiwhakatu valley back to Mitre. Something for another day!

The track to Mitre is known by some sources as the Barra Track, or just the Mitre Flats track. It’s reached about 15 minutes from Masterton, past the Mt Holdsworth turn off and after skirting around the outside of Masterton on the ring road that follows along the train tracks. The rural road turns to gravel a few kms before the track start. The road seems to terminate around the start of the track, near a fenced-off bridge over the Waingawa River. At the track start, there is a lay-by on both sides of the road for parking, and a small shelter. No toilets or fresh water supplies here, so you’ll be pretty quick to set off through the pedestrian gate next to the cattle grid and along the farm road.

A slightly different approach to the hills than we’re used to around Wellington, you’ll be continuing along this farm road for the first 30-45 minutes before the park boundary is reached. It’s not a particularly dull walk though, as the track follows alongside the river, which looks like it has some great swimming holes. After a short time, you’ll reach the entrance to one of the farm houses, where the track turns 90 degrees to head along the property boundary, before crossing farm road beside a shearing shed and dropping off the terrace to cross a swampy bit and a small bridge across a creek before arriving next to the river. About 10-15 minutes after this point the bush edge is reached, with a short and steep climb back up to terrace level where there is a boundary fence with a stile, and a pretty old and rotten looking sign noting that you’ve just entered the Tararua Forest Park.

From here on in, the track follows a pretty similar pattern until the swingbridge across the Waingawa River is reached a couple of minutes before the hut. The track initially climbs until a point approximately 100m above the river is reached. The track continues along this contour for the next couple of hours - don’t make the mistake I made of assuming it’ll be cruisy though - there is plenty of undulation going on, with a few rocky scrambles. The track is mostly well marked, except for a couple of points around the halfway point where some small rock slips have concealed the original track route. You won’t get lost here, but after going through some knee-high grass, the track will appear to drop off a couple of metres to the left where you’ll then get to a slippy slope to scramble up. Instead of going to the left here, go to the less-obvious right hand route which will save this scramble.

You’ll know that you’ve almost reached the swingbridge when the track begins descending - in traditional Tararua fashion, this is straight down the hill to river level. The bridge is planked rather than aluminum struts or mesh, so it’s quick and easy to cross. On the far side of the bridge is the intersection with the track to the Atiwhakatu Valley, with the turn off to the track to the summit of Mitre just after this intersection. The hut sits in a really nice clearing less than 5 minutes after the bridge.

On this particular trip, we had chosen to camp (since we had the dog with us). I’d say it’s one of the nicest, if not the nicest spots we’ve been to with this. The clearing was large enough for us to comfortably share with a high school outdoor ed group of about 20 students, most of whom were also camping. There’s some nice flat areas past the hut and just inside the bushline on the track to Cow Creek. The hut itself is well maintained, I believe by the Masterton Tramping Club, and sleeps 20. There is a supply of rainwater from the roof and a nice verandah area to enjoy the last of the sun before it disappears (quite early, since the hut is deep in the valley!). Since we were just in for the right, I had treated myself to carrying in an air couch to relax on and catch up on some reading. A perfect spot, however it did get below zero overnight - cold enough to freeze water we had outside the tent.

Returning is much the same, however due to the undulation, the track does feel quite different going the other way. The climb up from the swingbridge is mercifully brief, and after that it’s just a matter of knocking off the distance until the farmland is reached. Meandering back along the farm road in the sun was a nice way to wrap up the trip, while looking at the tempting pools in the river below.

This was very much a recon trip for us, as we’ve had our eye on Mid King Biv, Broken Axe Pinnacles, and the Cow Creek area for a while. I think we were probably hoping for a slightly easier trip than we ended up with, but the great campsite more than made up for the slightly tougher walk!

The DOC time for this track is described as 4 hours. It took us 3.5 moving quite slowly with lots of breathers, but I think that because of the track surface and terrain that time is probably what most people could expect to do it in. For more information on hut facilities and up to date alerts, see the Mitre Flats Hut page on the DoC website.

Sending DHT22 readings from a Raspberry Pi Zero to MQTT

Sun, 30 Sep 2018 00:00:00 +1200

I spent a couple of hours last weekend getting a RaspberryPi Zero (W) up and running as a home automation node, joining my existing fleet of MySensor Arduino nodes. The Raspberry Pi is being used instead of an Arduino as it is eventually going to perform a number of functions along with climate sensing - for example, bluetooth presence detection for room-based location.

I found a number of resources documenting how to do parts of the system that I cobbled together, but not an overview of the entire thing - that’s what this post, and https://github.com/joshmcarthur/raspberrry-pi-dht22 is intended to be.

1. Hardware

I used a RaspberryPi 0 (W). The W means it has onboard wifi and bluetooth. I’ve had no problems with connectivity and the hardware itself is looking super stable. I got the RaspberryPi boards of Aliexpress, so there’s no telling if they’re official or not. They were around NZD$20 each from memory.

The DHT22 sensors I got in a 5 pack a few months ago to eventually replace the DHT11 sensors I have on my Arduino nodes. Also from Aliexpress.

2. OS

I’m using Raspbian Stretch Lite on the Raspberry Pi board. I run all of the home automation stuff that I can in Docker (which now has official support for ARM boards, the Raspberry Pi in particular), using docker-compose as a lightweight container manager.

While there is an overhead associated with running containers in Docker, I find that it is useful to establish separation of concerns and boundaries between different tooling for different tasks. It’s not necessary to use Docker for this, but my repo does include a working Dockerfile and docker-compose.yml.

3. Wiring

The wiring for these sensors is pretty simple. See https://pinout.xyz for a pin layout of the Raspberry Pi GPIO pins, but the wiring is simple - 3.3v from the bottom left corner of the GPIO goes to the left hand pin, ground from the bottom right corner of the GPIO goes to the right hand pin, and the data pin is connected to GPIO4. The data pin is the second from left (the remaining pin on the DHT22 is unused for most use-cases), and GPIO is 4th from the left on the bottom row of the GPIO pins.

I found many forum posts that suggested that a resistor should be placed within this circuit, however I found that I was able to get readings without one. GPIO4 in particular does have a pull-up resistor that may have been sufficient to work with, so other GPIO pins may not work as well.

4. The Python script

I’m using Adafruit_Python_DHT to communicate with the DHT22 sensor. This library is Python with a small C component for GPIO integration that abstracts over the actual sensor reading. I initially tried to just use the example script built into this library, but I found that having the output serialized for humans wasn’t particularly easy to convert into an MQTT payload. Instead, I used the same API that the example script uses, but I serialize the message into a JSON datastructure. Errors are logged to STDERR so that they can be logged without accidentally piping error messages into MQTT.

5. The bash script

The bash script glues everything together. It runs a while true loop with a sleep-based delay that calls the Python script, checks it succeeds, parses the JSON to extract the temperature and humidity, and sends this data to MQTT. This script should work for most purposes, but this is also where most of the ‘me-specific’ configuration lies.

6. Tying it all together: docker-compose

The Docker image is Alpine linux based, and is just over 200mb when built (I haven’t optimised this particularly). The image installs Adafruit_Python_DHT via a git clone, and then replaces a shebang that is hardcoded to expect Python in a particular path that is mising in the Python base image. It then adds the Python and bash scripts and sets up the polling script as the CMD (so this will run by default if no other command is given).

The docker-compose file mostly just mounts the current working directory in /opt/dht22 so that the poll and python files can be revised easily. The docker-compose.yml file also instructs the container to be restarted if it fails for any reason (restart: always) - this will also cause the container to be started on boot. For GPIO access, privileged: true is the easiest way to allow the container to communicate with the GPIO pins, although there are alternatives if you’re using your Raspberry Pi in a secure environment. Finally, there are a couple of environment variables that define the IP address of MQTT and the topic to publish to.

7. Result!

With a quick MQTT Sensor configuration in HomeAssistant, the sensor shows up!

Resources:

Github Repository

Automatic text wrapping in vim

Wed, 19 Sep 2018 00:00:00 +1200

This is something I use all the time. I don’t have it in my ~/.vimrc yet - partially because I haven’t got around to it, partially because I sometimes want to use different line lenghts, and partially because I don’t want to bikeshed myself trying to decide on what line length I want for different file formats.

Vim supports soft wrapping, which is something a bit different - it only affects the display of the line, not the actual line length. Softwrap is great for scripting or programming, where a line break in the wrong place might have a negative impact on your ability to actually compile and/or run the file!

Hard wrapping actually adds line break characters when the line exceeds a number of characters, and is therefore great for git commit messages, emails, blog posts, letters, and any other contexts where actual content is being written.

To set a hard wrap at a particular character length, simply run set tw= within a vim window. Don’t forget to use the ‘:’ character to enter into the appropriate mode (I think this is called ‘command mode’?) before running this. I usually use one of the following settings:

set tw=80
set tw=100
set tw=120
trending towards shorter lines for things like git commit messages, and longer lines for longform content like a blog post. When the line character count reaches the wrap limit you have set, it will automatically drop onto a new line, increasing the line count in the gutter on the left.

An access token state store in Elixir

Wed, 19 Sep 2018 00:00:00 +1200

I’ve recently been working on a small project that uses OAuth for authentication, which is proving to be a great approach for what I need it for. After I got the OAuth authentication layer working through, I realised that I needed a place to store the hash of user data I receive back from the OAuth provider, along with some kind of access token I can use to authenticate requests to my GraphQL API. This post describes the state store I created to manage access tokens.

Elixir has a built in module for background processes called GenServer. I actually managed to implement most of what I needed simply based on the excellent documentation. The API for my module wasn’t very complicated - I just needed to insert access tokens, destroy access tokens (for logout and invalidation), and check whether a given token exists.

The module ended up quite simple:

defmodule ElixirAuthTokenStoreExample.Repo do
  use GenServer
  @name __MODULE__

  # Client

  @doc """
  Start the token repo server

  ## Example

    iex> {:ok, pid} = ElixirAuthTokenStoreExample.Repo.start_link(name: :repo_start_link_doctest)
    iex> is_pid(pid)
    true
  """
  def start_link(opts \\ []) do
    opts = Keyword.put_new(opts, :name, @name)
    store = []
    GenServer.start_link(__MODULE__, store, opts)
  end

  @doc """
  Insert a new token into the repo

  ## Example

    iex> {:ok, pid} = ElixirAuthTokenStoreExample.Repo.start_link(name: :repo_insert_doctest_1)
    iex> ElixirAuthTokenStoreExample.Repo.insert(pid, "abc123")
    iex> :sys.get_state(pid)
    ["abc123"]
  """
  def insert(pid \\ @name, token), do: GenServer.cast(pid, {:insert, token})

  @doc """
  Remove a token from the repo

  ## Example

    iex> {:ok, pid} = ElixirAuthTokenStoreExample.Repo.start_link(name: :repo_destroy_doctest_1)
    iex> ElixirAuthTokenStoreExample.Repo.insert(pid, "abc123")
    iex> ElixirAuthTokenStoreExample.Repo.destroy(pid, "abc123")
    iex> :sys.get_state(pid)
    []

  """
  def destroy(pid \\ @name, token), do: GenServer.cast(pid, {:destroy, token})

  @doc """
  Check whether a given token exists in the store

  ## Example

    iex> {:ok, pid} = ElixirAuthTokenStoreExample.Repo.start_link(name: :repo_exists_doctest_1)
    iex> ElixirAuthTokenStoreExample.Repo.insert(pid, "abc123")
    iex> ElixirAuthTokenStoreExample.Repo.exists?(pid, "abc123")
    true

    iex> {:ok, pid} = ElixirAuthTokenStoreExample.Repo.start_link(name: :repo_exists_doctest_2)
    iex> ElixirAuthTokenStoreExample.Repo.insert(pid, "abc123")
    iex> ElixirAuthTokenStoreExample.Repo.exists?(pid, "abc1234")
    false

  """
  def exists?(pid \\ @name, token), do: GenServer.call(pid, {:exists, token})

  # Server

  @impl true
  def init(tokens) do
    # This is the 'state' of the server
    {:ok, tokens}
  end

  @impl true
  def handle_cast({:insert, token}, state) do
    {:noreply, [token | state]}
  end

  def handle_cast({:destroy, token}, state) do
    {:noreply, state |> List.delete(token)}
  end

  @impl true
  def handle_call({:exists, token}, _from, state) do
    {:reply, state |> Enum.member?(token), state}
  end
end

This genserver can be added to your Mix application supervisor, which will ensure that a a process will always be running holding the state that is available application-wide. Because of the simple API we have defined, interacting with the token store is quite simple:

alias ElixirAuthTokenStoreExample.Repo

Repo.start_link()

# Insert a token
Repo.insert("abc123")

# Check it exists
Repo.exists?("abc123")

# Destroy a token
Repo.destroy("abc123")

I have placed this example code into a reference mix application, available at https://github.com/joshmcarthur/elixir-auth-token-store-example.

Generating error messages using ActiveModel::Errors

Mon, 10 Sep 2018 00:00:00 +1200

ActiveModel::Errors are widely used in ActiveRecord and therefore Rails, and while custom messages can be provided to each validation using the message key, it’s much more flexible to instead pass a symbol as the message, which will cause the message to be looked up using the internationalisation framework provided by the i18n gem that is a dependency of Rails. Here are some examples:

validates :title, presence: true, message: "needs to be filled in"
validates :title, presence: true # Will use built-in validation message look up
validates :title, presence: true, message: :blank # ^ The same effect as the above
validate do
  errors.add(:title, :contains_today_i_learned) unless title.includes?("TIL")
end

Using an internationalised message is great for a number of reasons:

It keeps strings out of your model
It allows for custom error messages to be constrained to a single location
(The obvious) - it allows error messages to be customized based on the current locale

Once these custom errors are built into your model, you may want to add a test to assert that the expected error messages are generated. You can assert against the string itself of course, however if you run your tests in multiple locales, or just want to keep validation message strings out of your model TEST as well as your model, the following method can come in handy.

When you run validations, an instance of ActiveModel::Errors is available as the return value of the errors method on your model instance. This instance is normally used for accessing all the validation errors on a model, but it also has a number of convenience functions - in particular, the one we’re interested in is called generate_message. This method accepts an attribute, and the custom error (the symbol passed to the message option of your validation, or a built-in validation such as presence, inclusion, etc., and returns the string form of the message. It will respect the current setting of I18n.locale when looking up error strings.

Here are some examples of looking up errors on a model (the instance of which is called subject in this example):

subject.errors.generate_message(:title, :blank) 
# => "can't be blank"

subject.errors.generate_message(:title, :contains_today_i_learned)
# => "must include the category 'TIL'"

I18n.locale = "mi-NZ"
subject.errors.generate_message(:title, :contains_today_i_learned)
# => "me whakauru i te tuhinga 'TIL'"

For more information on the I18n translation format, and exactly how these above messages are resolved, see the I18n Rails Guide section on error message scopes.

Configuring Cloudflare for a Heroku application

Wed, 05 Sep 2018 00:00:00 +1200

Cloudflare is a brilliant service, and I’ve been using it for personal use for a couple of years now. In particular, its universal SSL, CDN, and always online capabilities have really come in handy.

I use Cloudflare for my personal domain, and frequently build little toy applications that I usually host on Heroku free tier dynos. Because of how Heroku suggests that DNS records are set up, there are a couple of tips that I wanted to write about today. Overall, this post focusses on setting up a application hosted on Heroku on a subdomain. If you also want to have your application available on the root record of your domain, I outline how to do this at the end of the post.

The thing to be aware of with Cloudflare is that it completely takes over your domain to offer the capabilities it does. This means that your DNS records and managed and resolved by Cloudflare, as well as caching, SSL, etc. The difference between configuring an app in Cloudflare and with a “normal” DNS provider is that you do need to get all the settings correct for everything to work together.

The first thing to be aware of is where to direct your DNS records towards. If you follow Heroku’s guide, they will instruct you to specify a CNAME record against the subdomain which you wish to use, directed at “subdomain.your-domain.herokudns.com” (for example, “www.joshmcarthur.com.herokudns.com”). This is to allow Heroku to provision and serve an LetsEncrypt SSL certificate for the domain(s) you have configured. If you direct Cloudflare towards this domain though, you’re likely to run into redirect loops, or even find that Heroku is unable to find your app. For your app to resolve correctly, you should instead direct your subdomain at a CNAME record of your normal .herokuapp.com domain - for example, “www-joshmcarthur-com.herokuapp.com”.

Once your CNAME is correct, you will need to configure SSL for your application. At this point, if your application is set up to force SSL in production (in Rails, this is config.force_ssl = true, in Phoenix this is config :my_app, MyApp.Endpoint, force_ssl: true), then you might have a redirect loop going on. This is because the default with Cloudflare is to use ‘Flexible’ SSL. This means that the network connection is doing something like:

“https://test.domain.com” (Cloudflare) -> “http://test-domain-com.herokuapp.com” (Heroku) -> “https://test.domain.com” (Redirect from app forcing SSL) - repeated over and over.

In other words, flexible SSL will connect to your origin (Heroku), via HTTP. The protocol that your app will see in the headers will be “http”, so it will change the protocol to “https” and redirect - letting the same flow occur over and over again.

To resolve this problem, simply change the SSL mode (under the “Crypto” tab in Cloudflare) to “Full” - or even “Full (Strict)” if you want to be extra good. The ‘Full’ SSL mode means that Cloudflare will connect to your origin via HTTPS instead of HTTP, and your application will not need to perform a redirect to a secure origin, since the request is already secure. The difference between ‘Full’ and ‘Full (Strict)’ is simply that ‘Full’ accepts any SSL certificate, even if it is self-signed, expired, or has some other problem. ‘Full (Strict)’ requires that the certificate is valid for the domain, is from a trusted issuer, and has not expired.

With these two configurations in place, you should have your application functioning via SSL on your subdomain of choice. You can have a play around with different cache and perfomance settings if you wish, though I’ve found the defaults here to work well for me.

If you just needed to configure a subdomain here, then you can stop reading - you’re done! If you would like your domain root to also serve your application, there’s just a couple more configurations you should put in place to get this working.

The first is a DNS change specific to Cloudflare (well, specific-ish, several DNS providers support something similar) called “CNAME flattening”. CNAME flattening means that you can specify a CNAME record on the root of your domain (this is not normally valid, normally an ‘A’ record is used, and ‘A’ records must be an IP address which Heroku does not support). When a client asks Cloudflare what the domain root record should resolve to, Cloudflare will resolve the domain configured as the CNAME to an IP address, and return this as an ‘A’ record. This allows for autoscaling services like Heroku, Cloudfront, and AWS ELB to be specified as the root record of a domain - all of these services use a full hostname to the endpoint, but autoscale servers in the background. With CNAME flattening, it will not matter what IP address these hostnames have, Cloudflare will figure it out and ‘follow’ them as they scale.

For your root domain record, the easiest thing you can do is specify a CNAME of the subdomain you configured earlier - for example setting the CNAME record of “joshmcarthur.com.” to “www.joshmcarthur.com.”. Setting this value means that if you change where your subdomain points to, your root record will follow that setting automatically. Once that setting is applied, you should be able go to your domain, and see your app - still available via SSL. Neat!

If you’re like me though, you really only want your app to be available on one particular hostname - normally www.myapp.com. Luckily, Cloudflare supports “Page Rules” which you can use to redirect requests - this means that you can put a Page Rule in place that will redirect any requests to your root domain to your subdomain. This redirect will be marked with a 301 Permanent Redirect status, which means that client browsers will cache and remember to just go straight to your subdomain next time. To add a Page Rule, go to the “Page Rules” tab, and select the option to create a new rule. The first thing to provide is the pattern that the URL must match for the rule. You’ll want to enter mydomain.com/*, which will match your domain without any subdomain, with any path. You can then add one or more settings - we will just need one today, but you can add many settings to your rules to change how different URLs will behave. After clicking on ‘Add Setting’, select a setting type of ‘Forwarding URL’, and select ‘301 - Permanent Redirect’ as the status code. The destination URL should be your subdomain that you’d like to use, followed by ‘$1’ - the ‘$1’ will be replaced with the wildcard value you specified in the pattern, which will be the path the client is going to. An example of a destination URL would be “https://www.joshmcarthur.com/$1”.

The complete rule definition should look something like:

With these steps complete, you’ve achieved the following:

Heroku app available on a subdomain - e.g. www.joshmcarthur.com -> www-joshmcarthur-com.herokuapp.com
SSL available on both root and subdomain. SSL is forced by the Heroku app.
Root domain forwards to www subdomain - e.g. joshmcarthur.com -> 301 www.joshmcarthur.com.

Nice!

Configuring S3 redirect rules

Tue, 04 Sep 2018 00:00:00 +1200

S3 supports redirecting content in a number of ways. In most cases, providing an XML-formatted collection of redirect rules is a good way of centralising where content is going. Using the XML format of redirect rules is also a neat way of extending an S3 bucket with additional behaviour by redirecting to a Lambda function or similar dynamic generation endpoint.

Redirection rules are specified in the following basic format:

<RoutingRules>
  <RoutingRule>
    <Condition>
      <KeyPrefixEquals>KEY</KeyPrefixEquals>
    </Condition>
    <Redirect>
      <Protocol>PROTOCOL</Protocol>
      <HostName>HOSTNAME</HostName>
      <!-- This can be left as an empty element to use the current key as the path -->
      <ReplaceKeyWith>PATH</ReplaceKeyWith>
    </Redirect>
  </RoutingRule>
</RoutingRules>

Within <RoutingRules>, as many rules can be provided as you wish. There are also alternative conditions available, such as <HttpErrorCodeReturnedEquals>, which allows for a redirect if the normal S3 response results in a particular HTTP status code - redirects of this kind can be used in conjunction with an external application or FaaS platform to dynamically render or return content. Examples of this I have seen include generating image thumbnails on the fly when they do not already exist, and redirecting to a full-text search of a static site.

If you have a large number of routing rules, you can order them such that the highest-priority matches will be matched and redirected before more general rules. This allows for quite precise control over redirects.

One big gotcha with redirect rules that I found was not really documented anywhere was that the <KeyPrefixEquals> must not begin with a leading slash. This seems a bit contrary to the path that you are trying to match against, but is mostly in keeping with the fact that S3 keys cannot start with a leading slash. Keep in mind that it is the Key that S3 is matching against, not the web path.

Redirect rules cannot be used out of the box with Cloudfront, if you are using a Cloudfront distribution to act as a CDN and SSL termination for your S3 website, however with a small adjustment redirect rules work fine - when you configure the Cloudfront origin, simply provide the full URL to the S3 website location, rather than selecting your S3 bucket from the typeahead dropdown. This will cause Cloudfront to treat your S3 website as a “Custom” origin instead of an “S3” origin, so it will forward the Location header back to the client.

Test CORS headers with Rails in development

Mon, 03 Sep 2018 00:00:00 +1200

When testing CORS headers in Rails, an additional step is needed to check that the headers are working correctly. This is because of a behaviour in Chrome specifically, where requests that are to the same origin do not perform CORS validation - so it’s very difficult to tell whether CORS is working correctly or not. Chrome does not even appear to send the ‘Origin’ header for same-origin requests, meaning that CORS headers never show up in the response.

The way I found of testing this is to start an additional Rails server in development, and then set this as your asset host. Using the asset host setting means that Rails will prefix URLS to your images, stylesheets, javascripts and other assets with the asset host, rather than the current request host. Since the host and port are different, this is considered a different origin, and so triggers normal CORS behaviour.

Configure asset host:

# config/environments/development.rb
Rails.application.configure do
  # ...
  config.asset_host = "http://localhost:3001"
  # ...
end

Starting the first Rails server:

bundle exec rails s

Starting the second Rails server:

bundle exec rails s -p3001 --pid=tmp/pids/server1.pid

Then visit http://localhost:3000. If you inspect the network requests, you should see some CORS warnings (unless you already have CORS configured correctly of course!).

Pattern for testing ActiveRecord ideas in a standalone script

Sun, 19 Aug 2018 00:00:00 +1200

This code snippet is a reusable pattern for testing out a model/association structure in ActiveRecord to see whether it’s worth proceeding with. The snippet covers installing necessary dependencies, setting up an in-memory SQLite3 database, and loading a database schema into this database.

The example included in the pattern was prepared to demonstrate a has many through association between users and other users that are followed.

Logging to multiple destinations using ActiveSupport 4+

Thu, 16 Aug 2018 00:00:00 +1200

Logging to multiple destinations is something I seem to have to do all the time, and until now, I’ve never really been able to find an approach I’m happy with. Almost always, I’m writing some kind of developer-script, and I want to write messages to both the terminal (to let the dev know what is going on), and to a log file (for future reference/audit). I don’t normally want to use Rails.logger for this - I prefer to have a brand new logger specifically for my output. Having said that, I have verified that the approach described in this post does work with a standard Rails logger (defined in Rails.logger or config.logger depending on where you’re trying to work with it).

This approach relies on a method added to ActiveSupport version ~> 4.0. Versions of ActiveSupport map directly to Rails versions (ActiveSupport is part of the rails/rails repository after all), so Rails 4+ will also work here.

When working with logging in Ruby, the Logger class is normally the one is used. This is completely reasonable, as it’s part of the standard library, and is almost always sufficient. For those scenarios where something more is needed, ActiveSupport::Logger is available. The API for ActiveSupport’s Logger is broadly the same as Ruby’s Logger, with a few bonuses, such as tagged logging and the ability to extend loggers with other loggers!

Extending any logger is possible using the extend and broadcast method available on ActiveSupport::Logger. Unfortunately, I can’t link to documentation for either of these methods, as they don’t seem to be documented.

Here’s an example not using any Rails stuff, just ActiveSupport, as an example:

require "active_support/logger"
console_logger = ActiveSupport::Logger.new(STDOUT)
file_logger = ActiveSupport::Logger.new("my_log.log")
combined_logger = console_logger.extend(ActiveSupport::Logger.broadcast(file_logger))

combined_logger.debug "Debug level"
combined_logger.info "Info level"
combined_logger.error "Error level"

With the console output being:

Debug level
Info level
Error level

And mylog.log containing:

# Logfile created on 2018-08-16 15:24:10 +1200 by logger.rb/61378
Debug level
Info level
Error level

Usage in Rails really isn’t that different, the only difference is that you will already have a logger set up to log somewhere defined in Rails.logger (or config.logger if you are modifying a config file). You can just go ahead and extend that logger to add another logging destination:

# NOTE: The actual config file doesn't really matter here.
# config/application.rb
extra_logger = ActiveSupport::Logger.new("extra.log")
config.logger.extend(ActiveSupport::Logger.broadcast(extra_logger))

Multiple logging is a handy trick to keep in mind, as there’s a few places it can come in handy. As mentioned at the start of this post, my usual use case is to log to a file and STDOUT at the same time, but there’s all sorts of times you might want to post a log message multiple places. Maybe you want logs to go to a third party service, as well as local log file, as well as STDOUT for good measure? Sure!

Calculating a SHA1 fingerprint for a file in Ruby

Wed, 15 Aug 2018 00:00:00 +1200

Fingerprinting a file isn’t something I need to do super often, but it is useful to detect duplicates of files, or verify that the file has not been changed (this can be useful for more than auditing and security - for example, it’s particulalry useful for caching, which is exactly why Rails’ asset pipeline calculates hash digests for asset files when they are precompiled).

The manual approach to fingerprinting is to read the contents of the file (perform a buffered read for performance if you do this), but there’s actually a nice shortcut:

require "digest"

Digest::SHA1.file "/path/to/file"

# Example:
# fingerprint = Digest::SHA1.file Rails.root.join("public", "my-file.txt").to_s

ActiveRecord: Mark all records of type as read only

Wed, 15 Aug 2018 00:00:00 +1200

Note: This post is created based on this Stackoverflow Answer.

Occasionally, you may have a type of record that already exists in a database which you wish to read from, but never modify. In these situations, it can be worth adding a little code to mark these types of records as read-only. This will prevent most ActiveRecord data modification methods to raise an ActiveRecord::ReadOnly exception if code attempts to create or update such a record. Note that read only records do not appear to be protected from destruction!

To mark an individual record as readonly, there is an ActiveRecord::Base method available to help with this. This post concerns marking an entire type of model as readonly. Examples of when this may be necessary include integrating with an external or third-party database, migrating content from one table to another, and supporting shared or private data.

There are two approaches I have seen for marking an entire type of record as readonly, affecting every instance.

The first is the one I prefer, as it doesn’t rely on ActiveRecord callbacks:

class MyModel < ApplicationRecord
  ##
  # Mark all instances of this type of record as readonly.
  # The logic for this method can be changed if more complicated logic is required.
  def readonly?
    true
  end
end

If you prefer a simpler implementation and don’t mind callbacks, the following implementation also works:

class MyModel < ApplicationRecord
  after_initialize :readonly!
end

ActiveRecord's establish_connection

Tue, 14 Aug 2018 00:00:00 +1200

If you’ve ever wondered how your Rails ActiveRecord models end up magically translating some connection details (probably you have these in config/database.yml) into data you can interact with, establish_connection is a pretty significant part of that magic. In this post, I’m going to be laying out the different arguments this method accepts to determine how to connect to a database. Just so you know, this is largely a paraphrasing of the APIDock documentation to reinforce these arguments in my memory, so if you would prefer to directly read the documentation, go ahead.

Normal operation

Your standard Rails set up has your database connection settings listed per-environment in config/database.yml. You probably at least have settings in this file for development and test, but may have other environments there as well. The normal definition of a model (e.g. what you get if your application models inherit from ActiveRecord::Base and don’t change any settiings) is to call establish_connection, passing a symbolized version of Rails.env - :development, :test and so on.

When establish_connection is passed a symbol, it will assume that the symbol is a “configuration”. A configuration in this context is a key of the ActiveRecord::Base.configurations hash, where the value contains the actual connection settings for your database. When Rails starts, it loads and parses your config/database.yml YAML file into this hash, which is exactly how your Rails app knows how to connect to your development, test or even production database.

Passing connection information via a hash

This argument is the lowest level form of argument that can be passed in to establish_connection. A hash is passed in, containing (at least) a key specifying the ““adapter”. Any additional options are delegated to the particular adapter - MySQL, PostgreSQL, SQLite, etc - however, many of these options are common across database engines - option names like “user”, “host”, “password” and “database”. This is why, if you do ever change from one database to another, often the only key that needs to be changed is “adapter”.

This hash is accessed with indifferent access, which means that you can pass in hash keys as either symbols or strings - either is acceptable. A situation where you might use this kind of argument to establish a database connection could be a scenario where you have dynamic connection details - perhaps you have a multi tenanted application where customers can provide their own database to use, for exmaple.

An example of passing a hash to establish_connection:

ActiveRecord::Base.establish_connection(
  adapter: "postgresql"
  database: "myapp_development_1"
)

Passing connection information via a connection URI (string)

Finally, a database URL can be passed to establish_connection, which will detect the correct adapter to use from the connection string, and parse other connection details, such as the host, port, username, password and database to use from the URI. The default of establish_connection is to try and find a URI to a databse in an environment variable named "DATABASE_URL", falling back to looking up a configuration matching the current Rails environment name (as described in ‘Normal operation’).

Passing a connection string to establish_connection is quite a flexible way of establishing connections to different databases, and can be used in a number of scenarios where you might establish to different databases, either when the app starts, or during runtime.

An example of passing a connection string to establish_connection:

ActiveRecord::Base.establish_connection("postgres://postgres@localhost:5432/mydatabase")

Note: For actual applications, it would be more appropriate to store this connection string as configuration, or even as a secret, rather than hardcoding it into your script.

I wanted to blog about this particular ActiveRecord for a couple of reasons. First, it is actually a method that is very widely used, but probably not very well known. Second, while I knew of the existence of this method, and suspected some of what the method could do, I learned a lot while having a read through the documentation and implementation of establish_connection. It’s certainly one worth remembering.

Certbot renewals with Route53

Sun, 12 Aug 2018 00:00:00 +1200

This is a bit of documentation for the next time I run into this problem, however you might find it useful.

The certbot and certbot-auto commands support plugins, but do not come with many out of the box. The installation method for plugins (certbot being written in Python), is pip, however certbot may or may not pick up plugins installed in this manner, depending on the environment.

I had certbot installed in Mac OS for example, and it could not find the certbot-dns-route53 package I installed with pip. The documentation suggests that if you run into problems installing plugins, to run with Docker - however no such instructions for how to install these plugins in the official Docker images are included - An issue has been raised on github.

I do prefer to run commands that have dependencies like these in Docker anyway, so these are the steps I followed to address my specific problem. Note that at one point, I do hit this problem with rather a large hammer (mounting my entire /Users directory as a volume) - if this upsets you, feel free to find another path forward.

First, my environment:

I have all of my projects in my home directory, namespaced by repository host, user or organisation name, then the repo name. In this case, I was renewing the cert for the Pawfit Blog, so my code repository was located in: /Users/josh/Projects/gitlab.com/joshmcarthur/pawfit-blog.
I keep the letsencrypt file tree in this repo under “certs”, with things like private keys ignored (the repo is private but still). This just makes it easier to do renewals and see waht I did last time to fix certs). This file tree is what you would normally find under “/etc/letsencrypt”.
I have valid AWS credentials set up in my Mac OS home directory, inside the default “.aws” folder.
I have an expiring certificate inside the aforementioned “certs” directory.

How I renewed the certificate:

Start the docker container: docker run --rm -it --entrypoint=sh -v /Users/josh:/Users/josh certbot/cerbot. This opens a shell session inside the container. The /Users/josh volume mount was required because certbot sets up relative symlinks from the “live” certificate files to the “archive” folder. When Docker mounts the volume, these seem to be converted to absolute symlinks.
Install the Route53 plugin: pip install certbot-dns-route53. This should work outside of docker, but may not depending on how you have Python, Pip, and certbot installed (i.e. it didn’t work for me).
Symlink the AWS credentials folder from your host environment into the container’s home directory - this is so boto3 (which certbot-dns-route53 uses to connect to AWS) can resolve your AWS access keys. There are also a number of other ways you can provide boto3 with AWS credentials, such as environment variables.
Change directory to the project directory: `cd /Users/josh/Projects/gitlab.com/joshmcarthur/pawfit-blog”. See above for information about where I put code and certs, as this might be different for you.
Run certbot! certbot renew --config-dir=certs --work-dir=certs
You can now exit the container if you would like (by typing exit), or use cat to extract the information you need from the new certs. I exited at this point, as I wanted to use the Mac OS-native pbcopy command to pipe the output of cat to (e.g. cat certs/live/blog.pawfit.nz/fullchain.pem | pbcopy).

Internationalised views with Rails

Thu, 21 Jun 2018 00:00:00 +1200

Rails has great built in support for internationalisation. Generally, many people, including myself, end up focussing their efforts on internatising strings - for example, changing form labels, errors, and maybe some minor copy like headings, using I18n.t (an alias for translate), which looks up which string to return for the current locale (determined from I18n.locale, which can be set from a cookie, geo lookup, param, or whatever).

I’ve recently been working on an app that’s pretty well established in NZ, and is moving some operations to Australia. As part of this move, they have a bunch of content that needs to be worded slightly differently (and some behaviour in core models that has to work differently based on the locale, but that’s a story for another blog post). Something that often gets skipped over in internationalisation is the ability for Rails to automatically serve templates for the current locale.

The Rails Guides lay out localised views with the following description:

Let’s say you have a BooksController in your application. Your index action renders content in app/views/books/index.html.erb template. When you put a localized variant of this template: index.es.html.erb in the same directory, Rails will render content in this template, when the locale is set to :es. When the locale is set to the default locale, the generic index.html.erb view will be used. (Future Rails versions may well bring this automagic localization to assets in public, etc.)

This is certainly true, but it’s worth exploring a little more. First of all, while the example uses Español as the locale (determined by the es locale code), any locale that your Rails app is aware of will work just the same. For my example, I have two locales within my application - “en-NZ” and “en-AU”. This means that I can keep my existing templates named as they are (e.g. “app/views/books/index.html.erb”), and provide a new template containing the copy for Australia in “app/views/books/index.en-AU.html.erb”). When the locale is set to the default of NZ, the default template will be used. When the locale is set to en-AU, the matching template will take precedence and will be used instead.

The main thing to watch out for with localised views is the extent to which you are duplicating content. It’s quite simple to copy and paste a file, add the locale to the filename, and change a couple of words in the template - but this isn’t particularly maintainable if your template also has a bunch of content that does not require translation. To avoid this problem, carefully consider your template structure. If you have a section of a template that needs to be adjusted for each locale, consider refactoring that content into a partial. If the partial itself ends up particularly simple, you might then refactor to go back to a simple string lookup. Whichever approach you end up with, always try and reduce the content requiring per-locale translation to the smallest unit. This ensures that you have minimal duplication between views, while still providing all of the advantages.

Running a command on a server directly from SSH

Wed, 20 Jun 2018 00:00:00 +1200

The SSH command accepts a very useful argument that I think is underused by many devops folks looking to run a quick command and get on with their day.

The output of the synopsis of the SSH command man page (man ssh) ends with:

[user@]hostname [command]

This [command] is the useful bit - you can pop any command you like here, and SSH will connect and authenticate to the server, run the command (directing IO to your client as normal), and then exit. This makes it great for standalone tasks that can just…run, for example:

ssh user@host df -h - to show the disk usage on a server
ssh user@host ls -lah - to list the contents of the authenticating user’s home directory
ssh user@host "cd /u/app && bundle exec rake db:migrate" - to change into a Rails app working directory and run migrations

Taking little shortcuts with common commands like this can seem kind of simple, but I’ve found that the time I save really adds up. You can also make use of the command argument for scripting - for example, to automate updates or provisioning.

Extracting XML data with curl and xmllint

Tue, 19 Jun 2018 00:00:00 +1200

(I start off talking about why I was doing this. You can jump to the tip if you want.)

I use HomeAssistant heavily at home for converging a range of smart home data from sensors around my house, as well as from a range of open data sources.

HomeAssistant has a REST sensor which is good for a known data source with a friendly format, but, because of the architecture of HomeAssistant, this sensor is not particularly flexible to use when the data is coming from a third-party in a format that isn’t readily usable. For more complex data consumption, a command line sensor may be appropriate. A command line sensor captures the output of running a command as the sensor value. The trick with this is to use curl and some kind of data parsing routing to extract the data you actually want as your sensor value. That’s what I’m going over.

I used two utilities for this - curl, and xmllint. curl is preinstalled a lot these days, but if you don’t have it, it’s a simple brew install curl/apt-get install curl, etc. away. xmllint is a little harder. It’s technically either part of, or strongly related to libxml2 (it seems to be a reference or demo of how to use libxml2?). I have it on Mac OS by having libxml2 installed using Homebrew, but on Ubuntu Server I had to apt-get install libxml2-utils. YMMV, but I suspect that -utils package is required in most Linux distributions. Once you do have it though, you can take advantage of Unix pipes to pluck out the XML you need using XPath.

Here’s an example that I’ll break down:

curl "http://api.opendata.tld/example.xml" |\
  xmllint --xpath "//Measurements/Data/Reading[last()]/Value/text()" -

First of all, curl "http://api.opendata.tld/example.xml". This requests the given URI, streaming the response to the pipe. For the sake of the further examples, let’s assume it returns XML data in the following format:

<Measurements>
  <Author>Data Publisher</Author>
  <Data DateFormat="Calendar" NumItems="1">
    <Reading>
      <Time>2018-06-19T08:45:00</Time>
      <Value>11.8</Value>
    </Reading>
    <Reading>
      <Time>2018-06-19T09:00:00</Time>
      <Value>11.8</Value>
    </Reading>
    <Reading>
      <Time>2018-06-19T09:15:00</Time>
      <Value>11.8</Value>
    </Reading>
  </Data>
</Measurements>

The second part of this command is the pipe to xmllint. xmllint is passed two key arguments. The first, --xpath, contains an XPath expression for waht to select, and the trailing dash (-) prompts xmllint to accept XML from STDIN (the pipe from curl).

Given a valid XPath expression, the result of this command will be exactly what is required - a value to assign to the sensor. In this case, the XPath expression will be interpreted as:

Start at the root Measurements node
Traverse down the tree into Data
Select the last() Reading node
Traverse into the Value node
Select the text() (inner text) of the node

In other words, the complete command:

curl "http://api.opendata.tld/example.xml" |\
  xmllint --xpath "//Measurements/Data/Reading[last()]/Value/text()" -

Will return…..11.8. Exactly what we need!

We can now define our HomeAssistant sensor:

sensor:
  - platform: command_line
    command: "curl \"http://api.opendata.tld/example.xml\" | xmllint --xpath \"//Measurements/Data/Reading[last()]/Value/text()\" -"
    unit_of_measurement: "C"

Migrate on push with Heroku releases

Thu, 31 May 2018 00:00:00 +1200

I use Heroku for most of my hosting, and also use it a lot in my work at Rabid Technologies. Something I frequently run into is a migration not being run after a git push heroku master, often causing an embarrasing application error page until someone can get to running heroku run rake db:migrate.

Fortunately, Heroku supports release commands. These commands are run after each successful deployment, and also delay the restart of the dyno(s). This allows for any post-code-update steps to be taken before the application is updated for users.

Declaring release commands is very simple - a line simply needs to be added to the Procfile stating which commands should be run upon release. For example, this is the line I use to run migrations:

# Procfile
# ...
# release: bundle exec rake db:migrate

Adding this to your Procfile will change your post-receive hook output from Heroku slightly, as it will now output the status of the release command:

remote: -----> Launching...        
remote:  !     Release command declared: this new release will not be available until the command succeeds.        
remote:        Released v160        
remote:        https://myapp.herokuapp.com/ deployed to Heroku        
remote: 
remote: Verifying deploy... done.        
remote: Running release command....
remote: 
remote:    (0.6ms)  SELECT pg_try_advisory_lock(7931432920140052365)        
remote:    (1.2ms)  SELECT "schema_migrations"."version" FROM "schema_migrations" ORDER BY "schema_migrations"."version" ASC        
remote:   ActiveRecord::InternalMetadata Load (0.7ms)  SELECT  "ar_internal_metadata".* FROM "ar_internal_metadata" WHERE "ar_internal_metadata"."key" = $1 LIMIT $2  [["key", "environment"], ["LIMIT", 1]]        
remote:    (0.7ms)  BEGIN        
remote:    (0.5ms)  COMMIT        
remote:    (0.9ms)  SELECT pg_advisory_unlock(7931432920140052365)        
remote: Waiting for release.... done. 

I find that this functionality is particularly useful for continuous deployment setup, as it only requires git to be installed to perform a Heroku deployment AND automatically run migrations. Without this functionality, it is slightly harder, as the deploy needs to happen in two steps - one to do the git push, and then another to do the heroku run. This process also requires the Heroku CLI to be installed and authenticated, which isn’t always the easiest thing in CI environments.

Simple Ruby API clients with HTTParty

Tue, 29 May 2018 00:00:00 +1200

I frequently write code that interacts with a third-party HTTP API. When I create this code, I find the best approach is to create a very thin abstraction layer over the API that can easily be stubbed out under test. If the data returned from the API is complex or requires further processing, I occasionally provide additional abstraction layers on top of the HTTP client, but frequently, just having a consistent API client is enough.

In terms of creating this client, there are many, many options in Ruby, ranging from a relatively complicated approach by using the Ruby standard library net/http module, up to a series of gems that either abstract this API or provide another HTTP interface altogether. An example of a gem that abstracts across the HTTP driver, and the one I’m going to be discussing today is httparty.

Despite having a stupid post-install message that can’t (easily) be turned off, HTTParty provides a very easy way of defining an API client.

Here’s an example of an API client I put together recently:

require "httparty"

class MyAwesomeAPI
  include HTTParty
  format :json

  def initialize(username, api_key)
    @auth = { username: username, password: api_key }
    self.class.base_uri("https://api.myawesomeapi.com")
  end

  def get(path, options = {})
    options = default_request_options.merge(options)
    self.class.get(path, options)
  end

  private

  def default_request_options
    { 
      headers: { "Accept" => "application/vnd.myawesomeapi.v3+json" }, 
      basic_auth: @auth 
    }
  end
end

This API client:

Will automatically serialize and deserialize requests using JSON encoding
Defines the base URI in a single place
Adds basic authentication to the request
Adds an outgoing header to each request defining the mimetype of content it will accept - in this case, a versioned custom type.

In the case of this API, I only required GET access to resources, so that’s all I’ve defined. It’s entirely possible to implement other HTTP methods in a very similar manner, or even to provide further abstractions, such as methods named after the API endpoint in question. As an example, if this API allowed for a listing of “widgets”, an all_widgets method could be defined that returned an array of Widget instances created from the JSON response.

HTTParty has many options for customizing how the request and response will be handled, but at the end of the day, what it provides most of all is a simple, abstract, class that allows for an API interface to be quickly defined that will Just Work.

Git commit message templates

Mon, 28 May 2018 00:00:00 +1200

Writing high-quality git commit messages is one of the most responsible things you can do as a developer who cares to write considerate code that is well thought-out and easy to maintain. There’s a great template for git commit messages that’s floating around in a number of places. It’s referred to the “Contributing to a Project” chapter of the Git Book and originally credited to Tim Pope. In short, a good template of a commit message is a short, one-liner summary of the code change, followed by the ‘body’ of the commit. The body of the commit may contain whichever links, bullet point or other emphasis required to communicate to the commit message reader the full context of the change.

While this is a good day-to-day commit message template to follow, there are some situations which might require additional contextual prompts to result in all the necessary information being communicated. Examples of such scenarios include:

A commit resolving a bug ticket, where consistently communicating the ticket number, the URL of the ticket, and the requester.
A commit layout out a particular architectural decision in a clear format, such as the model layed out in Sustainable Architectural Design Decisions.
A commit implementing an agile story, where the developer may want to lay out the role and motivation of the feature.

While it’s entirely posible to write these git commits ad-hoc as required, an additional option I discovered today was the ability to customize which template file is loaded when committing. As laid out on the git-commit man page, this option is:

-t <file>
--template=<file>
  When editing the commit message, start the editor with the
  contents in the given file.The commit.template configuration
  variable is often used to give this option implicitly to the
  command. This mechanism can be used by projects that want to
  guide participants with some hints on what to write in the
  message in what order. If the user exits the editor without
  editing the message, the commit is aborted. This has no effect
  when a message is given by other means,
  e.g. with the -m or -F options.

The trick is to combine this option with the ability to set up Git aliases to make it easy to start commits using a particular template, for example:

git config --global alias.bugfix-commit 'commit --template=~/dotfiles/git/templates/bugfix-commit.txt'

Given a template that exists in that location, your $EDITOR will open with the contents of the template prefilled. Any lines beginning with a pound # can be used to provide instructions to the committer, and will be ignored from the final commit message.

As a final example, this is how I have set up a template for committing at key decision points:

My template in ~/.dotfiles/git/templates/commit-decision.txt:

# Write a useful commit message here:

# Now provide more information about why you've made this
# decision:
In the context of [USE CASE],
facing [CONCERN]
we decided for [OPTION]
to achieve [QUALITY],
accepting [DOWNSIDES].

My git alias:

git config --global alias.commit-decision 'commit --template=~/.dotfiles/git/templates/commit-decision.txt'

Usage:

URI.regexp in Ruby

Wed, 23 May 2018 00:00:00 +1200

Recognizing or validating URIs in a string or strings of text is a fairly common problem. Unfortunately, just as common is the range of regular expressions that exist to validate URIs (or, generally, URLs). There are many, many StackOverflow answers and blog posts that lay out a massive variant of expressions in function and quality, and it’s not the easist form to understand.

Fortunately, the Ruby programming language comes with a very capable regexp for parsing URIs - it’s just a matter of knowing where to look. Where to look, in this case, is the URI module. URI.regexp will match most valid URIs, using =~, scan, gsub, or any other string method that deals with regexp matching. Additionally, an array of schemes can be passed into the regexp method, which will restrict the returned regexp to match these schemes only. This regexp is particularly useful as the capture groups that are returned include the parsed components of the URI, allowing for very precise control of how the string is processed from there.

Examples:

irb(main):001:0> URI.regexp.match("https://joshmcarthur.com/posts?page=2#top").to_a
=> ["https://joshmcarthur.com/posts?page=2#top", "https", nil, nil, "joshmcarthur.com", nil, nil, "/posts", "page=2", "top"]

irb(main):009:0> URI.regexp(["ftp"].match("https://joshmcarthur.com/posts?page=2#top").to_a
=> []

TIL: Begin/rescue/end assignment in Ruby

Tue, 22 May 2018 00:00:00 +1200

Today, while working on consuming a third-party API, I discovered that the result of a begin/rescue/end block in Ruby can assign to a variable, just like an if statement can.

This isn’t a technique I’d recommend for every single situation where it could be used, but it is convenient for operations where a particular error scenario can result in a single return value.

Here’s an example of reading a remote file and assigning it to an object. If the file operation fails (in this case, if the HTTP request fails with anything other than a 2xx or 3xx status), a fallback value is assigned instead:

require "ostruct"
require "open-uri"

model = OpenStruct.new
model.title = begin
  open("https://gist.githubusercontent.com/joshmcarthur/69cffbb395fc3781e23b56d9281f6f73/raw/7b31ea464799077cddd79cfe22105d7364497cdd/title.txt")
rescue OpenURI::HTTPError
  "Default Title"
end

puts model.title
# => This is a title from a remote file

TIL how to fork a Heroku application

Thu, 17 May 2018 00:00:00 +1200

Tonight I had a need to create a duplicate of a running app on Heroku, and discovered heroku-fork. This is a plugin which was extracted from the core Heroku CLI some time ago, but is easily installable using the command heroku plugins:install heroku-fork. It is run via:

heroku fork --from [app name] --to [app name, can be new app]

The command adds functionality which can copy one app to another, including
code, addons, dyno configuration, and databases. In my case, I used it to quickly test some changes to a service worker against a clone of a production application before pushing these changes live. It no more than a couple of minutes to copy the application, and then I simply destroyed it when I was done.

A super handy trick that I’ll have to keep in mind for future support work.

TIL how to italicize comments in VS Code

Tue, 15 May 2018 00:00:00 +1200

Continuing my transformation of my development environment to match Wes Bos’, today I discovered from a Stack Overflow answer how to change how comments in any language are rendered in VS Code.

It turns out that there is a setting to control it, it’s just deeply nested. The necessary snippet for user settings is:

"editor.tokenColorCustomizations": {
    "textMateRules": [
        {
            "scope": "comment",
            "settings": {
                "fontStyle": "italic"
            }
        }
    ]
}

(You can access user settings under “Code > Preferences > Settings” on Mac).

It turns out that VS Code is yet another editor to support themes and colour highlighting rules in TextMate format. Once we’re into the correct setting context, it’s just a case of targeting the "comment" scope and instructing the editor to render comments in italic. This should work for a bunch of other style properties as well, if there are other adjustments you’d like to make.

TIL about console.table

Fri, 04 May 2018 00:00:00 +1200

I’m working through Wes Bos’ ES6 for Everyone, and he’s dropped yet another useful tip for me to build into my day-to-day work - console.table.

console.table supports both objects and arrays, displaying data in a table format that is much easier to read than simply logging the data:

console.table(user)

console.table(repos)

To restrict which columns are shown for a collection of objects, a second argument can be passed to restrict which property names are shown:

console.table(repos, ["name", "description"])

console.table appears to be supported in all modern browsers, so it’s ready to go for any development workflow!

Rails console --sandbox

Thu, 03 May 2018 00:00:00 +1200

Occasionally, a particular bug or customer query will necessitate jumping into a Rails console connected to a live database, or a copy of the live database, that we wish to read from, but not change.

I was reminded today of an option that can be passed to the rails console command, -s, or --sandbox. This wraps the entire console command in a database transaction that will be rolled back when the console is terminated.

The implementation of the --sandbox option is actually very simple. When the console starts, begin_transaction is called on the ActiveRecord::Base connection. A hook is set up to trap at_exit, which will rollback any changes made within the sandbox. Here’s the implementation:

ActiveRecord::Base.connection.begin_transaction(joinable: false)

at_exit do
  ActiveRecord::Base.connection.rollback_transaction
end

(source)

Effectively, this means that when a rails console --sandbox command is run, records can be added, removed, changed (and anything else that is processed inside a database transaction can be done), without it affecting any other part of the Rails application.

What I learned from Command Line Power User

Wed, 02 May 2018 00:00:00 +1200

My tech education has been almost exclusively Wes Bos based at the moment - I’ve finished React for Beginners, am halfway through his ES6 course, have Redux queued up, and am keeping an eye on his progress with his advanced React course. I’m also a listener of the podcast he produces along with Scott Tolinski, Syntax.

As a short break from ES6, I powered through a slightly older course, Command Line Power User. Because of how long I’ve been working in a shell, most of the lessons Wes had to share I had already learned the hard way - but I did pick up a couple of tips that I’m posting about today.

1. `ls -m`

This flag can be used to change the format of the directory listing from a column-based view to instead of be a comma-separated list of files and directories. I can see this being useful for passing into scripts in the future, or at least providing a list of directory contents without all the surrounding formatting and whitespace.

This flag was shown as a demonstration of ZSH’s argument tab-completion, and it looks like there’s a bunch of other arguments I’ll have to check out in more depth.

2. `trash`

I think everyone’s had a little rm whoopsie before. I know I have. One other tip that came up was having a command to send files and directories to a trash folder, rather than directly removing them. Seems obvious now, but I’d never thought of doing that. Wes even suggests aliasing rm to a trash command, which is a great idea.

It looks as though there is an NPM package for a trash command. I’m going to do a bit more investigation here, as, since I work exclusively on my Macbook, I might be able to find something more lightweight than a Node.js library.

3. The `z` command

The z command is a ZSH script that can be dropped in to provide “frecent” jumping to directories (frecent being “frequent” and/or “recent” directories). This looked interesting, and I might try it out sometime, but I’ve been using autojump for some time, and it seems to have a similar feature set - so I don’t really have a reason to switch, other than z being written purely in ZSH scripting language (looks like autojump is Python).

4. The `take` command

I’ve got to try and build this one into my muscle memory, because I would use it all the time.

take is a command built into zsh.. It’s literally:

take () {
	mkdir -p $1
	cd $1
}

…in other words, make a directory (or directory tree), and immediately change into it. Fantastic!

5. The `extract` plugin

extract is a plugin available as part of oh-my-zsh. It simply acts as an abstraction over the many, many archiving formats available, delegating to the appropriate utility (tar, ar, zip, etc). I’ve run into frustration before trying to get the right set of utilities and arguments to extract an archive, so I’ve also added this to my oh-my-zsh plugins!

Those were my takeaways, but the whole course was really interesting. I was pleased to see how well my current setup aligns with the productive setup demonstrated in the course - I’m also a ZSH user, though I only use a fraction of it’s features. I jumped on the oh-my-zsh bandwagon when it was released - mostly for the pretty themes of course, and eventually even ended up creating my own small set of plugins and a theme. The only difference from the course content was that I have begun using antigen to control how much of oh-my-zsh is loaded when my ~/.zshrc is sourced into my shell, as I found the default oh-my-zsh install method was getting a little slow to initialize.

TIL: Debugging Net::HTTP

Wed, 02 May 2018 00:00:00 +1200

Today I had to debug a library to try and determine why a particular HTTP request was failing. The problem was, this particular library uses Net::HTTP, without any particular hooks to customise how the request will be executed.

I discovered the following handy code snippet at https://gist.github.com/ahoward/736721, which forces debug output to be on for any instance of Net::HTTP created. This snippet can be pasted into an initializer or even an IRB/pry debugging session to enable debug logging of Net::HTTP to STDERR:

BEGIN {

  require 'net/http'

  Net::HTTP.module_eval do
    alias_method '__initialize__', 'initialize'

    def initialize(*args,&block)
      __initialize__(*args, &block)
    ensure
      @debug_output = $stderr ### if ENV['HTTP_DEBUG']
    end
  end

}

I wouldn’t recommend this code for any deployed environment, since it’s monkeypatching a pretty core Ruby class. It’s a great debugging tool though, if you’re not too sure how to get to a particular HTTP request.

TIL: Cherry picking without a commit

Mon, 23 Apr 2018 00:00:00 +1200

Cherry-picking git commits isn’t part of my day-to-day workflow, but I do need to do it occasionally. Generally, and in the case I ran into today, I had a range of commits that required cherry-picking from master onto a long-lived feature branch.

Normally cherry-picking in git will take each commit and “copy” it onto the current branch. This is generally fine, but in the case where you might be cherry-picking a larger range of commits, or you find that the message against the commit being cherry-picked isn’t adequately descriptive of the commit in the context of the branch, the -n flag can come in handy.

The -n flag is described in the manual:

-n, –no-commit Usually the command automatically creates a sequence of commits. This flag applies the changes necessary to cherry-pick each named commit to your working tree and the index, without making any commit.

In other words, using git cherry-pick -n will cherry-pick a commit across into the branch, without committing. This allows for the changes to be committed differently - maybe you want to ‘squash’ all of the changes into a single commit, or rephrase one or more of the commit messages - all of this is quite simple with a no commit cherry-pick, since while the changes are applied by git, none of the changes are recorded as actual commits - how the committing is done is entirely up to you.

HTML5 hidden attribute on option tags

Sun, 15 Apr 2018 00:00:00 +1200

This morning I learned about the hidden attribute that can be added to <option> tags within a <select> tag. The hidden attribute causes the option to simply not be shown in the list. Hidden elements will are also not selectable and will not return any selected value.

The hidden attribute works in most browsers, with ~97.5% of browsers supported. For IE10 and below (depending on the browser breakdown of your application), an acceptable fallback would be to also add the disabled attribute to the option. The browser in this case would still show the option, but it would be greyed out and not selectable.

As an example, this HTML:

<select>
  <option>First option</option>
  <option>Second option</option>
  <option hidden disabled>Third option</option>
</select>

Yields the following result:

And in IE10:

TIL: ActionMailer Preview

Thu, 12 Apr 2018 00:00:00 +1200

It’s been a while since I’ve worked with transactional email directly with Rails. This is because I’ve previously been working on some larger projects that have justified the use of an external mailing system like Mandrill, where email templates were used to generate email within that system, and sending an email from the Rails app was just an API call.

Recently, I’ve been working on applications for smaller businesses, and that’s a problem I approach quite differently. Smaller businesses really need something that will work, and continue to work, with very little ongoing investment. Because of this, when I’ve got the need to send transactional email for a client like this, I go back to the bedrock - SMTP and ActionMailer.

Because I haven’t worked with ActionMailer for some time, I seem to have missed a fantastic feature that I just learned about today. It’s even in the Guides, so I’m annoyed at myself for missing it!

It turns out, since Rails 4.1, there is built-in support for rendering either an HTML or plain-text version of an ActionMailer template, right in a web browser. This allows for easy iterative development and testing, and since the preview follows exactly the same code paths that sending does (up to actual delivery of course!), you can see the final product in the browser - with inlined CSS, rewritten markup, and any other enhancements you’ve included.

Writing a preview is very easy - so easy in fact that Rails will do it for you by default with rails generate mailer:

=> rails generate mailer user
      create  app/mailers/user_mailer.rb
      invoke  erb
      create    app/views/user_mailer
      create    app/views/layouts/mailer.text.erb
      create    app/views/layouts/mailer.html.erb
      invoke  rspec
      create    spec/mailers/user_spec.rb
      create    spec/mailers/previews/user_preview.rb

The preview class is very simple - one or more methods can be defined, with each method expected to invoke a corresponding mailer method. The preview class can take whatever steps it needs to to set up the data required to prepare a particular email, so long as the Mail object returned from a mailer call is returned from the method.

Here’s an example of a preview class:

class UserMailerPreview < ActionMailer::Preview
  def confirmation
    UserMailer.confirmation(FactoryBot.build(:user))
  end

  def password_reset
    UserMailer.password_reset(FactoryBot.build(:user))
  end
end

Rails creates a simple navigation structure for previews - going to http://localhost:3000/rails/mailers will serve an index of all the available preview classes, and then going to a particular preview name (in the example above, http://localhost:3000/rails/mailers/user) will serve an index of all the mailer methods available for testing:

Clicking on a particular mailer method will present the email within the web browser, showing the email metadata (recipient, subject, bcc etc) at the top of the page, with the HTML or plain text version selectable to show in the bottom. Updating the mailer template or mailer itself will be autoloaded as usual, so the mailer preview can just be refreshed to see the latest change.

This is a great feature built into Rails, and certainly something I’ve tried to emulate before with tools such as Mailcatcher and letter_opener. Finding little pieces of functionality built into the framework like this is always great, and continues to encourage me to keep an eye on the changelog!

TIL: toLocaleString() for currency formatting

Wed, 11 Apr 2018 00:00:00 +1200

Yesterday I learned about toLocaleString, a Javascript function defined on the Number prototype since ECMAScript 3. It has had a couple of iterations of arguments, but the version supported by most browsers provides a very convenient API for formatting currency in a user’s locale:

Examples:

Note: Wherever I have passed undefined as the first argument to toLocaleString, this simply means that I am expressing no preference for a particular locale. In this case, the browser will use the system locale.

Format an amount in USD using the user’s locale (en-NZ for me)

var number = 1234567
console.log(
  number.toLocaleString(undefined, { 
    style: "currency", 
    currency: "USD"
  })
);
// => US$1,234.57

Format an amount in USD using a defined locale

var number = 1234.567
console.log(
  number.toLocaleString("en-US", { 
    style: "currency", 
    currency: "USD"
  })
);
// => $1,234.57

Format an amount in NZD using a defined locale

var number = 1234.567
console.log(
  number.toLocaleString("en-US", { 
    style: "currency", 
    currency: "NZD"
  })
);
// => NZD$1,234.57

Format an amount in NZD using European formatting

var number = 1234.567
console.log(
  number.toLocaleString("de-DE", { 
    style: "currency",
    currency: "NZD"
  })
)
// => NZD$1.234,57

Format an amount in Pounds, rounded to a whole number

var number = 1234.567
console.log(
  number.toLocaleString(undefined, { 
    style: "currency", 
    currency: "GBP", 
    minimumFractionDigits: 0, 
    maximumFractionDigits: 0
  })
);
// => £1,235

Format an amount in NZD, using the default locale (en-NZ for me), rounding to a whole number when possible and otherwise using 3dp.

var roundNumber = 1234.0;
var fractionalNumber = 1234.5678;
var options = { 
  style: "currency", 
  currency: "NZD", 
  minimumFractionDigits: 0, 
  maximumFractionDigits: 3
};

console.log(roundNumber.toLocaleString(undefined, options))
// => $1,234

console.log(fractionalNumber.toLocaleString(undefined, options));
// => $1,234.568

Browser support

According to the MDN documentation for toLocaleString, browser support is good across the board for “Basic support” (this is formatting a number for a particular locale, but not in a particular style, e.g. “currency”). Support for locales and options is good for all mainstream browsers.

While MDN’s compatibility chart indicates no support for locales and options on Android, my testing showed that it is in fact supported in Chrome running Android 6.0.1, and I suspect down to ~ Android 5 when the system webview was replaced by Chrome.

For those looking to work in browsers that do not support non-Basic usage of toLocaleString, the MDN page has feature detection functions that can be used to fallback to a different implementation. I created a fallback called formatCurrency, which used the toLocaleString function directly if possible, falling back to concatenating the amount and currency.

Accessing selected element in Chrome DevTools

Tue, 10 Apr 2018 00:00:00 +1200

Chrome’s DevTools are something that I use for a decent portion of my day whenever I’m doing frontend work. I really appreciate how much work has gone into making for a great developer experience, and I’m always stumbing across new tricks to help make me more efficient.

Just one of these tricks I picked up this morning from Wes Bos’ “React for Beginners” course in passing, and it’s been a bit of a revelation for me.

When an element is selected in the Elements tab of the devtools, you may have noticed it displays something like this:

<div class="innerWrapper clearfix">…</div> == $0

I’ve always seen the == $0 and kind of skipped over it, assuming it’s some kind of notation that I’m just not familiar with.

It turns out though, that this is Chrome communicating that the currently selected DOM element has been assigned to a variable named $0. This variable can be used straight from the Console for looking up more detailed information, or even manipulating the DOM node in some way.

Just a really nice trick that is going to make my life that much easier next time I need to drill down into an element.

Accessing main_app routes from a Rails engine's templates

Mon, 09 Apr 2018 00:00:00 +1200

If you are working with a template that has been rendered by a Rails engine (the example I most commonly have is refinery, but any mounted Rails engine behaves the same way), then you will frequently run into a strange problem where route helpers (such as widgets_path, new_whatever_path, etc), result in an undefined method error, which can be quite frustrating to track down if you’ve not run into it before.

This errors occurs because the view context that the template is being rendered in is the engine’s context, not the main application - long story short, the template is only looking at the engine’s routes for route helper methods, not the app’s routes.

The trick to get this to work is to prefix your route helpers with main_app. With the examples I gave just above, this means that the route helper method names would be main_app.widgets_path and main_app.new_whatever_path. This prompts Rails to look in the correct application namespace for the route helpers, which it can then resolve.

Running Bitbucket Pipelines builds with docker compose

Fri, 06 Apr 2018 00:00:00 +1200

As a companion to [my post describing how to run tests with docker-compose on Gitlab CI], I converted the build steps this morning to work with Bitbucket Pipelines.

Here’s the adapted bitbucket-pipelines.yml file:

image: docker:stable

pipelines:
  default:
    - step:
        services:
          - docker
        script: 
          - apk add --no-cache py-pip bash 
          - pip install --no-cache-dir docker-compose
          - docker-compose -v
          - docker-compose run -e RAILS_ENV=test app bin/ci-setup
          - docker-compose run -e RAILS_ENV=test app bin/ci-run

Only minor differences from the Gitlab CI, mostly relating to the base image being used:

docker:stable is used as the base image. This is a nice small image for running DinD (Docker-in-Docker), and is (unsuprisingly) based on the latest stable release of Docker.
We declare that we need docker to be running for this build.
Our script installs docker-compose using pip, and then runs the CI commands.

There is a small performance penalty for this method, since docker-compose needs to build an image from scratch each time (unless your docker-compose.yml declares a complete image rather than a build instruction), however I’ve found this time to be comparable to running in a traditional, non-Docker CI environment.

SASS Colour Functions

Wed, 04 Apr 2018 00:00:00 +1200

I’ve been implementing a style guide this week, and in particular have been trying to focus on creating clean, decoupled, and reusable components.

Part of this effort has involved ensuring that all “global” settings that are represented in the style guide are contained in a single settings file. A perfect example of one of these settings is the colour palette.

In order to try and avoid defining too many colours unnecessarily, I’ve been working a lot with SASS colour functions, and wanted to talk about those today.

scale_color

Documentation

This function is great for ‘tinting’ a colour. For example, if you have a palette colour defined, say $green: #00ff00;, you can lighten it with scale_color($green, $lightness: 15%) to make the colour 15% lighter. scale_color has some minor details relating to how it changes the lightness in a relative sense - for a more absolute measurement, then there is adjust_color.

rgba

Documentation

rgba is great for introducing an alpha value to a palette colour. I’ve been using this a lot for things like button states, where the background colour should be 85% opaque. It’s very simple to use, as it can accept either red, green, blue and alpha arguments, or a Sass “color” type and alpha arguments. A Sass color type can be basically any representation of a color - named colours (like “green”), and hex values (like “#00ff00”) are most common. Variables are also supported.

As an example of introducing alpha to a colour value, given a color palette variable of $green: #00ff00;", RGBA can be used to change the alpha of the color to 85% as follows: rgba($green, 0.85).

These functions are incredibly useful for enabling stylesheets to stick to a small set of core variables, and simply adjust these, rather than adding variables for every possible variant of a colour. There are many, many more utility functions documented on the sass-lang site, and I’m going to be spending a lot more time investigating these more to further improve my componentized styles.

Running Gitlab CI tests with docker compose

Mon, 02 Apr 2018 00:00:00 +1200

I’ve been working for some time to get Pawfit (my pet project, literally), continuously deploying. I’ve only recently added tests, so now that I have them, I want a green check each time I push.

I run just about all my Rails apps now using Docker and docker-compose, almost always using my own base Docker* files. I choose not to push these files to a repository like Docker Hub, because I don’t consider them usable on their own - their intention is to provide a starting point to grow from, rather than an out-of-the-box solution.

Since this project is a small project and is running on a constrained budget (read: my own funds), I’ve constantly been looking for ways to optimise for cost. Along these lines, I’ve started pushing to Gitlab, mostly to make the most of their excellent CI product. I’ve been having quite a few problems getting my tests running though, as I want to use my own Dockerfile to have a consistent build - I don’t want to have my Dockerfile and then a bunch of duplicated set up steps in my .gitlab-ci.yml. Gitlab supports Docker-in-Docker, which instantly puts it up there in terms of CI provider, but I still had a few issues. These issues were almost exclusively related to being unable to resolve linked container hostnames from my ‘app’ container - i.e. my ‘app’ container could be built and run, but could not see or communicate to port 5432 on my ‘db’ container.

In the end, I gave up on a pure Docker solution to CI, and went with docker-compose, which is what I use for development. Whatever problems plain Docker commands were causing me, docker-compose seemed able to overcome, and I now have my builds running nicely. A future refactoring will be to make use of Gitlab’s built-in Docker image repository to keep track of app images each time I push, eventually moving to container based deployment.

Anyway, here is the Gitlab CI YML file I ended up with:

test:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKER_DRIVER: overlay2
  before_script:
    - apk add --no-cache py-pip
    - pip install docker-compose
  script:
    - docker-compose build
    - docker-compose run -e RAILS_ENV=test app rake db:test:prepare test test:system
production:
  stage: deploy
  script:
  - gem install dpl
  - dpl --provider=heroku --app=pawfit --api-key=$HEROKU_PRODUCTION_API_KEY
  only:
  - master

What this does:

Defines a test job, and production job. I can disregard the production job, as it just exists to auto-deploy my app to Heroku.
The test job uses the latest Docker as a base image, and tells Gitlab to use it’s “Docker-in-Docker” service (docker:dind). This sets up appropriate services and environment variables to be able to use Docker from within their Docker-based job runners.
Sets overlay2 as the Docker driver. I’m told this is faster, your job should work without this.
Before the job script runs, docker-compose is installed using pip.
The script runs - this builds the image, then runs preparation for the tests, the unit tests, and the system tests (using headless Chrome).

TIL: Mocking with Minitest in Ruby 2.0+

Wed, 28 Mar 2018 00:00:00 +1300

I normally work within an RSpec testing envionment, but I definitely enjoy working with more traditional Test::Unit tests when I get the chance.

Today I discovered that Minitest, the testing toolkit that has been part of the Ruby standard library since 2.0 has built-in support for mocks and stubs.

To use this in a Test::Unit test, simply require "minitest/mocks".

Mocking

It looks like a mock object has to be specifically made. Minitest/Mocks does not include double support - you’ll need something like Mocha for that. The method signature on Minitest::Mock is: expect(:method_name, return_value, [args]).

Here’s an example:

message_sender = Minitest::Mock.new
message_sender.expect(send_message, true, [{email: "test@example.com", message: "Test"}])
result = message_sender.send_message(email: "test@example.com", message: "Test")
assert result, "Message sends"
message_sender.verify

A major difference from Rspec mocks to be aware of is that the verify method must be called on the mock object to actually check that the expected methods were returned.

Stubbing

Minitest also includes basic support for stubs. The method signature for stubs is: ObjectUnderTest.stub(:method_name, stub_value, &block). Minitest does not have an unstub method - instead, it is expected that the test code that requires the stub should go inside the block passed to stub, and minitest will automatically unstub when the block completes.

Example:

message_sender = MessageSender.new
message_sender.stub(:perform_deliveries?, true) do
  message_sneder.send_message(email: "test@example.com", message: "Test")
end

TIL: let! and before order is important

Wed, 28 Mar 2018 00:00:00 +1300

I’ve just come across this issue. It’s minor, but it tripped me up!

In RSpec, there are three methods I find myself using all the time:

before - this is equivalent to the setup method present in many assertion-based test frameworks, and runs before each test.
let - this is a lazy-evaluated variable that will be made available to the context of each test - BUT it is undefined until called for the first time.
let! - this is the “evaluate right away” version of let. It will run before each example and already be available at that point.

Normally, in a feature or integration test that require data to be present, there will be some standard code:

RSpec.describe "My cool feature", type: :system do
  let!(:widget) { FactoryBot.create(:widget) }
  before { visit "/widgets" }

  it "shows the widget" do
    expect(page).to have_content widget.name
  end
end

This is normally the pattern that I follow. What I found is that, especially for let! blocks where you want the data to be present in the database before the route is visited, the order of the let! and before matters.

In the below example the test fails, as the let! blocks actually run as they are evaluated, as in, after the before block:

RSpec.describe "My cool feature", type: :system do
  before { visit "/widgets" }
  let!(:widget) { FactoryBot.create(:widget) }

  it "shows the widget" do
    expect(page).to have_content widget.name
  end
end

Failure: Unable to to find content: "My widget"

There is an interesting issue lodged against rspec-core that has a bit more background as to why this happens, as well as some suggested workarounds. In my case, I just corrected the order of my before and let! calls to make sure the before block came after the let!s.

TIL: Nested layouts in Rails

Mon, 26 Mar 2018 00:00:00 +1300

Nested layouts in Rails aren’t a technique that I’ve had to use a lot in the past, but I’m finding myself using them quite a bit in my current work.

Nesting layouts allows for some sharing of a common view structure. The best example of this that I have come across is in the app I am working on. This app is about 50/50 Refinery CMS and normal Rails app. The Refinery pages are of several ‘types’ of content, but are all presented within the same container structure. The normal Rails app views are presented in a higher-level layout that has a little less markup surrounding it.

Nesting layouts is actually quite easy. It uses the content_for method to declare content for a particular named block, and then render the layout that you wish to use.

Here’s an example:

# app/views/layouts/application.html.erb
<html>
  <head>
    <title>My Rails App</title>
    <!-- Head content -->
  </head>

  <body>
    <nav>
      <h1>My navigation</h1>
    </nav>
    <%= content_for?(:content) ? yield(:content) : yield %>
    <footer>
      <small>Thanks for visiting</small>
    </footer>
  </body>
</html>

So, that’s the normal application layout. The only difference is that the normal yield has a ternary operator wrapped around it to render content for content, if it is set. This is important to let our nested layout provide content to the application layout.

Our nested layout looks like this:

# app/views/layouts/content_page.html.erb
<% content_for :content do %>
  <h2>This is a content page</h2>
  <%= yield %>
<% end %>
<%= render template: "layouts/application" %>

This template provides content for the block named :content, and then renders the application layout. The app layout, seeing content in the :content block, renders this, rather than the implicit yield.

Provided a controller declared the correct layout, e.g.

# app/controllers/pages_controller.rb
class PagesController < ApplicationController
  layout "content_page"
end

Then a pages controller view will be rendered inside the content page layout, resulting in a compiled ERB template of:

<html>
  <head>
    <title>My Rails App</title>
    <!-- Head content -->
  </head>

  <body>
    <nav>
      <h1>My navigation</h1>
    </nav>
    <h2>This is a content page</h2>
    <%= yield %>
    <footer>
      <small>Thanks for visiting</small>
    </footer>
  </body>
</html>

TIL: Handling browser dialogs with headless Chrome in Capybara

Mon, 26 Mar 2018 00:00:00 +1300

My normal Rails 5.x development stack now uses Chrome, running headlessly, powered by the following gems:

# Gemfile
# ...
gem 'capybara'
gem 'chromedriver-helper'
gem 'capybara-selenium'

Unlike using things like Poltergeist, the only external dependency you need to run these tests is Chrome itself. If you wish to reduce your gem dependencies, you can also remove chromedriver-helper, but this will require that you get chromedriver set up.

One of the behaviours I’ve recently wanted to test is interactions with browser dialogs. This is particularly common when using Rails’ UJS helpers, such as:

<%= link_to "Cancel", "#", data: { confirm: "Are you sure?" } %>

And straight Javascript such as:

var projectName = "my-project"
var shouldCancel = (prompt("Enter your project name to cancel") === projectName);

Interacting with these kinds of browser interfaces can be hard from an abstraction such as Capybara. In fact, because each browser can present these dialogs slightly differently, the method used depends on the browser. The rest of this post assumes you’re using Chrome as your driver, but if you’re just after a summary, I found this Gist to be super useful.

Accepting a prompt - i.e. prompt(“Are you sure?”);

confirm = page.driver.browser.switch_to.alert
assert_equal confirm.text, "Are you sure?
confirm.accept # You can also `confirm.dismiss` if you want to test the negative path

Dismissing a messge - i.e. alert(“Reset password instructions have been delivered.”);

alert = page.driver.browser.switch_to.alert
assert_equal alert.text, "Reset password instructions have been delivered."
alert.accept

Inputting data to a prompt - i.e. prompt(“Enter your project name to cancel”)

prompt = page.driver.browser.switch_to.alert
assert_equal prompt.text, "Enter your project name to cancel"
prompt.send_keys "my-project"
prompt.accept

TIL: SASS @at root

Wed, 21 Mar 2018 00:00:00 +1300

The @at-root directive can be quite useful for creating SASS rules where you need to jump outside of a SASS scoping block back to the root level when your styles are compiled.

In particular, this directive can be useful for generating BEM styles, since the & abbreviation continues to work for targeting the parent selector.

Here’s an example of a component without @at-root:

.button {
  &.collapse-to-icon {
    padding: 1rem;
    &__text { display: none; }
    &__icon { display: inline-block; }
  }
}

Which is compiled to:

.button.collapse-to-icon {
  padding: 1rem;
}

.button.collapse-to-icon .button.collapse-to-icon.collapse-to-icon__text {
  display: none;
}
.button.collapse-to-icon .button.collapse-to-icon.collapse-to-icon__icon {
  display: inline-block;
}

The problem with this CSS is that because of the parent nesting, unnecessary conditions are prefixed onto the elements of the collapse-to-icon block.

With @at-root, something like this can be written:

.button {
  &.collapse-to-icon {
    padding: 1rem;
    @at-root {
      &__text { display: none; }
      &__icon { display: inline-block; }
    }
  }
}

Which is compiled to:

.button.collapse-to-icon { 
  padding: 1rem;
}

.collapse-to-icon__text {
  display: none;
}

.collapse-to-icon__icon {}
  display: inline-block;
}

TIL: before(:all) callbacks run outside the test transaction

Wed, 14 Mar 2018 00:00:00 +1300

While working on a feature test today, I learned that any database data that is modified in a before(:all) callback in an RSpec test is retained outside of the test run. This is documented quite clearly in the RSpec documentation, I just haven’t had the opportunity to work with these types of callbacks regularly before.

The test I ran into issues with looked like this:

require "rails_helper"
RSpec.describe "MyFeature", type: :system do
  before(:all) do
    @widgets = FactoryBot.create_list(:widget, 5)
  end

  # ... tests
end

The first time this test ran against an empty database, the test passed, no problems. The test also ran fine on CI, even after repeated attempts. If I tried to run the test multiple times locally though, the second run onwards failed with a uniquenss constraint error on widget name (which had a unique index).

After reading the documentation, I found that while before(:each) callbacks run within the ActiveRecord transaction that wraps each RSpec example, and therefore rollback any data modifications after the test completes, before(:all) does not. In the example of the test above, this meant that on the first test run, 5 widgets were being committed to my test database. The next test run, these widgets were already present, causing the uniqueness validation to be violated when widgets with identical names were created.

Generally, the most appropriate solution with this type of test data is to move the data being created in a before(:all) callback to a before(:each), like so:

before(:each) do
  @widgets = FactoryBot.create_list(:widget, 5)
end

If this is the entirity of your callback block, you may as well make this a let block (but don’t forget to understand when to use let vs let!):

let(:widgets) { FactoryBot.create_list(:widget, 5) }

If you are doing more complex set up in your before(:all) or cannot set up this data in before(:each) callback, the before all callback can still be used, it just becomes important to use a corresponding after(:all) callback to clean up the test data to prevent subsequent test failures:

before(:all) do
  @widgets = FactoryBot.create_list(:widget, 5)
end

after(:all) do
  @widgets.destroy_all
end

TIL: text underline position

Tue, 13 Mar 2018 00:00:00 +1300

TIL about the CSS property text-underline-position. This property is a useful progressive enhancement to achieve a particular look and feel for links, in particular for content with many descenders.

Examples:

This is standard underlined text.

This text has the underline position set to "under".

This text has a bottom border instead of an underline. Notice that the border overflows the text and doesn't work for multiline.

Browser support is not so good for this feature, it being fully supported only in Chrome and Edge on desktop, and Edge on mobile, so it is important to consider this a progressive enhancement. A similar effect can be achived with a CSS bottom border, however this can make alignment of “underlined” and non-underlined text difficult.

TIL: The |= pipe equals CSS selector

Mon, 12 Mar 2018 00:00:00 +1300

Today I learned about a CSS selector I have not used a great deal before - pipe-equals. The spec describes this operator as:

Represents an element with the att attribute, its value either being exactly “val” or beginning with “val” immediately followed by “-“

This selector is particularly useful for loose matching of BEM CSS classes, and other dasherized class naming systems, for example:

[class|="icon"] {
  background: #656565;
}

.icon--small {
  width: 32px;
  height: 32px;
}

.icon--medium {
  width: 64px;
  height: 64px;
}

.icon--large {
  width: 128px;
  height: 128px;
}

For information:

Calibre Content Server

Fri, 09 Mar 2018 00:00:00 +1300

I’ve used Calibre for years to manage my eBook collection. It’s a great piece of software that has always worked really well for me.

I discovered today that Calibre has the ability to start a “content server”. This provides an endpoint which serves information about the Calibre library, including covers, titles, and descriptions. The content server can be started by clicking “Connect/Share” in the top menu bar, and then selecting “Start Content server”. This starts a server listening on all hosts on port 8080.

Under “Preferences” -> “Sharing over the net”, there are a slew of options allowing for some customization of how the content server works, including authentication mechanisms.

Visiting http://localhost:8080 will allow a library to be selected and will show a nice grid view of all the books in the library, along with cover images, titles/descriptions, and options to download books in all the formats which have been generated for that particular title.

I’m making use of this feature of Calibre to begin moving my library off my own laptop onto the Raspberry Pi which manages much of my home network infrastructure. In particular, the content server will allow me to create a shared library with the rest of my household.

Rails: config_for

Wed, 07 Mar 2018 00:00:00 +1300

Sometimes an application can have pretty complex configuration for certain components. There’s always temptation to put these settings straight in your service, but today I learned about a simple method built into Rails from 4.2 onwards that lets this config live outside of your Ruby classses: Rails.application.config_for.

This method allows you to place your config in an environment-namespaced YAML file within your config/ folder, just like database.yml or secrets.yml. Here’s an example of a config file I created for Shopify:

# config/shopify.yml
development:
  shop_name: 'mystore-development'
  api_key: 'abc123'
  password: 'secret!'
test:
  shop_name: 'mystore-test'
  api_key: 'abc245'
  password: 'secret!'
production:
  shop_name: <%= ENV['SHOPIFY_SHOP_NAME'] %>
  api_key: <%= ENV['SHOPIFY_API_KEY'] %>
  password: <%= ENV['SHOPIFY_PASSWORD'] %>

It can then be accessed within an initializer or even your service object:

ShopifyService.configuration = Rails.application.config_for(:shopify)

With the example above, this would return a hash:

{ 
  shop_name: "mystore-development",
  api_key: "abc123",
  password: "secret!"
}

Pattern for configuring integration objects

Wed, 07 Mar 2018 00:00:00 +1300

I often struggle with the correct way to set any kind of configuration on a class, especially when it is a class intended to integrate with an external API that usually needs several configuration values.

I initially implemented my own configuration pattern, before coming across this great pattern by ThoughtBot that has been used for their open-source library Clearance:

class MyApiIntegration 
  class << self
    attr_accessor :configuration
  end

  def self.configure
    self.configuration ||= Configuration.new
    yield(configuration)
  end

  class Configuration 
    attr_accessor :api_key, :api_secret
  end
end

This exposes a very simple API for configuring the object:

MyApiIntegration.configure do |config|
  config.api_key = "wow"
  config.api_secret = "api"
end

I found the attr_accessor on class << self particularly useful, as I had been running into issues with class_attribute, class-level variables, and cattr_accessor where I was losing configuration between test runs. The only issue I have with this code snippet is that the default configuration (if any) is not assigned until configure is called. I am going to experiment with this pattern in my investment time to try and identify a way of having the default configuration used if configure does not need to be called.

Thanks ThoughtBot!

A Brief History of Donnelly Flat

Wed, 07 Mar 2018 00:00:00 +1300

I frequently find myself tramping into the Tararua Forest Park from the Mount Holdsworth road-end, travelling through Donnelly Flat, a popular picnic spot around 10 minutes from the roadend. After considering the lack of signed history regarding the history of the Flat, and after being unable to find out any information from Archives, National Library, Papers Past or regional libraries, I got in touch with local DOC rangers. They have kindly provided me with a short history of the Mount Holdsworth area by Ben Iorns (1883-1977), a Masterton historian. This history has provided background information for this post.

The Mount Holdsworth area is one of the oldest and most popular park accesses, largely due to the relative accessibility of the peak to the Masterton area, along with easy access up ridgelines to the summit. The first documented trip according to Iorns occurred in 1863, and by 1907 the Mount Holdsworth Committee was formed to cut a track and construct the first of three Mountain Houses, all located on the same site at Pig Flat.

A particular feature noted by early visitors to the Holdsworth area was the profusion of the undisturbed native bush, especially in the Atiwhakatu valley around Donnelly Flat. Sawmillers expressed a keen interest in milling this bush, but fortunately faced determined opposition over many years which prevented milling from ever occurring inside the park boundaries.

Sawmillers quite naturally coveted the tall clean trunks of those rimus, kahikatea, miro and beeches of 3 species, very generously offering to leave us the stumps and branches and a promise of a newer and more virile forest that would soon replace its “over mature” ancestors.

Donnelly Flat itself is a grassy clearing occupying a couple of acres. The clearing is natural, or at least not cleared by human occupation as far as it known. The flat has always been popular with picnickers and campers due to it’s sheltered position in the valley and proximity to Atiwhakatu Stream. The Flat was named for Thomas Donnelly, a prospector from Masterton who established a camp at the flat, being supplied weekly from the road-end. In April 1910, he failed to collect his resupply, kicking off an intensive search. A year later, his remains were found, with indications that he had injured himself in a fall and perished from exposure.

Given this history, it’s understandable that the origin of the name is somewhat under documented, but I think it is still important to acknowledge the events and people attached to these special locations. I’d like to acknowledge the efforts of the Masterton office of the Department of Conservation for their speed and efforts in locating and passing on this fascinating history, and Ben Iorns for documenting his recollections of the Holdsworth area for future generations to learn from. I hope that others find the history of Donnelly Flat as interesting as I have.

Trip Report: Akatawara Hill Road Summit to Renata (Attempt)

Tue, 25 Jul 2017 00:00:00 +1200

This post outlines a day trip from the Akatawara Hill Road saddle towards Renata Hut. After a couple of recent winter storms, I ran into a lot of tree fall that slowed things down sufficiently so that I ran into my time limit and turned back, however, it was a struggle to find information about the route, so I’m happy to document what I ran into up to about 3/4 of the way to Renata Hut.

Quick facts:

Start time: 8:30am from Akatawara Hill Rd saddle

Finish time: ~ 2pm

Walking time: 5.5 hours - 3 up, 2.5 down

Topomap: BP32 and BP33, here’s the frame on topomap.co.nz.

This route uses the rarely used Akatawara Hill road entrance to the Tararua Forest Park (NOT the Ngatiawa Road entrance used to access Kapakapanui). The entrance is located directly across the road from the gravel dump spot at the summit of the Akatawara Hill Road. I’ve read of multiple break-ins at this spot, and passing a burnt out car on the way up (that was not there last time I passed!) is evidence enough that while it’s safe enough for day parking, it might not be the spot to leave a vehicle overnight. A pity, as the walk would make a great short overnight (an alternative overnight, albeit with a longer day, would be to park at Ngatiawa and walk over Kapakapanui and down the ridge to Renata).

The route is entirely on 4WD track as far as Maymorn Juction, where the 4WD track continues to Waiotauru Hut. I didn’t get up to the Junction, so my entire experience is on the 4WD. Note that I say “track”, not “road” here - I was impressed that any vehicles can get up at all, but did see some tyre treads in the mud, so assume the track is still used. Because the track is 4WD, it is a relatively gentle gradient, and there is gravel along much of it.

Starting at the saddle carpark, there is a steep rough section for a couple of hundred metres to start - I suspect this has intentionally been dug in to discourage anyone who doesn’t have an adequate vehicle from venturing up (I had optimistically hoped to get my all-wheel drive up the “road” (as it is marked on maps) some of the way to shorten the day, but this is impossible). After this, the track more or less levels, and winds its way up the hillside. I did this in winter, but judging from the size and location of the puddles, in the first 20 minutes of walking, expect to get wet feet. Most have a dry-feet route around them, but overall, your chances of keeping those feet dry are low - you might be better off just wading through.

The 4WD track remains roughly the same all the way up. There are occasional rough patches, and it is clay, so parts are slippery, but it’s easy to move pretty quickly. There are a few places where the track forks - I think these have been added by 4WD-ers after some variation in the route and it’s obstacles - I just picked the option that looked the most used and it worked out fine - I think that all of the forks rejoined the main track in any case.

I was checking the topomap on my phone frequently and still managed to miss when the “road” turns into a normal track. For others’ reference, it is where the road goes around a wide bend, with a small patch of grass. While the track remains pretty much the same underfoot, it does close in a bit.

The track continues - on the topomap it is shown as if the track goes directly along the ridgeline. In reality, it actually traverses on the eastern side and about 40-60 meters below the ridge. Along this section is where I encountered a lot of tree fall, all of it looking pretty recent. The entire route is unmarked with tape or triangles, so it’s just a case of sticking to the track and doing frequent map checks.

It was all relatively small stuff, but because of this, some parts took a bit of work to fight through, over, or around. There weren’t any trunks thicker than about 20cm, so hopefully a kind 4WD club member or even DOC might make their way up with a chainsaw sometime. When I went up, the treefall was every 50 meters or so. Since I also had a relatively small dog to get through each bit as well, I found that it was taking me 2 or 3 minutes to negotiate each section. Probably without the dog it would be faster, but it’s still tough going.

My turnaround point was a large slip in the saddle immediately before the unnamed high point at an altitude of 762m on topomap BP33. While the DOC time to Renata Hut from the saddle was described as four hours, I had expected to make it up the hill a bit quicker, and couldn’t quite fit a full 8 hour day in. I had set my turnaround time to get to Maymorn Junction at least at 3 hours, knowing it was an additional 30-40 minutes to Renata Hut along the Junction.

I reached the slip bang on 3 hours. It was about as good a weather as I have ever had in the Tararuas, but I still eyed the cloud hovering around Kapakapanui with a bit of caution, and considered that I had to make my way back through all of the treefall, and made the correct decision to stick to my cutoff time. While it did take me less time to head downhill than uphill, I still spent a decent chunk of time maneuvering through downed trees, reaching the road again after about 2.5 hours. I estimate that it would have taken me at least an additional half an hour to reach Maymorn Junction, where the track from Kapakapanui meets this track, and the 4WD track drops off, and at least a half hour beyond that to reach the Hut.

This trip was another great reconnaissance for future adventures. If there was slightly less dodgy looking car parking, this trip would be a great short overnight and I suspect would be a lot more popular. With the treefall, I’m not sure it’s a great day walk right now, as fighting through fall after fall becomes tiring quite quickly, especially when contemplating the return journey that same day - however, the forest is classic Tararuas, and the track is well graded and in good condition (aside from the trees!).

Temporary PostgreSQL Database with Docker

Wed, 11 Jan 2017 00:00:00 +1300

Every now and then I need to restore an old database backup to grab some data out of. I use Postgres containers for every app I work on, so I don’t really want to go polluting those. Here’s how I start a postgres container, restore a DB into it, do what I need to do, then destroy it.

Start the container: docker run --rm -it -v `pwd`:/data postgres bash
- --rm means remove the container when it stops (we exit bash)
- -it means give me a interactive terminal (I need to type stuff and see output)
- -v means mount the directory I’m currently in to /data - assuming the directory I’m in has a PG dump I want to use.
Now in the container, we need to set up the database engine. The postgres container has an entrypoint script that normally does this for us, but we’ve told Docker to run bash rather than this script.
- Create the cluster: pg_createcluster 9.5 main (replace 9.5 with whatever version of Postgres you have used)
- Start Postgres: service postgresql start
- Create a user for ‘root’: su - postgres -c "createuser -s root"
Now we have a superuser Postgres user named “root”, we can do whatever we want, for example:
- createdb my_database
- cd /data
- psql my_database < my_database_backup.sql
- psql my_database
- SELECT count(*) from widgets
When we’re done, we just need to exit bash. The container will stop and be removed by Docker.

Totara Flats, Tararuas

Fri, 31 Jul 2015 00:00:00 +1200

A little while ago, I blogged about the trip to Powell Hut. This post decribes an alternative trip into the Tararuas, from the Waiohine road-end into Totara Flats Hut. I found this trip much more challenging than the trip up the Gentle Annie track to Powell Hut, but at the same time a lot more fulfiling.

The track, marked in the thin red line View Topomap in browser

We began the track from the Waiohine road-end. The road-end is reached by turning off just after the bridge over the Waiohine River just outside Greytown, and then following a narrow but sealed country road. After crossing the railway tracks keep an eye out for the turn off on your left - it’s just marked with an old yellow AA sign, and I missed the turn at first.

The road soon turns into gravel but is a great drive up the gorge towards the picnic area. It’s similar to the Otaki Gorge road, but not as long or windy. After heading through the picnic area, a turnaround bay is reached with a chain across the road. In 2008 a large slip and washout closed the road further up, and while repairs are going on the road is impassable. I’ve not heard either way about the security at this carpark, but there were several cars parked there when we walked.

Leaving your car at the turnaround area, you can cross the chain and walk up the road. You’ll reach the slip after about 5-10 minutes, with the campground another 5-10 minutes after that.

This was the slip & washout when we were there. Repairs have started so it may be easier to cross soon - but progress is slow.

The campground has been a popular spot in the past, but cutting off road access gives it a strange abandoned air. It’s no longer serviced but there is plenty of space for camping, and flush toilets here, if you are planning on stopping overnight.

To get to the tracks into the park, you’ll need to cross a large swing bridge. This bridge is pretty recent, but due to it’s size and positioning in the narrow gorge you can expect some movement when the wind is blowing. It’s quite high up above the river but a nice uphill slope allows nervous trampers to look uphill and focus on the end of the bridge.

After crossing, there’s a intersection, with the left hand fork crossing over a ridge and dropping to the Tauherenikau River and Cone Hut (apparently about 2-3 hours), and the right hand fork turning to follow the gorge up towards the flats. Taking the right hand fork, the track condition soon deteriorates into large (semi-continuous) mud pools along the track route, interspersed with large tree roots. The track condition was a slight shock to us after the well-maintained Gentle Annie Track, but seems more typical of the classical tramping tracks in the Tararuas.

The track carries on up the gorge, undulating along the river bank. Typically the track is routed over small bluffs so expect steep but short climbs and descents. There are several small streams and creeks along the gorge, but all major crossings are fitted with swing-bridges (The bridges don’t look terribly old and they are not on the topo map, so I imagine they were all done in one go). After about an hour, you’ll reach a nice Beech clearing just before Clem Creek. A good spot for a rest, and it’s looks like a popular camping spot judging from the number of old campfires. Clem Creek is about halfway along the narrow gorge section before opening out onto the flats so is also a good milestone to judge progress. The next hour continues much as the first - mud and roots.

One of the small swing-bridges over the smaller creeks going up the gorge

About halfway between Clem Creek and the flats you’ll reach an intersection with another track that heads over Cone Saddle and continues to Cone Hut to meet the left-hand fork of the track from the swing-bridge. This seems like a nice overnight trip that is on our to-do list, although I’ve read that the track continues to be muddy and rutted.

Soon after the intersection is an impressively large old slip - there are two options for getting around it, both marked - either the river route or climbing up and around the slip. I recommend the river route unless the river is in flood - the slip is a lot larger than it looks from the intersection. Clamber down the river bank and then head upriver keeping an eye out for the orange triangles that mark the exit point for the diversion at the beginning of a short section of open grass. Keep an eye out from now on for sign of deer and pigs, both apparently common in this area. After the short bit of flats, the track climbs again steeply around some bluffs, and then drops back to the main body of flats.

If you’re interested in checking out a historic hut, and don’t mind getting your boots wet, after joining the flats head off the track to the river. You should emerge within sight of a small rapid that diagonally crosses the river, with the end closest to you slightly further upstream. You can cross the river here in the right conditions and check out Sayer’s Hut, an old Forest Service hut constructed of native timber with an open fire. Finding it from the river can be a little tough, but there are clear trails climbing the river bank after you cross the river that should get you within sight of the hut - it’s not directly visible from either bank of the river. We crossed over to the hut on the second day on the way back down the valley, but doing it on the first day probably makes more sense - less distance to travel in wet boots after crossing back over the river! The hut itself is in a lovely spot tucked into the bush back from the river, and this is also on our to-do list for an overnight trip. It sounds as though it doesn’t get frequent visitors - mostly locals and hunters.

The river crossing point we used to reach Sayer’s Hut (a little wonky!)

From the crossing to Sayer’s the Totara Flats Hut is a quick walk up the valley through the grass. After the tough terrain up the gorge, the grass is a bit of a treat. Soon before reaching the hut, you’ll head back into some scrubby bush - the hut is 5 or 10 minutes on from this point.

Totara Flats Hut is a 26-bunk hut, so there is plenty of space. It’s often used as an alternative to Powell Hut, as the track that you have travelled on continues to the head of the valley and climbs to meet Gentle Annie track near Mountain House. The hut is a pretty standard design, with two bunk rooms and a common area with a mixed-fuel (wood/coal stove). There used to be gas cookers provided here, but DOC has removed them (though they’ve left the hut fee the same at 3 tickets ($15) per person - pah!). We had a slightly chilly night after the rain during the week, as there wasn’t much dry wood available. If you use some of the dry firewood from the wood shed, don’t forget to pop a bit more under cover to dry out for the next group! Also a hut to watch out for rats in - there were droppings everywhere when we were there - probably due to the quantity of food and little bits and pieces left behind by others. We initially had the hut to ourselves but were joined later in the evening by a couple of others who had diverted from Powell Hut due to ice higher up.

The following day there are a couple of options. We had been hoping to do a long day, heading back down the valley to the track turnoff to Cone Hut, and then crossing over the saddle and back over the ridge back to the Waiohine Swing-bridge, but after seeing how long it took us to get up the valley we decided to head back the the way we came. On the way back we took our time a little bit more, and checked out a couple of geocaches on the way. It did take us about the same amount of time to get out as it did to reach the hut, so we must have been moving a little faster - perhaps because we weren’t trying to keep dry boots for as long as the first day!

The whole trip up the valley took us pretty much bang on the DOC time of 4 hours. We normally do slightly under the DOC time, but the tough terrain certainly slowed us down - the roots require a lot of clambering and can be a bit slippery. This is also the trip on which we decided we needed a personal locator beacon - the country is beautiful, but has an air of inaccessibility. Despite all this, this is one of the best overnighter trips I’ve ever done. The closed off road, the steep and tough terrain, and the solitude in the valley (we saw 3 people in two days), makes for a fantastic escape. I highly recommend it!

Powell Hut, Tararuas

Fri, 19 Jun 2015 00:00:00 +1200

The Tararua ranges have a fearsome reputation in New Zealand. Frequently cloudy, snowy, rainy, misty, almost always windy, and very occasionally, sunny (though not to be trusted). Despite this bad rap, they are accessible, scenic, and have a broad range of trip options - from short dawdles through the bush to challenging alpine tramps over the tops.

With Wellington so close by, these ranges offer so much. If you’re a Wellingtonian, there’s something there for you. Today, I’m going to be talking about a perfect intro trip - from the Holdsworth carpark near Masterton to Powell Hut and back. If you live anywhere within a 2 hour drive of this trip, it’s a must-do. It ascends gently from a river valley, travelling up through beech forest and subalpine bush and popping out above the bushline straight before the hut. In winter, you can expect to see snow around, but there shouldn’t be enough of it to worry too much.

I’d consider the Holdswork carpark to be the main entry to the Tararuas, along with Otaki Forks on the Western side of the range. The carpark is just over 90 minutes drive from central Wellington along State Highway 2. The turn is between Carterton and Masterton (though closer to the latter), and is signposted ‘Mt Holdsworth’. There’s a small wood pulp processing plant (from memory, it’s something of the sort) next to the turn, so you can keep an eye out for the chimney stacks from that. Drive past the farm supplies place and the livestock sales yard and head along the rural road for about 15 kilometers until it narrows and then goes through the gate into the Holdsworth campsite. This campsite isn’t particular interesting as a destination in it’s own right, but can make a good staging post if you’re travelling from the city in the evening (Camping is DOC-standard $6 adult per night which will get you a grass spot of your choice, a cooking shelter and flush toilets).

This carpark should be safe to leave your car (plenty of others do) - the ranger’s house is just up the driveway next to the carpark and while there’s campers around this should deter most - but take all precautions - don’t leave valuables in your car, make sure all the doors are locked etc. Don’t forget to make sure you’ve turned your headlights off!

The track starts along a gravel road that goes to Holdsworth Lodge, a DOC managed accomodation facility that offers group or sole-occupancy room options. There’s an intentions book on the left hand side as you face the lodge where you should write down your plans and when you’re expecting to be out - this will be the first thing that is checked if somebody has to come and find you. Once you’ve done that, the road carries on just past the lodge then becomes a narrower quad-bike-width track before reaching the bridge over the Atiwhakatu Stream.

The track up towards Powell Hut is the aptly named “Gentle Annie” Track. For the first few minutes, it follows alongside the stream. The track is wide here with a good surface. Expect to see a few locals with walking their dogs (and kids!). After 10 minutes or so you’ll reach an intersection - the track continues straight ahead up the Atiwhakatu Valley, passing through the Donnelly Flat picnic area about 15 minutes after the intersection, eventually reaching Atiwhakatu Hut.

The Gentle Annie track narrows from the quad-bike sized track to more of a normal bush track - though the track surface remains excellent. It quickly enters a small valley, climbing up a spur towards the first stop, Rocky Lookout. The climb can be steeper in places, but there’s steps everywhere there should be, and plenty of places to pull over for a breather. Eventually you’ll reach the head of the valley and begin to traverse around the hillside, before joining another short spur for another bit of climbing. Eventually, you’ll emerge from the Beech forest, and if you look up the hill, you should see the railings of Rocky Lookout above you. This is a great spot with fantastic views. The rock the lookout is based around is fun to climb, and it’s a good chance to get your breath back, have some water and food. Expect the track from Holdsworth to Rocky Lookout to take between 30 and 40 minutes with one or two breather breaks on the way up. At Rocky Lookout, keep an eye out to the left for Powell Hut high up on the bushline.

Fortunately, after Rocky Lookout there’s a bit more of a gentler climb up the last remaining bit of spur before reaching a saddle, where the track turns and begins to climb along the hill. It’s pretty easy going, with an easier gradient as it continues around the hill before meeting the start of a ridge where there is a key intersection and a couple of benches about 30 minutes after Rocky Lookout. Another good spot for a rest and some water! Going straight at this intersection descends down a valley to the Waiohine River and Totara Flats Hut, which can be a good bad-weather alternative to Powell, but turn right to head towards Mountain House, the next landmark on the way to Powell Hut. The track climbs just a little more, before flattening off, crossing a couple of boardwalked sections, and then descending moderately to Mountain House. This section allegedly takes 20 minutes, but always seems to last much longer.

Mountain House can be a good place for a quick stop, but no doubt you will have been having to look further and further up to be able to see Powell Hut and may have a slight dread of the hill that is to come - best to make it a quick stop and get stuck right into it. The track climbs up along the hill for a short period, before zigzagging a couple of times and then starting to climb up the left hand side of a spur that goes all the way up to the hut. The trick to this bit is to take breaks as needed - the DOC time to get up this section is generous and already allows for a few breaks. There’s a flat bit next to a couple of drainage holes that makes a good spot for a break, as it’s about halfway up the bush part of the section before the views emerge.

Carrying on up the spur, you’ll eventually come to patchier sections of bush, and get into rockier terrain with steps on the steeper climbs to make things a bit easier. Watch out for ice at this level. There’s great views all the way up back down towards Holdsworth, down into the Waiohine Valley to Totara Flats, and out across the Wairarapa.

It seems like there’s always just one more ridge to overcome, but eventually you’ll pop up to a surprisingly flat section, with the hut just a few metres along the hill.

Powell Hut is a extremely popular hut, with bookings required in the summer months (Nov - Apr). In winter it seems to be variable, but with 28 bunks (plus plenty of floor space), it’s a pretty safe bet. There’s cookers installed with LPG tanks helicoptered in occasionally (making use of an interesting telecommunications system to send alerts), and a dual-fuel wood/coal burning stove for heating. This hut is a Serviced hut, which means it requires 3 hut tickets per person, which translates to $15pp. Hut tickets can be bought from DOC offices and some outdoors stores, and go towards paying for hut maintenance and servicing. There doesn’t seem to be a great deal of good camping sites nearby the hut, but if need be there’s probably a couple of places to wedge a tent. There’s a great outlook from the hut over the whole of the Wairarapa, and the deck is quite nice if it’s not too cold and/or windy.

Powell Hut was first built in 1939 by the Hutt Valley Tramping Club as a bit of a ski lodge - back in those days, a trip up to Ruapehu wasn’t quite as easy as it is today, and skiing in the Tararua’s was popular, both here and around Field Hut on the other side of the range. The hut was rebuilt as a Lockwood hut (as many NZ huts of that generate were, and are) in 1981, but burned down in 1999 (many huts burn down due to the fire being mishandled - for example hot ashes or embers escaping). In 2000 it was replaced by DOC and this is the iteration of the hut that stands today. This area is also the scene of an unsolved murder and even has a local ghost!

If you’re staying the night at Powell (or have made good time on the climb up), you can continue up to the summit of Mt Holdsworth during summer, if the weather is reasonable. In winter there is snow and ice at this altitude and it is not safe to go all the way to the trig without the proper experience and equipment. From the hut, the track climbs steeply up to the ridge, and then clambers over, around, and through tussock before reaching a turn off to High Ridge (I believe this descents to Cone Ridge above the Waiohine). This is as far as I’ve ever been, but I believe this turnoff is about halfway to the trig.

The descent is exactly the same as you came up (though there are steeper alternatives down from Mountain House to the Atiwhakatu Valley and out from there if you’d prefer a more difficult finish to the day). The steep bit down to Mountain House can be a bit tough on the knees and it’s worthwhile stopping once or twice on the way down this bit. After that it’s a nice doddle back down to Rocky Lookout, then the stream track back to the bridge, finally emerging at the carpark. Depending on how quick you are on the way down, the descent takes us about 2-2.5 hours, but will be longer if you stop on the way down as well.

From the carpark, it’s time to head back out to the state highway. On the way home, we usually can’t resist stopping off at The White Swan, for a bit of a treat!

Simple Ember CLI deploys

Sat, 25 Apr 2015 00:00:00 +1200

Note: I can’t vouch as to the best-practise compliance of this technique, but it’s worked well for me.

One of the neat things about using a Javascript framework such as Ember.js is that you can host it just about anywhere you like - it’s just static files after all. My first stop for hosting things like this is Github Pages - it means the site is backed by Git (as it should be anyway), and your files are being served by Github’s pretty solid platform.

If you’ve got a Github Pages site though, generally you are deploying to a branch named gh-pages. The files you push to the gh-pages branch should also be the files you would normally place at the root location of your web server (i.e. there should be a file named index.html that will be served by default. This becomes a bit tricky with git, as you need the files contained in one particular folder of your repository to be pushed to a different branch than the one your on.

Based on a bit of research, I found that Git has a way of doing this, and it’s a pretty neat technique. The subcommand is called subtree push, and it basically let’s you push a particular folder of your repository to a remote branch. In the context of Ember (Ember CLI in particular), this means pushing your dist folder to the gh-pages branch, or:

git subtree push --prefix dist origin gh-pages

It’s not quite as simple as that though! Applications generated with ember-cli (quite rightly in most cases), add the dist folder to the .gitignore file, which will cause that subtree command to ignore files in that directory when pushing (i.e. it appears to succeed, but nothing is pushed). I haven’t yet found a good solution for this, so for now I’ve just stopped ignoring the dist folder. This invovles checking in changes every time I build, however it seems to be necessary for this deploy process to work.

My workflow for making changes to an Ember CLI application now then is:

Make the changes
Run the tests (if any, this is an area of Ember I need to catch up on)
Run ember build production to update the contents of the dist folder
Run git subtree push --prefix dist origin gh-pages

Most of the time, I’ll put those last two commands into a shell script I can invoke, and call that deploy.sh - but then - that’s it, that’s my deploy process. My changes go up in a couple of seconds and are live straight away. I can then check in my changes to master as I normally would, and master will continue to include a full copy of the Ember application that anyone can clone and hack away on.

Tips, Tricks and Notes

If you’ve got any files that you just want to go into the dist folder, put them in the public folder of your Ember CLI application. A perfect use case for this is the CNAME file that Github requires be in your gh-pages branch if you wish to direct a custom domain name to your Github Pages site.
If you are not using a custom domain, and you’re pushing to a project repository, then your Github pages site will be available at: https://[github username].github.io/[your repository name]. In order to get your assets, etc. loading correctly (because the site is in a subfolder), you need to add the following to your config/environment.js:

      if (environment === 'production') {
        ENV.baseURL = '/name-of-your-repo'
      }

If you would prefer a more integrated approach, you might be interested in the ember-cli-github-pages addon for Ember CLI. Personally, it’s not for me, as I prefer to put together my own git commands - I’m not so sure it’s an Ember addon should be taking care of the whole deployment like that.

Walking the Waikaremoana

Mon, 05 Jan 2015 00:00:00 +1300

The Waikaremoana track is one of the Department of Conservation’s nine great walks, and is located in the Te Urewara area of the North Island, inland from the East Cape of the North Island of New Zealand. Recently, Te Urewera was one of New Zealand’s many National Parks, however a recent Treaty of Waitangi settlement has made the park a seperate legal entity, managed by a board comprised of Tūhoe and crown stakeholders (This page on the DOC website has more info).

The Track is 46km long, and has both huts and campsites along the route (unlike many other great walks, which tend to have hut accomodation only). It is not a loop track, and so requires some form of transport to get between the two ends of the track - Onepoto and Hopuruahine Landings. It can be walked in either direction, but the most common direction is to start at Onepoto, the south end of the track, and walk through to finish at Hopuruahine.

I had the pleasure to walk the Waikaremoana track recently, over the 2014/2015 summer break. The rest of this post covers my personal experience of the track, it’s accomodations, facilities, and transport options.

The Track

Interestingly, although it is most common to walk the track from South to North (that is, from Onepoto to Hopuruahine), the spaces on the track when I went to book made it necessary to go the opposite direction, starting at Hopuruahine and finishing at Onepoto. I found this direction much better. It avoids the tough climb on the first day of the tramp (in the usual direction), as the climb from the other side is much better track, and is on the second-to-last day, so packs are just that little bit lighter.

The track condition is variable, depending on the track’s age. With the Urewera’s high rainfall, bogs and mud is pretty regular, with some creative workarounds on the edges of some bogs. Given the cost of doing the track (Either $14 or $32 depending on whether you are camping or staying in the huts), I would expect more work on maintaining the track though. During summer, there’s DOC staff on the track full-time, and while we saw a crew doing some weed-eating to keep the grass back around a couple of huts, there’s plenty of scope to add a bit of gravel on some of the boggier bits. To DOC’s credit, the section of track on the second-to-last day climbing to the Panekire bluff is fantastic (and looks very recent). 454 (that my partner counted) steps make the very steep ascent tiring, but over quickly. The old track can be seen beside, and at times underneath the staircases on this part of the track, and after seeing this I can appreciate the improvement.

The track is clearly marked with DOC-standard orange triangles, with a reassuring larger triangle every now and then. With no other tracks in the close vicinity of the track, though, it’s largely a case of following the track - getting lost on this one would be pretty tough (which isn’t to say it doesn’t happen!). There was just one confusing section, on the last day just past the halfway point on the descent down the Panekire ridgeline, where what looked like a new benched track had been cut, but the orange triangles followed the old track for a few hundred meters. We followed the bench track, preferring a better surface and gradient to track markers, but this wasn’t necessarily the right move. I’m not sure why there isn’t a sign or markers on both routes here to provide guidance, but can advise that the two sections of track do re-join eventually. Expect to be without markers for 10 or 15 minutes, and carry on!

A bit of a mud bog on the Waikaremoana Track

The Accomodation

There’s two options for accomodation on the track - huts, or campsites. Generally, the huts seem to be the more popular option, especially with families with younger children (sidenote: this track has the highest number of families with kids I’ve seen - but I still think Abel Tasman is the easier track to start with). The huts, though, are twice the price of the campsites, and you certainly shouldn’t expect any luxuries. During the tramp, I checked out most of the huts on the track, and they’re all of a similar standard. You can expect a DOC-standard vinyl mattress, a roof over your head, a wood-fired stove, and, if you’re staying in one of the cutting-edge huts, solar-powered LED lighting.

Since I had a lightweight tramping tent, I went with the camping option, and was glad that I did. The campsites have shelters with water, a long-drop toilet and a cleared area of grass for tents. Each campsites have a maximum capacity of 15-20 tents, though some are smaller. I enjoyed the privacy and self-contained nature of the camping option, although I stayed in a hut the last night, at Panekire Hut (there’s no camping available there, I assume due to the fairly exposed location of the hut on the edge of a bluff, and the frequently adverse weather there.

Typical campsite - on the first day

One important note is that neither the huts or campsites are provided with gas cookers. This is a key difference from some of the South Island great walks, where cookers are provided in the Summer season. I ran into one family starting the walk at Panekire, who only realized this after arriving at the hut after a 4 to 5 hour ascent up the ridgeline. Hopefully they were able to beg and borrow a cooker and gas for the remainer of their walk! At a pinch, you can cook on top of the wood-burner, but this’ll only work if you’re staying in huts, and will certainly add to your cooking time. Best to take along a small gas cooker and a couple of cannisters of gas.

One aspect of the track accomodation that disappointed me was the communication and service from the rangers who are assigned to stay at the huts for a number of days. Previous great walks that I’ve done have had weather forecasts posted somewhere in each hut, along with any notes about upcoming sections of track. Waikaremoana huts do not do this, and this was unanticipated by me, as I had expected to be able to get updates on the weather in this way. I’m also used to rangers carefully checking hut and campsite bookings to make sure everyone is where they’re supposed to be, and this only happened for me at Panekire Hut. To be fair, this is the only hut I stayed at on the track, but the campsites all require booking plenty of time in advance as well. It was really disappointing to have paid for a space in the campsite, and then not have this enforced at all. I heard after I got off the track that this is fairly well known amongst backpackers especially, and quite a few people camp the track without a booking and just gamble on not being checked on. I feel like this is a dishonest way of doing the track, since the fees go into maintenance and upkeep of the facilities, but I also feel like it reflects a failure of DOC to enforce their own restrictions on usage of facilities. At a minimum, having a ranger visit all of the campsites that are easily accessible from their assigned hut should really be happening - it gives campers the opportunity to ask questions and find out about the track ahead, and is probably useful for search and rescue purposes if trampers are checking in with a ranger as they reach each campsite.

Start & End Accomodation

DOC recommends the Waikaremoana Holiday Park for accomodation before and after the track - which is reasonable, since it is DOC that runs this park. Instead of the Waikaremoana Holiday Park though, I would recommend Big Bush. Big Bush Holiday Park is a few kms down the road from one end of the track at Onepoto Landing, and has campsites and backpackers. It also looked to have some cabins as well. Big Bush is the base of the water taxis and the predominant shuttle service in the area, and is also a lot closer to the start (or end!) of the track than Waikaremoana Holiday Park, which is right in the middle between the start and end.

I got the impression that they weren’t particularly impressed with the DOC holiday park - apparently, Big Bush used to be the Waikaremoana Holiday Park, and DOC renamed their park to match, forcing them to re-brand. Between that, and DOC recommending their park on all the track documentation, I believe that the DOC Waikaremoana Holiday Park gets the lion’s share of the visitors.

Full disclosure though, I stayed at the DOC park. Why? Really, it never occurred to me as an option. I left the car parked at Big Bush while I did the track though, and I wish I’d stayed there. It was a pain to travel the 13km (20 minutes on gravel road) between Waikaremoana Holiday Park and Big Bush, especially after finishing the track.

Even without this, I wouldn’t rate Waikaremoana Holiday Park. It’s typically extremely busy during the summer months, and seemed a little cramped in places. Tent site boundaries are strictly enforced, and as a tramper with an early start, it’s not great to be randomly placed next to a multi-family group, all with excitable young kids. So I think - stay at Big Bush. There’s tonnes more room for camping, seems that you can pop your tent wherever you want, and the water taxi van leaves right from where you’re staying!

Transport

Transport arrangements depend on which end of the track you start from, and how you want to get there The most common option seems to be a water taxi from Big Bush (there’s theoretically other providers, but I saw no sign of them). Road shuttles are also available (either from Big Bush, or others), but the gravel roads in the area are downright unpleasant and should be avoided wherever possible. The water taxi is a pretty typical dual-engine type similar to the smaller water taxis used on the Abel Tasman track (they seat about 15). I opted to start from Big Bush, with a short ride in a van packed with people and packs to Onepoto Landing (one end of the track), and then a 20 minute water taxi across the lake to the opposite end of the track and Hopuruahine Landing. This costs $50pp, and included a pick-up at Onepoto Landing a few days later to return to Big Bush and save me from walking the extra 4km down the road to where the car was parked at the holiday park.

One thing that I was unsure about before heading into the Ureweras was track parking. DOC holds no responsibility for vehicle break-ins or theft, which is fair enough - both ends of the track are remote and unmonitored, and there’s not really anything stopping thieves. Because of this, I parked at Big Bush Holiday Park for the duration of the track, so at least there’d be people around. I did see a few cars parked at the track end though, so it does happen. Really, it’s the same conditions as any other tramp - there’s no guarantee of security, but in many ways, the remoteness is in your favour. There’s plenty of parking space at Onepoto Landing, but there wasn’t much space at Hopuruahine. If you’re determined to park at the track end, you’re probably walking from Hopuruahine back towards your vehicle parked at Onepoto.

Generally though, I had no problems with transport. It was smooth, quick, and pretty affordable considering that the $50pp fee covered two van rides, parking for the duration, and the boat ride itself.

Timing

This is one of these tracks where you can spend as little or as long as you want doing it (DOC bookings permitting of course!). Three or four nights seemed to be the common number, and I did it in three, starting at Hopuruahine, staying at Waiharuru campsite, Waiopaoa campsite, and Panekire hut. The obvious effect in how long you’d like to spend doing the track is how long you’d like to spend each day walking. I ended up with a fairly short day the first day of about two hours, resulting in a lazy afternoon at the campsite, but meaning I had a long, seven-hour day the following day to reach Waiopaoa. The final two days were fairly consistent at about 3 hours each. Many families add an extra night to break that 7-hour day up, but conversely, quite a few people seem to do it in two nights with long walking days, and there were two or three people I ran into doing the whole track in a single day.

I feel like the three night option is about right for most people, but it was restricting being required to choose the campsites we would stay at months in advance. Looking at a map, the track between Waiharuru and Waiopaoa looks like it follows the lake and would be pretty flat, but it had a couple of saddle crossings, rocks to clamber over, mud pools, and the terrain undulated right through the day, meaning that it was unexpectedly tiring. If I’d known how tough that day was going to be, I would have stayed at the campsite back from Waiopaoa, at Korokoro, and made up the time the following morning. I guess the lesson here is to consider not only the distance, but also the type of track you’ll be travelling. DOC times tend to be slightly imprecise due to the need to cater to many different walking paces, and I usually run shorter than the DOC time - I wasn’t expecting to take as long as the track guide said it would on the second day, and didn’t have a great time then.

Equipment

I almost always over-equip when I tramp. I’m a bit of an accessorizer, but that’s not uncommon on Great walks I think. I’m slowly coming round to more minimalist view though, which is why I’m writing this section.

If you’re doing a great walk for the first time, this is probably your first serious multi-day tramp. The temptation is to throw everything in you might possibly need, and it’s surprisingly tough to resist (I know, I’ve been there). The trick is to focus on the essentials, and prepare to make-do for anything you might forget to bring.

If you’re camping, a tent and a sleeping mat are pretty important. Remember you’re carrying the tent for hours, not just from your car to the ground. I split our two-person tent with my partner - she takes the poles and pegs, and I take the tent itself. This works out pretty evenly.

Clothes are also important, but you don’t need as many as you think you might. I now take one set of walking clothes (light and breathable), one set of clothes to wear around the campsite (relatively warm and comfortable), and a few clothes designed for particular conditions (thermals and fleece for if it’s cold, rain jacket for rain, sun hat and glasses etc). The epiphany I had was that, after a couple of hours walking with a heavy pack, after the first day - everybody smells. There’s no point taking different clothes to wear each day for a 3 or 4 day walk - just make sure that when you’re going to finish up your day, you’ve got a less whiffy set of clothes to change into for closer quarters, such as a hut or tent, where your stench may be more noticible.

Food is vital (for me, anyway). Try and take lightweight things that don’t need a bunch of ingredients to cook. On this tramp, I took these neat things called Backcountry Meals. They’re full “gourmet” meals that have been dehydrated, so they weigh less than 200 grams for a satchet that feeds two people. To cook, you boil water, add water to sachet, seal and wait for 10 minutes. Delightfully easy after a long day. Each sachet costs about $13-15 each though, so they’re not the cheapest option. Dehydrated food also has a kind of distinctive texture as well, which can get tiring after a few days. Next trip, I’m planning to do more soup & noodle type dishes, and might even splash out and take some veggies. Perhaps just one Backcountry Meal, as a treat, but I found it too much night after night.

I just used my Android phone for photos on this trip, and it did mostly fine. I turned Airplane mode on and the battery lasted the whole trip (I turned the phone off about 7pm each night, but it was on from about 8:30am each morning til then, so lasted well). Don’t take an SLR unless you’re going on a dedicated photography trip and are willing to take care of it constantly! I also took a cool little HD video camera that I have called a Polaroid Cube on this walk. I did way more videoing than I usually do, and I hope that it’ll eventually become my main image AND video capture tool when tramping, since it’s cheap and fairly resilient to damage, but I still don’t quite trust it to not lose anything yet, we’ll see.

I do hope that this post hasn’t become to preachy. When I was researching my tramp on the Waikaremoana track, I was struck by the lack of guidance aside from the official DOC great walk guide, and I hope that this provides just a little useful information when anyone keen to get out there and knock this one off!

Helpful information:

Lake Waikaremoana Great Walk Guide (PDF) - The official guide.
Lake Waikaremoana Walk Page on the DOC website - A bit more information than fits in the guide, likely to be more up to date.
Big Bush Holiday Park - accomodation, water taxis. Seemed like nice Kiwi people.
Waikaremoana Holiday Park - the official DOC holiday park - central between track ends but can be busy and cramped.

Multistep form validations with Rails and Wicked

Tue, 23 Dec 2014 02:00:00 +1300

Note: This blog post was originally posted at the Rabid Brains blog (my employer’s blog).

Multistep forms are the bane of the developer’s existence. No matter how you cut it, the fact that multiple request/response cycles are required to create a single resource goes against the grain of a whole bunch of acronyms representing fairly popular patterns and specifications (e.g. HTTP). Despite that, they’re a pretty well established usability pattern when you just have a tonne of information to collect and not much space to do it in, so it’s well worth keeping a method on hand to throw these types of forms together when it’s gotta be done.

In this post, I’m going to talk about how to create a multi-step form process in Ruby on Rails without throwing all that the framework can offer out the window. In particular, I’m going to discuss the structure the form controller can take, and how to perform incremental validations to provide useful feedback to users as they move through the fields. If you’d rather just look through the code, I’ve developed a demo application using the instructions in the blog post. You can check out the source code, or the application on Heroku.

First of all, I suggest that you don’t set out and try and build the logic for managing the form steps yourself. This part in particular can end up quite messy in Rails, and is always consistent between applications, making it prime for extraction into a Rubygem. Luckily for us, a stable, maintained gem named wicked exists that includes this exact behaviour. Use the gem, read the README, and be glad that your controllers can remain free of the numerous helpers you would otherwise have to write to manage and move between steps.

With Wicked installed, you’re all set to create your controller structure. The key thing to realise here is that if you are adding a multistep form, you are really creating a first-class resource in your application - at least, as far as your controllers are concerned. What I’m getting at here is that you musn’t try and shoehorn the controller actions you will be adding to your controller for your model resource - the fact that there are multiple steps being presented, and different fieldsets being submitted, justifies an individual controller to contain all this logic.

Let’s say the top-level resource is named Pet. We’ll go into the attributes of the Pet model in just a minute, but for now, that tells us that our top-level resource controller should be named PetsController, and be routed like so:

# config.routes.rb
resources :pets, only: [:new, :create, :show, :index]
root to: 'pets#index'

Now we have a controller for managing our Pet resource, we now need to create a controller for managing our Pet resource form. For clarity, this should probably be in it’s own namespace (otherwise, you’ll have a fairly generic-looking FormStepsController). You can call this controller whatever you like, but something representative of what it does would be most useful - perhaps Pet::FormStepsController, Pet::StepsController, or something more high-level, such as Pet::BuildController. For the rest of this post, I’m going to use Pet::StepsController, but you can use what you like - just be sure to make appropriate replacements when necessary. Let’s create that controller now:

./bin/rails generate controller pet/steps_controller show update

You’ll notice that I’ve generated the controller with just two actions - show and update. This is actually all you need for this form - remember, Wicked is handling the step logic for us, and this controller is responsible for a form step, not the resource. This means that our actions map to show-ing a form step, and update-ing the record with the attributes for that step. Let’s add the routes for this new controller now - remember, it will be nested within our top-level resource controller:

# config/routes.rb
resources :pets, only: [:new, :create, :show, :index] do
  resources :steps, only: [:show, :update], controller: 'pet/steps'
end

root to: 'pets#index'

Notice how we can still use resources with our new controller? Rails is going to cleverly pass through the form step in the :id parameter for this route, as this identifies which step to show or update. So, the route to a form step with these routes will look like this:

/pets/1/steps/identity - where 1 is the :pet_id, and identity is the :id. Nice!

Now that we’ve got our controller structure set up, let’s leave this area for now and go take a look at our Pet model - this is where we’ll set up our form steps, and tweak our validation to let us just check validity for a particular step.

Let’s say (for the purposes of having a model complex enough to justify multiple steps), our Pet model has the following schema:

 create_table "pets", force: true do |t|
    t.string   "name"
    t.string   "colour"
    t.string   "owner_name"
    t.text     "identifying_characteristics"
    t.text     "special_instructions"
    t.datetime "created_at"
    t.datetime "updated_at"
  end

We’re going to have three form steps for this model - ‘identity’ (which will collect name, and owner_name), ‘characteristics’ (which will collect identifying_characteristics and colour), and ‘instructions’ (which will collect special_instructions). We’ll define a class-level accessor for holding our form steps:

# app/models/pet.rb
class Pet < ActiveRecord::Base
  cattr_accessor :form_steps do
  	%w(identity characteristics instructions)
  end
end

Now we can access our form steps using the following method call: Pet.form_steps.

Next, let’s set up our validations. We’ll be requiring that all the fields be filled in, but only if you’re on the appropriate step. To check this, we need to implement a method that checks whether certain validations should be run based on what step we’re on. The validation check isn’t too simple - it’s not enough to just check the current step, because validations should still be run on previous steps, and if the current form step is nil. Let’s go ahead and implement that method now:

# app/models/pet.rb

def required_for_step?(step)
  # All fields are required if no form step is present
  return true if form_step.nil?

  # All fields from previous steps are required if the
  # step parameter appears before or we are on the current step
  return true if self.form_steps.index(step.to_s) <= self.form_steps.index(form_step)
end

…hang on a minute - how do we know what form step we’re on at the moment? We don’t! Let’s add an instance-level accessor to our model to store the current form step:

# app/models/pet.rb
class Pet < ActiveRecord::Base
 # ...

 attr_accessor :form_step

 # ...
end

Great! Now we can detect whether we need to run validations for a set of attributes or not, based on which step we’re on. Conveniently, Rails allows us to pass in an if option to each validation that determines whether it should be run or not. We could implement the validations for the first step like so then:

validates :name, :owner_name, presence: true,
		  if: -> { required_for_step?(:identity)

We can go ahead and implement the validations for the other steps the exact same way - I’ll leave that as an exercise to the reader.

Before we head back to our controller, I’m just going to make a very brief suggestion. If you’re building a multistep form yourself your validations are likely to be multiple lines for each step, and possibly even for each attribute. If you do find yourself having to do that, I would encourage you to look into the with_options helper - this will let you define your if option in a single place, and then apply all the validations within the block. Here’s an example, showing each of the attributes for the first step on individual lines:

with_options if: -> { required_for_step?(:identity) do |step|
  step.validates :name, presence: true
  step.validates :owner_name, presence: true
end

As you can see, it’s overkill for one-liner, simple validations, but it’s quite a nice syntax if you’ve got more complex rules.

When we last looked at our controller, it had been generated with two actions - show and update. Now it’s time to add Wicked to our controller, and implement our actions.

The first, and most obvious thing to do, is to include Wicked’s helpers into our controller - as per their README:

class Pet::StepsController
  include Wicked::Wizard

  def show
    # TODO
  end

  def update
    # TODO
  end
end

Once that’s done, we need to tell Wicked what our form steps are. Now, the README suggests putting this into the controller, but we’ve already gota collection of our form steps that we can access in our model. Let’s just use that!

class Pet::StepsController
  include Wicked::Wizard
  steps *Pet.form_steps

  # ...
end

Now our controller is set up to use Wicked, and knows which form steps to use. There’s just one more thing we need to consider before we can test this out - how do we get into our wizard form? Remember, our routes rely on having a Pet ID already present, so we’re going to need to create our Pet instance before we enter the wizard form.

The most appropriate place to implement this is back in our top-level controller. Remember, our wizard controller is responsible for showing and updating steps, but our top-level controller is still responsible for managing our Pet models. Semantically, this means that our create action of our PetsController should create the new Pet instance. This action will work a little differently from a normal create action that you might be used to, as it doesn’t strictly need a new action - we won’t be saving this Pet model with any data - just putting it in the database so that our StepsController can access that.

Your create action can be as simple as this:

class PetsController < ApplicationController
  # ...

  def create
    @pet = Pet.new
    @pet.save(validate: false)
    redirect_to pet_step_path(@pet, Pet.steps.first)
  end

  # ...
end

In other words, create a new pet with no data, and redirect to the first step. If you would like a new action, you can of course add one - this could render a view that explains the form steps, or something similar to that. It’s not required for this controller to work correctly though - you can just point any links to create a new pet straight to the create action like so:

  <%= link_to 'Record a Pet', pets_path, method: :post %>

So, that will handle creating a new Pet for us, and will redirect to the start of the form. We haven’t implemented our show action yet though - let’s do that now.

Our show action needs to find the pet (so that we can use form_for @pet in our templates), and then call a special Wicked method called render_wizard - this will render a template with the same name of our step - for example, if we are on the identity step, this will render “app/views/pet/steps/identity.html.erb”. Here’s what the show action looks like:

class Pet::StepsController < ApplicationController

  def show
    @pet = Pet.find(params[:pet_id])
    render_wizard
  end

  def update
    # TODO
  end

end

Before we give this a test, let’s quickly implement the template for the first form step:

# app/views/pet/steps/identity.html.erb
<%= form_for @pet, method: :put, url: wizard_path do |f| %>
  <% if f.object.errors.any? %>
    <div class="error_messages">
      <% f.object.errors.full_messages.each do |error| %>
        <p><%= error %></p>
      <% end %>
    </div>
  <% end %>

  <fieldset>
    <legend>Pet Identity</legend>

    <div>
      <%= f.label :name %>
      <%= f.text_field :name %>
    </div>

    <div>
      <%= f.label :owner_name %>
      <%= f.text_field :owner_name %>
    </div>

    <div>
      <%= f.submit 'Next Step' %>
    </div>
  </fieldset>
<% end %>

If you visit your first form step now, you should see your identity form. If you see this - congratulations! If you don’t, go back and check that your routes are all correct - remember you need to access the wizard form using a Pet ID that exists in the database.

We can view the form now, but if you try and hit save, you’ll notice you run into an error. This is because we have not yet implemented our update action. This action is actually going to be very simple, because Wicked handily takes up all the heavy lifting of working out where we need to go next in our form. Let’s implement this, pretty much based off the Wicked README:

# app/controllers/pet/steps_controller.rb
class Pet::StepsController < ApplicationController
  # ...

  def update
    @pet = Pet.find(params[:pet_id])
    @pet.update(pet_params(step))
    render_wizard @pet
  end

end

We’re using render_wizard, just like we did in the show action, but this time we’re passing it an object. In this case, Wicked will check the object - if it is valid, it will continue to the next step, but if it is invalid, it will not move onto the next step, but re-render the template for the current step (where errors will be shown as appropriate if you are rendering the error messages like the identity template above).

This looks good, but you may have noticed that we’ve introduced another new method - this one’s to do with strong parameters. If you cast your mind back to when we were working on our model, you may also remember that our conditional validations rely on having the form_step set in the model to work correctly, and we’re not setting that anywhere here. Luckily, we can fix both of these problems by implementing the pet_params method!

This method will use a “case/when” statement (I’ve found Alan Skorkin’s blog post to be a good summary of case/when if you need a refresher) to return the different Pet attributes that can be updated based on the step that is passed in. It will also ‘mix-in’ the current form step to the parameters it returns so that the model knows which step it is on and can run validations accordingly.

Here’s the implementation:

# app/controllers/pet/steps_controller.rb
class Pet::StepsController < ApplicationController
  # ...

  private

  def pet_params(step)
  	permitted_attributes = case step
  	  when "identity"
  	    [:name, :owner_name]
  	  when "characteristics"
  	    [:colour, :identifying_characteristics]
  	  when "instructions"
  	    [:special_instructions]
  	  end

  	params.require(:pet).permit(permitted_attributes).merge(form_step: step)
  end

end

As an example, were this to be called with the first step, it would return: {name: 'Tinkerbell', owner_name: 'Bob Jones', form_step: 'identity'}. It will raise an error and return an appropriate status code if someone attempts to submit an attribute that is not allowed on the current step.

We now have our model set up to perform validations for whichever step we are up to, and our controller set up with Wicked to manage working through the steps. Our form is not yet complete - we are still missing templates for the “characteristics” and “instructions” steps, but the implementation of those is left as an exercise - they will very much resemble the template for the “identity” step, just with different fields.

In terms of where you go from here, there’s plenty of scope for improvement! There is some code in our steps controller that can be refactored to not be repeated, and you might want to split some of the common elements in your step templates into partials (for example, the submit button and/or the error rendering). You might also want to tweak when validation errors should/shouldn’t be run (as an example of this, I have implemented a variant of this form where validations are not run until the final step). You may also want to check out the following resources:

Demo application developed using this blog post: App, Source
Ryan Bates’ Railscast on Wicked
Wicked README

Coming or Going? The North American P-82

Thu, 23 Oct 2014 00:00:00 +1300

By United States Air Force [Public domain], via Wikimedia Commons

This aircraft is literally two P-51 Mustangs banged together.

I entirely appreciate the thinking behind taking a very successful platform together and doubling the awesome, but I can’t help imagining a number of entertaining scenarios in which a pilot in each fuselage attempts to wrest control of the ‘plane off the other.

I’m also imagining a number of flights in which the pilots spend most of the cruising time in giggling fits as they make faces and thumb their noses at each other along with similar hijinks.

There’s good reasons why this type of combination of two fuselages would be successful, but at first glance it’s such a seemingly-odd thing to do that I can’t help and be amused.

For more technical information, see the Wikipedia page.

Sidenote: The first versions of these aircraft really did fly with two pilots - but not at the same time! The pilots would alternate on long journeys to provide relief for the other. Later versions replaced one pilot position with space for a radar operator.

IFTTT, Grabaseat, and You: Getting the flights you deserve.

Tue, 27 May 2014 00:00:00 +1200

This guide covers setting up If This Then That to check for flights on Air New Zealand’s Grab a Seat service. If you’re not at all interested in cheap NZ air travel, this might not be for you.

Grabaseat, Air NZ’s discount flights service, has got to be one of the hottest battlegrounds in New Zealand’s internet. Popular flights sell out in minutes, and it’s up to you to frantically refresh, hoping and hoping to see that exact flight you want, nay, need.

In this post I’m going to explain how to set up If This Then That to do all this frantic checking for you, 24x7. If This Then That is awesome, and useful for so many things (the essence if these ‘things’ being: ‘when this happens, do that’), so I can’t possibly cover everything - instead, I’m going describe this particular trick.

Note that this seems like a long post, but it’s mostly just a lot of detail that you can skim over - in reality, this takes about 5 minutes to set up, and is totally worth it.

Right, let’s get into it. For the purposes of this exercise, let’s assume you live in Wellington, but are a traffic enthusiast keen to check out this ‘Southern Motorway’ you’ve heard your friends up in Auckland chat about. Traffic enthusiasts don’t make much money, so we’re after the cheapest flight possible, and we know that traffic isn’t going to go anywhere while we look for a flight, so we’re in no hurry.

The first thing to do is to sign up for an If This Then That (IFTTT) account - it’s super simple, and will only take a moment - go and do that now. Once you’ve signed in, you’ll see a fun little landing page with pulsing buttons and things that looks a little like this:

You can explore all the pre-built recipes that are there - we’re going to jump right in and create our own. Hit the ‘Create’ link in the top bar.

First of all, we need to specify the ‘this’ part of ‘If THIS then THAT’ - click the bit that says ‘THIS’ to do that, which will prompt you to select something that the IFTTT team calls a ‘trigger channel’.

A trigger channel can be lots of things, but you can think of it as the data source - where the information we will need will be coming from. To find our flights, we’re going to use a handy little service that Air New Zealand makes available to us called an RSS Feed - you can check out the Wikipedia page, but it’s basically a list of links, usually sorted by date, that is easy for computers to understand.

If you have a look in the list of trigger channels, you’ll see ‘Feed’ as an option, with a lovely (?) orange logo:

If you click on that, the page will scroll down in a disorienting way until you reach a selection between two options - ‘New feed item’, which will act every time a new link is added to the RSS Feed, and ‘New feed item matches’, which will only act if that new link matches a search term.

Hit the ‘New feed item matches’ - we don’t want to be alerted everytime a new deal is posted on Grabseat, only when the ones we’re interested in become available.

After clicking on that bit, you’ll be taken to a form where you can specify what IFTTT should look for in new links. What you want to look for is the name of your journey, as Grabaseat shows it on their website - it’s usually City you live in to City you want to go to. Watch out for special cases though - for example, Grabaseat shows flights to ‘Napier/Hastings’, not to ‘Napier’ and ‘Hastings’ - and the search requires that you be exact!

With this example, we’re interested in flights from ‘Wellington’ to ‘Auckland’, so in the ‘Keyword or simple phrase’ box, we should type: Wellington to Auckland.

The RSS Feed has a special address that shows the feed information, and it’s not just ‘https://grabaseat.co.nz’ - it’s got an ‘/rss’ stuck on the end of that (so altogether that’s https://grabaseat.co.nz/rss). Feel free to visit that page in a new tab if you’re curious to see what that looks like. If you’re wondering how I found that out, it’s listed down the bottom of the Grabaseat website - it’s just not very obvious.

With both of those things filled in, your form should look like this:

Now you can hit ‘Create Trigger’, and begin the ‘that’ part of ‘If THIS then THAT’, by hitting the word ‘That’.

Now, an action channel needs to be selected. If the ‘this’ bit was the data source, you can think of this as the data destination. Similarly to the trigger channels, there’s a whole bunch of them, but in this case we just want to be emailed when there are matching results, so let’s select the ‘Email’ option (Note: Gmail is also in that list - don’t select that option, even if you have a gmail account - it’s something else, and not what we want in this case).

Once you’ve hit that, you get one option - to ‘Send an email’ - click on that to proceed.

Now you get a form that lets you specify what text should be used when flights match your criteria and an email is sent to you. The defaults are fine, but if you’d like to add more text, that’s fine too - just type it into the fields. If you’d like to use fields from the RSS Feed, hit the ‘+’ icon next to each field to see what the options are.

Once you’re happy, hit the ‘Create Action’ button to proceed. You’ll be taken back to the ‘If This Then That’ screen, with the trigger and action you have set up shown. The description is just so you can find the ‘recipe’ again in IFTTT, so you can leave it as is, or change it if you wish.

If you’re happy, hit the ‘Create Recipe’ button to lock that in, and then that’s created! The RSS Feed will be checked every 15 minutes, and if any flights match, then an email will be sent to you right away with a link directly to Grabaseat. Easy as!

If you’d like to set up multiple alerts, that’s fine - just hit the ‘Create’ link again to go through the process - the only thing you need to change is your search term when you’re setting up the RSS Feed. As an example, I usually set up another alert for the return flight, so that I can match up both legs of the trip that I’d like to do.

Happy flighting!

— Josh

Sshhh...Secrets in Rails 4.1

Thu, 10 Apr 2014 00:00:00 +1200

With the recent release of Rails 4.1, has come a feature that I’ve really been looking forward to. That feature is automatic support for storing your application’s secrets in a YAML file.

Whenever you have a repository of code, it’s a great idea to always work under the assumption that the code you write could be open-sourced at any time. Keeping this in mind helps to uphold coding standards, comment quality, and generally what goes into your codebase.

Something I see a lot is files that have been committed with data in them that is specific to the project. Sometimes this data is an API key, or a password (this is really bad), and sometime’s it’s just something that is configuration specific to a particular user that will make the project hard for others to use.

The Twelve-Factor App’s rules on config, state that:

Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.

and:

A litmus test for whether an app has all config correctly factored out of the code is whether the codebase could be made open source at any moment, without compromising any credentials.

Obviously then, just putting keys, passwords, or environment-specific information into a repository is undesirable.

Previous to Rails 4.1, keeping this sort of information out of the codebase required undertaking some special steps. The most common option, popularized by Heroku’s implementation of config, is to store configuration in environment variables. To this end, the dotenv gem was created, to centralize where these variables were stored (you could also export them in your shell, but this is a little more difficult to maintain). Another option was to implement your own YAML file that was read in to your application using an initializer. This option is also good for more specific settings that aren’t secrets, but are configuration - for example, my pattern for loading ActionMailer settings from a YAML file.

The key factor here is, however secrets used to be stored, almost all of the solutions suggested around the internet required using something more than just Rails. What the core team have done is to build in support for just one of these solutions, that is going to work for 90% of developers - reading a file named config/secrets.yml, and automatically parsing, and namespacing to your Rails environment, the data in the file.

Here’s an example:

Let’s say I’ve got a recipe-book application named ‘Chef’, that has the ability to log in using Facebook using Oauth via the Omniauth gem. In order to peform the authentication process, I need to provide Omniauth with my ‘app_id’, and ‘app secret’. Clearly, these two values are configuration, so we want to de-couple them from the application. They’re also pretty confidential, as anyone could use these two values to connect to Facebook as our application.

With support for secrets.yml in Rails 4.1, we can simply put these values straight into our configuration file:

# config/secrets.yml
development:
  facebook_app_id: 123456
  facebook_app_secret: s3cret!!!
test:
  facebook_app_id: 1234567
  facebook_app_secret: s3cret!!!

and then access the values in our Omniauth setup:

# config/initializers/omniauth.rb
Rails.application.config.middleware.use OmniAuth::Builder do
  provider :facebook, 
           Rails.application.secrets.facebook_app_id, 
           Rails.application.secrets.facebook_app_secret
end

By default, git will ignore and never commit the secrets file, so the file remains as a local copy only - it’s considered helpful when you have a file like this to copy it, and remove all sensitive data - naming convention is to suffix the file name with ‘.example’ - e.g. config/secrets.yml.example - and commit this file. This gives other developers a template to work with - they can copy the file back to config/secrets.yml, and fill in their own configuration.

The only thing that I’ve spotted missing from the implementation of the parsing of the secrets file is that while the top-level keys of the YAML file can be accessed using dot-notation (as above, where I’ve gone secrets.facebook_app_id), the file does not appear to be parsed in such a way that this dot-notation works for nested keys. I think this would be a great addition as a way to keep this file nice and tidy, if I could for example, have a YAML file that looks like this:

development:
  facebook:
    app_id: 12345
    app_secret: s3cret!!

…and be able to access a secret using the code Rails.application.secrets.facebook.app_id. Hopefully support for nested keys will be coming in a later release!

Rails’ support for secrets is a long time coming, and I’m really glad to see that they’re making things even easier for us developers. There’s a whole lot of other neat changes in 4.1, so to find our more about the new features (and some more information about the secrets support), check out the Rails 4.1 Release Notes!

Keeping secrets in an Android Application

Sun, 16 Feb 2014 00:00:00 +1300

As a (mainly) Rails developer, I’m pretty accustomed to the need for secret keeping within a server-based application, and how it gets done (typically, config.yml’s and/or ENV variables). Getting into Android though, I’ve had the need to integrate with Parse to retrieve some simple data that I don’t want to manage a data service for. Since I want to open source this application, it’s not really enough to follow Parse’s guides and just paste in my application IDs and keys - I need something more adaptable than that.

After doing a bit of reading, I’ve had an epiphany about Android’s neat support for Strings - that they’re not just for internationalization. My conclusion is that a separate Strings XML file, called something like configuration.xml, and git-ignored, is perfect for storing secrets.

The process for doing this is very simple. In any Android application, there will be a folder somewhere called ‘res’ (I say somewhere, because Android Studio and Eclipse generate this folder in two separate places). Within the ‘res’ there is a ‘values’ folder, which normally contains a few XML files, typically containing dimensions, translations, and other generated files.

To store secrets, simply create another XML file in ‘res/values’, called ‘configuration.xml’. The content of this file is pretty much based on what your existing ‘values.xml’ file will contain - except, instead of containing internationalization strings such as button and text field labels and content, it will contain whatever secret configuration you need. An example of this file is below:

<?xml version="1.0" encoding="utf-8"?>
<resources>
    <string name="parse_application_id">xxxxxx</string>
    <string name="parse_client_secret">yyyyyy</string>
    <string name="google_maps_api_key">zzzzzz</string>
</resources>

Once these keys are in the file, Android will automatically merge it into your resources, where you can access them exactly as you would your normal strings. The only adaption you might need is when you literally need a string, rather than a string resource (which is an int pointing to the resource) - for example, the initialization of the Parse library - you can simply use getString() to get a string of your resource, like so:

Parse.initialize(
    this, 
    getString(R.string.parse_application_id),
    getString(R.string.parse_client_secret)
);

If you need your keys in another XML file (for example, your ‘AndroidManifest.xml’), you don’t need to worry about using getString() - you can just use the XML notation for accessing project resources - for example:

<meta-data
    android:name="com.google.android.maps.v2.API_KEY"
    android:value="@string/google_maps_api_key"/>

Since your secrets are now in an individual file, they’re simple to ignore in your source control system (for example, in Git, you would add this to the ‘.gitignore’ file in your repository):

echo "**/*/res/values/configuration.xml" > .gitignore

This process is not bulletproof. As resources, they are somewhat more vulnerable to decompilation of your application package, and so they are discoverable if somebody really wants to know them. This solution does, however, prevent your secrets just sitting in plaintext in source control waiting for someone to use, and also has the advantage of being simple to use, leveraging Android’s resource management system, and requiring no extra libraries.

Note:

I’ve seen encryption suggested around the place as an additional level of security combined with storing secrets in a strings file. The advice on that seems to be that it’s pointless, unless you can find a way to not have your encryption and decryption routine within your code, since the knowledge of how your secrets were encrypted is generally enough to decrypt the secret.

sam solomon social media buttons

Fri, 14 Feb 2014 00:00:00 +1300

I’m a minimalist. When it comes to web design, I value white space, typography and function. People come to your site for the writing, which is why a focus on these elements will almost always result in a good user experience. Adding non-essential elements to a page reduces signal and creates noise. — Sam Solomon

Very much inline with how I feel about design and UX in general.

Source controlling my toolkit

Thu, 21 Nov 2013 00:00:00 +1300

This morning, I created a new repository on Github called toolkit. I’m using it to keep track of the tools I use every day, in order to benefit others who are interested in how I work, and for my own benefit, to be able to easily refer to documentation, source code and links for tools that may be useful for a particular situation.

I’ve tried to do this type of thing many times in the past - sometimes by writing a blog post, or forking or ‘starring’ a repository on Github. There are a few reason’s why I’ve started off with Git repository this time though:

There is a history of the additions and changes that have been made to my toolkit over time.
Github automatically provides a nice, friendly layout to view items in my toolkit.
I can use Git submodules to track open source projects without needing to keep a full (and quickly outdated) copy of these projects in my toolkit.
It’s easy for others to view my toolkit, and even contribute to it (for example, by sending a pull request or lodging an issue), or even to fork it and adapt it to their own needs.
I can put absolutely any content in this repository - whatever works best for me - it could be an image, some markdown, a whole repository (as a submodule), or just a link - in all cases, Github tends to provide a nice friendly display for this content.

My toolkit is, in many ways, performing a similar function to blog post or ‘liking’ or ‘starring’ something, but I think that the semantics are important. Writing a post about something is all well and good, but it tends to get outdated, and buried amongst all the other things that I might blog about - it’s too much work to trawl through a bunch of posts looking for one particular link. Similarly with starring something - just because I star it, it doesn’t mean I use it - it just indicates that I find the project interesting and might follow up on it. My toolkit ONLY contains things that I have already found useful, or know that I’ll need, and doesn’t include any other extraneous material to get in the way when I need to find something.

I’ve made a start by adding a bunch of submodules to projects that I could think of off the top of my head, but I’m already coming up with ideas for more. If you like the concept, head over to Github and check it out - and feel free to contribute your favourite tools for me to try out!

Take advantage of Rails data attribute transformation to clean up your UJS

Wed, 30 Oct 2013 00:00:00 +1300

As a Rails developer, you’ll almost certainly need to do some unobtrusive Javascript in your applications - in fact, if you use Rails’ jquery-ujs gem, it’s almost encouraged!

Without writing any script yourself at all, you can have a form button that disables itself with a message while the form is submitted by simply constructing your button like so:

<%= submit_tag 'Submit', disable_with: 'Submitting...' %>

Which becomes:

<input type="submit" value="Submit" data-disable-with="Submitting..." />

With Rails 4, however, using the disable_with key directly has been deprecated - instead, Rails prefers that you place this attribute within a hash of data - presumably to more closely resemble the HTML attributes - like so:

<%= submit_tag 'Submit', data: {disable_with: 'Submitting...'} %>

Interestingly though, I have noticed that any hash keys placed within the data hash will be transformed into HTML data attributes - and most importantly, you can continue to use symbols, not strings, as the hash keys will automatically have underscores replaced with dashes, so:

<%= submit_tag 'Submit', data: {disable_with: 'Submitting...', collapse_element: '#share-modal'} %>

Becomes:

<input type="submit" value="Submit" data-disable-with="Submitting..." data-collapse-element="#share-modal" />

This is a really neat enhancement, and I’ve already found myself using it regularly. It makes it easier to stick with Rails helpers, even when binding to complex Javascript behaviours, and produces much neater template code.

Finding records by ID with Ransack

Tue, 29 Oct 2013 00:00:00 +1300

Ransack is one of my go-to Rubygems, especially for admin applications. It provides a really simple interface to building up complex filtering and ‘searching’ of many records. Check out the README for more information about how it all works - this post covers a specific problem case of Ransack.

In Rails, we are often subtely educated that the ID of the record doesn’t really matter to anyone - obviously it’s necessary, but for the most part, it just sits there, auto-incrementing. Every so often though, a use case comes along where the ID does matter. When this happens, there’s a bit of a gotcha with Ransack, where searching will just not work.

The reason it doesn’t work, is because the ID column is (usually) an integer column, not a varchar (string) - so many of the search predicates that Ransack applies to the column break (since you can’t do things like ILIKE on an integer). When I ran into this problem, I thought I’d have to delve deep into how Ransack works to figure out a solution - but as it turns out, support for manipulating column data is already built into Ransack, and there was a perfect solution described in this Github issue:

Essentially, Ransack, provides a ransacker method in our ActiveRecord models which can be used for any transformations that are necessary for a particular column. In this case, we’re using some PostgreSQL functions - the inner function to convert the integer ID column to a string, and the outer function to remove whitespace from the string produced by the inner function (since that function pads the converted number with whitespace). This means that each of our ID column values will be automatically cast to a searchable string by PostgreSQL, via Ransack, allowing us to use the following predicates without any further work:

id_cont
id_eq
id_start
id_end

A note on peformance

I should note that this solution may not be the best for your situation - obviously, casting each column value every time a search is done takes more work than just querying the column’s value directly, and so you should consider alternatives. If a fast search is integral to your application, perhaps it would be a good idea to add an id_string column to your database table that mirrors the ID column, but is a natively-searchable string that will not require casting. If you are after a quick and easy solution though, and search is more of a secondary function, then what I’ve laid out above should work perfectly for you!

Parse Date/DateTime from date_select parameters

Fri, 25 Oct 2013 00:00:00 +1300

This Rails helper allows simple parsing of params that have originated from ActionView’s date_select helper method into a Date or DateTime object in a consistent and clean way.

How I built an open source product and SASS collaboratively in less than 48 hours

Wed, 27 Mar 2013 00:00:00 +1300

Last Friday, I headed off to Railscamp NZ with one guy from work, 8 cans of V, and a bag of carrots (my snack food of choice). Today it’s Wednesday, I’ve still got 8 cans of V, a few less carrots, one open-source application that I’m actually happy with (and Travis CI and CodeClimate say is pretty well put together), and one deployed-to-production, used-and-valued-by-a-company SASS based on said application.

About Inquest

Inquest is a bit like StackOverflow - but it’s designed for those questions that aren’t for the public. Just observe in your job how often questions like these get asked:

“How do I do x?”
“Why won’t x work?”
“Is this the best way to do x?”

The idea behind Inquest is to move these questions into a area that focusses and preserves valuable and insightful questions and answers, reducing the need to constantly re-ask the same questions.

If you think Inquest might be interesting to try out, you can either download the source code of the open source version on Github, or if you’re not too technical or want it to “just work”, Send me a Tweet or PM me on Twitter to find out more about an organization account.

About Railscamps

For those who don’t know about it, the general idea of a Railscamp is to disappear off somewhere into the wilderness with a bunch of other developers, and do whatever you feel like - whether that be socializing, running or attending presentations, or playing what seem to be constant and/or long-running games of [werewolf](http://en.wikipedia.org/wiki/Mafia_(party_game). It’s great fun, and a fantastic way to learn new things and expand your technical (or social) horizons.

Timing

When I headed there on Friday, I already had the expectation that I was basically going to be heads-down coding an app all weekend. As it turns out, I more or less spent the first night and all of the next day just socializing and going to talks - I just couldn’t get into coding. I’d already had an idea for a private, host-your-own version of StackOverflow (based on many senior developers I’ve seen who are constantly barraged by the same questions about the same projects), so I had a bit of a plan in my head. Here was my timeline for the weekend:

Friday - Sunday morning: initial build
Sunday morning - Sunday evening: finishing tests, and adding more features.

In reality, by Saturday afternoon, I had serious doubts that I would even bother continuing on the application, let along finishing the initial build before the end of the weekend. I remember talking to my partner on the phone, when asked how it was all going: “Meh - the socializing is good, but I just can’t get into my project”. In the end, I would actually say that starting on Sunday was the best possible motivation for me - it turned from building into a product into what the hell, let’s just see how much we can get through in a day.

Collaboration

One of the real benefits of a Railscamp is it tends to attract pretty damn competent developers, who have basically set aside the weekend to work on stuff. Having said that, this is a area where I really could have done better - after all, “I’m working on an open-source host-your-own question and answer site, but it’s not going that well” isn’t the best pitch - to anyone. Despite that, I was lucky enough to already have one guy on board from work, and had various people helping out throughout Sunday, mostly on questions along the lines of “Am I doing this right?”. In the end, what became obvious to me was the value of everyone involved in building a product being in the same place - not just in the same room, but at the same table, in the same mindset, with the same vision. In the case of Railscamp, this involved 3 or 4 of us sitting at a table, constantly communicating what we were working on, all with the same (albeit not very businessy) vision of let’s just do what we can in a day.

Management

By Suday morning, it was pretty obvious that we were going to have to do some pretty intense work to get to the point we’re I’d be happy with what we’d done - it was going to need more than one person working constantly throughout the day to get everything done that I wanted to.

To help keep track of everything, I borrowed a technique from my employer, and set up a scrum board, based on a tool that I use every day called Trello (I suggest that you check this out if you don’t already use it!) here’s our board by the end of Sunday:

We stuck this up on the wall, and simply got everyone to update it as they went. No daily standup (obviously this wouldn’t have been much use to us!), just a real-time board that was a glance up away when anyone needed to see where a feature was at, or needed something new to work on. Because this board was “public”, we also got a few people wandering past and offering help or suggestions for features, which was a great way of passively brainstorming for ideas beyond those we had already identified as parts we wanted to build it.

Another part of the management of the build was just the collaboaration. Because the board was right there, and so were us developers, we mostly relied on verbal announcements to know where things were at - and this worked really well. Just little things like being able to find out exactly where feature x was at, what technnique was being used for feature y, or whether feature z had been merged fully into master yet, made the whole thing really easy to manage, and I’m really happy with how this part of it went. We also made extensive use of a Github Enterprise instance to manage the code quality and merging with pull requests, which meant that throughout the day, our master branch was (almost!) always working - even if we were building 4 new features at different points of completion in at the time.

Open-Source vs Product

Something that I don’t get to do very much in my day-to-day job is develop SaaS products - so this seemed like a great opportunity. Despite that though, I’m very much in agreeement with Github’s ethos of open-sourcing everything, and so the last thing I wanted to do was get all these people working on a cool little project that was just going to become another product.

What I did instead was arrive at a middle ground - I would develop the core of the application fully-transparently, alongside all the other developers I could find who wanted to work on Inquest, and push this to Github as a ready-to-roll Rails project. Anyone who wanted to use it for anything can download it, install the dependencies, set up the database and use it. To satisfy my desire to work with a ‘product’ though, I have also made some further developments to make a hosted version of Inquest available to anyone who is either not comfortable setting up an open-source version of the application, or does not want to go to the bother of hosting and maintaining their own version (or both!).

The ‘product’ version is exactly the same as the open-source version, but with support for multi-tenancy, and with all data secured and scoped to a company-specific subdomain (for example, http://3months.inquestapp.com). Right now, there’s no company sign-up available, while I get it to a point where I’d be comfortable charging for access - when I do get to that point though, I plan on charging just enough to cover my hosting costs, and hopefully my time to continue adding features to the open-source version of Inquest. I’ve seen this done before, and its worked quite well - helped by the fact that I have a full-time job that can quite easily cover my rent if people just end up preferring the open-source version.

I’m really happy with how I’ve got these two versions working. The open-source version seems to me like a really good idea to help people try out the idea and the application, and will hopefully foster some pretty rapid completion of features (I know that at least a few of my workmates are already looking at building in a few things). The producticized version though will, over time, hopefully have some modest growth, to the point where I can fund my work and offer the best support I can on both versions of the application.

Outcomes

I’m blown away with how much I got done - and really quite pleased with a few developers who stepped into unfamiliar territory to try and get this built. The application’s certainly not finished, but it’s also not spartan - everything that I really wanted to be there, is there, working, and being used. I’ve got a lot of improvements I still want to make, and I’m already taking advantage of the velocity I achieved during the last weekend to power through a few of the tougher features I wanted to start.

Here’s the timeline of what I got done, when:

Friday night: ran rails new, created application, promptly stopped development for the night and went and socialized.
Saturday night: spent an hour doing the ‘Users can log in’ story. Promptly stopped development and went and socialized.
Sunday morning: started development on the application in earnest with the ability to create questions. Designed log in pages.
Sunday lunchtime: ability to create questions, answers, and mark answers as accepted.
Sunday evening: ability to vote on questions and answers, search for questions, and invite users.
Sunday night: worked on commenting on questions and answers, polishing interface. Demonstrate working application deployed to camp server in front of all Railscamp attendees.
Monday morning: travelled home, added multi-tenancy for hosted version (multiple organizations can have completely seperate data).
Monday early-afternoon: purchased domain name, production server and set up production server running application. Done!
Tuesday morning: set up organization account for my employed, was immediately used and continues to be used.

And the overall outcomes of this whole project are:

I have a bunch more knowledge in activity and event tracking, layout and design, rapid development & testing and presenting in front of a large group of people.
I ran a development team at a ridiculous pace, and managed the codebase to a point where there’s not really any really smelly code.
I pushed a complex Rails app as an open-source project on Github, and set it up with Travis CI to automatically run the application’s tests on multiple versions of Ruby, and CodeClimate to automatically analyse the code quality.
I augmented a project I had open-sourced, turned it into a product, and open-sourced it - in a day.
I had fun! And I still have fun!

I can’t really emphasize that last point enough - I went to Railscamp not to develop a kickass project, or product, but to learn new things, meet new people and overall, have fun. I’m really, really happy to say that a kickass product just happened to be a fringe benefit of having all that fun!

If you’d like to try out Inquest, feel free to check it out on Github. If you or the company you work at would like to try it out and see if it’s a good fit with you, feel free to get in touch with me - I’d love to hear from you. If you don’t think it’s for you - that’s fine as well! I’m constantly adding cool stuff to the open-source project though, so if you’re a developer, and you would like to support the project, please star it on github:

The curses of proprietary technologies

Sat, 12 Jan 2013 00:00:00 +1300

As a web developer, I’ve got a pretty good pick of websites - I know what companies are prepared to stick with tried and tested technologies, and combine it with just enough new stuff to make browsing their site pleasant and easy to do.

Every now and then though, I’m forced out of my comfort zone, and need to find information on something else, and when I do so, it’s not long before I come across some really nasty stuff. Sometimes, it’s just a hard to use website, when what I really need is specific product information - for example, trying to find out what BBQ’s Placemakers carry - when I don’t just want to be directed to enquire at the nearest store. The thing that really gets me though is when a retailer completely ignores the open and multi-platform/multi-device nature of the web, and instead chooses to use something that is closed, only works with special software installed, and drives customers away.

Typically, the most common example you will see is Adobe Flash - a decade-old technology that even Adobe is scrambing to get killed off. The example I want to bring to light this evening however is even worse - it’s Silverlight, on a local jeweller’s website, Micahel Hill Jewellers.

Silverlight is a piece of software that Microsoft built to compete with Flash, back when Flash was really popular. It provides the ability to read files from the user, gather input from webcams, microphones and animate content. It also requires that it be installed on each computer that needs to use it, and does not work on Google Chrome, or very well on Mac at all (Indeed, when I DID install Silverlight, Firefox blocked it, as it causes problems for the browser itself.

This, generally, is a case of the wrong techology for the job. Michael Hill is a retail jeweller - that means that they want as many customers checking out their products as possible, both in stores and online. I imagine online purchasing must seem tempting for them, but to me it seems like a dead-end - jewelerry purchases are typically expensive, and as such, it’s a better idea to focus on providing really good customer service support while you browse online (such as being able to ask dedicated support staff about specific products).

By depending on Silverlight, Michael Hill is making the wrong technology decision. Because they are using Silverlight, their (potential) customers can’t check prices or products on their iPhones, Android or any mobile device. They also can’t access the site from public or shared computers that they cannot install software onto, or do not wish to install software onto. In a way then, Michael Hill can claim the dubious honour of having one of the least accessible public websites available online.

Certainly, the capabilities of the site do not indicate that any sort of framework such as Silverlight is necessary, but what is the alternative?

Publishing with ActiveRecord: a guide to encapsulated code

Mon, 17 Dec 2012 00:00:00 +1300

It’s not at all uncommon to be asked by a client “So, I’d really like to be able to edit that bit of the page just there - can we do that?”. The answer, of course, is yes, but managing content has no end of complexities. Just one of these many is publishing - that is, controlling content, where it is visible, and to whom. In this blog post, I’m going to describe how I worked to move code out of several applications into a shared library, and the trials and tribulations I had ensuring this eventual Rubygem worked with as many combinations of other Rubygems as possible.

Moving Code

The first step to properly encapsulating code is to identify specific code functions that can be moved out of individual codebases. In the case of the has_publishing library, I identified the processes of publishing a record, creating a draft record, marking a record for future publishing and the querying of all the above states to be a common functionality that was required in a number of other sites. Once this functionality was identified, it was no big deal to copy-and-paste relevant bits of code into a new gem file structure, generated by the lovely bundle gem command. Of course, because this code had been pulled in from a number of different locations, it was still very disorganized and messy. Fortunately, I had the next step:

Refactoring Code

Once the code is gathered together in a central library, it’s all set to be tidied up, tested, and pushed out for the world to enjoy. For a gem that adds functionality to another application, the best way to rapidly prototype how things may fit togther is to set up a quick Rails app, and add the gem as a local dependency:

# Gemfile
gem 'has_publishing', :path => '../has_publishing'

This way, any changes you make to the gem are instantly reflected in your testing harness. The other part of refactoring code is to ensure that the code gets covered by comprehensive unit tests.

Refactoring code can take different forms, depending on the state of the code! Largely, you’ll be looking for duplicated or overly-complex code that can be merged into common behaviour, or any other code that you don’t feel is ready for the public to see.

Testing Code

Testing code in a gem is always more tricky that testing within an application, because you don’t have an exact context to test against - just behaviour. Typically, you need to do a bit more set up to have an environment that provides just enough context to run your tests against.

The easiest way to do this for this sort of project is to replicate the same sort of support that Rails provides for testing models against a database, in as little code as possible. Here’s how I set up a database-backed model for testing against in has_publishing:

# Establish a database connection 
# to a local SQLite3 database
ActiveRecord::Base.establish_connection(
  :adapter  => 'sqlite3',
  :database => File.join(File.dirname(__FILE__), 'has_publishing_test.db')
)


# Create the data table for the model
# I'll be testing against, and then
# manually migrate the database to this
# point 
class CreateTestModels < ActiveRecord::Migration
  def self.up
    if ActiveRecord::Base.connection.table_exists? "test_models"
      drop_table :test_models
    end

    create_table :test_models do |t|
      t.datetime :published_at
      t.datetime :embargoed_until
      t.string :kind
      t.boolean :dirty
      t.references :published

      t.timestamps
    end
  end

  def self.down
    drop_table :test_models
  end
end

CreateTestModels.migrate(:up)

# Add the testing model, complete
# with has_publishing
class TestModel < ActiveRecord::Base
  has_publishing
end

With this testing harness in place, it’s pretty simple to make assertions about how that model should behave with has_publishing - it’s simply ActiveRecord. The reason I prefer not to take the approach of many Rails-exclusive gems and generate an entire dummy application to test with is that I feel like this approach is much more transparent, and just enough set up to get the tests passing, without getting in the way.

Bootstrapping with Rails engines

Although this gem is framework-agnostic (only requiring ActiveRecord), I do want to make it easier for people using Rails to add publishing to their models. Because of this, I have added a Rails generator to my gem. This generator can add the migration to add publishing attributes for any model with a single command, making it much easier to get the correct database schema set up quickly.

Rails engines are related, but not the same as generators. The easiest way to think of them is pluggable sub-applications - they may include extra functionality, change existing functionality or perform some special function, but they are typically just bundles of controllers, models and views, often bundled with generators to set all of these objects up within a Rails application.

Adding generators to a gem is actually very easy to do, but it did have a couple of gotchas to ensure that Rails actually picks up the generator and makes it available for use.

Gem Compatiblity

HTML5 Notifications

Tue, 02 Oct 2012 00:00:00 +1300

On the Spot is a pet project that I tend to develop with bleeding-edge features in mind - something a bit more volatile than Latter, which is used heavily enough to justify a more cautious development process. In this post, I’m going to detail how I added HTML5 notifications to On the Spot, the upsides and downsides, and the Coffeescript class that does it all.

So, HTML5 Notifications are an interesting little API, in that they’re not a formal specification by W3C - in fact, this API was first implemented by the Chromium project, and it seems to be a feature that will continue on the fringe of modern browsers only (i.e. - this probably isn’t coming to Internet Explorer).

They are quite powerful though - it’s a non-intrusive way of letting the user know something has happened, without reverting to an alert(), or having to load a Javascript library, which will still only work if the page is currently visible.

Adding the notifications though, it actually very simple - all the heavy lifting is done by the web browser - all that we need to do is:

1. Request permission to display notifications

2. Check that we have permission to display notifications

3. Set up the notification

4. Display the notitication

Permission needs to be requested, since you’ll be displaying notifications even if the user is not currently looking right at the page. We need to check that we have permission before we try and display anything to avoid errors, and then we can actually show the user the notification.

My Implementation

I was able to centralize the processing of Notifications in On the Spot by writing a Coffeescript class that handled the permissions (request & check), set up and display of notifications. This class is easy extensible later on if I need to add any new notifications as well. Here it is!

OnTheSpot.Notification = {
	setup: ->
		window.notifications = window.webkitNotifications unless window.notifications
		if window.notifications && window.notifications.checkPermission() == 0
			OnTheSpot.Notification.supported = true
		else
			OnTheSpot.Notification.supported = false

	requestAccess: ->
		window.notifications.requestPermission();

	trackPlaying: (track) ->
		return unless OnTheSpot.Notification.supported == true && track

		window.notifications.createNotification(
			'/assets/apple-touch-icon-precomposed.png', 
			track.name,
			'On the Spot :: Track Playing'
		).show()
}

The flow of this class is:

Add a document.ready event listener in jQuery, and call the requestAccess function, when a button is clicked (you must request access from a button or link click, otherwise the browser will not actually ask the user for permission)
Call setup - this function performs some checks to calculate whether we’re able to use and display notifications.
To show a notification, just call trackPlaying with the track data - the class will set up the notification and display it.

Downsides

The main downside to notifications is browser support. Unless your users are running on Safari 6+, Chrome for Android, or Chrome, notifications won’t work for you. Firefox docs indicate that some support for notifications may be in the pipeline, but more likely this support is going to be in Mozilla’s efforts to support ‘desktop’ web apps. Internet Explorer, Opera and other mobile browsers have no support at all, with none planned.

The other downside to the notifications is that they are (unsuprisingly, but still), only going to work when the page generating the notifications is open in a tab somewhere. Page focus isn’t necessary, but the generating page must be open. There might be a way to get around this by using Web workers, but I’ve got no concrete proof that this works.

Upsides

The main upside to notifications is that they reflect the native theme of the OS, are expected and anticipated behaviour from the user, and are generally easy to do, without compromising on look or feel.

They are also easy to generate, and ‘float’ apart from the actual browser application. This means that you don’t need to have the browser visible on the screen to see notifications. They also work without the page being visible (which is required for all other notification techniques to work, which tend to style a div on the page to look and behave like a notification), and, on OS X, appear in the Notification Center, which is pretty neat - users can click on the notification to go back to the application.

Overall, notifications are pretty cool, albeit with the proviso that they’re not used for anything important. They do require user opt-in to work, but once this permission is granted, they are simple and easy to use.

Secret Keeping with Rails

Sun, 30 Sep 2012 00:00:00 +1200

Ever needed to store some secrets in Rails that you don’t want to share with the world? Yeah, same! In this post, I’m going to outline a really simple way to store your application’s secrets in a file called config/secrets.yml.

Rails configuration and you

First of all, let’s discuss exactly how this is going to work. If you’ve even configured a Rails app before, you may have noticed that you’re setting your configuration on a config object inside a block that looks a bit like this:

MyRailsApp::Application.configure do |config|
  config.compile_assets = false
end

Something you might not know, though,is that these configuration settings are then available on an object called Rails.configuration. What we’re going to do is add an initializer that loads a secrets file, and then loads them into the Rails.configuration object so that we can access them in our application.

The advantage of this method versus something like Foreman is that:

This method is much closer to Rails conventions - for example, instead of using a dotfile, it uses a basic initializer
It doesn’t require running your application through any special interface like Foreman does
It just does one thing, and does it well - storing configuration.

Reading the config file

As I’ve mentioned, we’ll be reading a file called config/secrets.yml. Let’s get our application set up to load this file.

NB: For this example, I’m assuming you’ve got a Rails 3.x application set up - you don’t need any models, controllers or anything else - we can test this from a rails console.

First of all, add the configuration file:

development:
  facebook:
    app_key: 123abc
    secret: 123abcdef

- as you can see, this secrets file just contains some credentials to use with Facebook OAuth - it can contain as many keys as you like, however.

Next we need to add a gem to our Gemfile. This gem is called Hashie, and extends Ruby’s Hash object to do some cool stuff - in our case, we can pass it a hash, and it will give us method access to our secrets - in other words, we can do this:

Rails.configuration.secrets.facebook.app_key

Instead of:

Rails.configuration.secrets['facebook']['app_key']

As a bonus, if you’re using Omniauth and any Omniauth strategies, you’ve probably already installed Hashie as a dependency of one of these - we’re just doing to re-use it here, so let’s add it to our Gemfile and be specific:

# Gemfile
# ...
gem 'hashie'

Next, let’s add an initializer to load this file in config/initializers/01_load_secrets. The “01” part is important; it tells Rails to load it before any other initializer runs (since we might need our secrets straight away in an initializer):

# config/initializers/01_load_secrets.rb
MyRailsApp::Application.configure do
  # Note: Hashie is a gem dependency
  config.secrets = Hashie::Mash.new(YAML.load_file(Rails.root.join('config', 'secrets.yml'))[Rails.env.to_s])
end

I’ll just talk through what this is doing:

First, we open a configuration block for our application - this gives us access to a config object we can set keys on
Next, we load the YAML file we created, but only pull in keys for the Rails environment we are running in - this let’s us have keys for test, development, etc. in a single file.
Finally, we wrap that YAML file load in a Hashie::Mash. Hashie is just a collection of Hash extensions that, in this case, gives turns our Hash (which is what YAML.load_file returns) into an object with method access (so we can go config.secrets.facebook.app_key)

That’s all we need to do to load the config in - as you can see, it’s actually a really simple operation, and use of Hashie means that we don’t need to mess up our code later down the track with lots of square brackets and hash keys.

Using configuration

Once we’ve got the file loaded, actually using the configuration is actually really easy - just access your top-secret secrets on Rails.configuration.secrets.

For reference, here’s how you might set [Omniauth] to support logging in with Facebook using our new secrets.yml file:

Rails.application.config.middleware.use OmniAuth::Builder do
  provider :facebook,
  	Rails.configuration.secrets.facebook.app_key,
  	Rails.configuration.secrets.facebook.secret 
end

Redesigning for readability - a retrospective

Tue, 25 Sep 2012 00:00:00 +1200

I’ll be the first to admit that I’m not that great at design, especially typography and detailed elements. I’m just one of those people who can see what they want in their heads just fine, but something just goes…wrong…between the head and the page. In this post, I’m going to explain some of the lessons I learned while focussing on making this blog clearer, more consistent and more readable in general.

My old blog - something wasn't quite right Open Full Size

My old blog wasn’t too bad - in fact, I’d had a few people profess to rather like it. The problem I had with it is that it felt like a cheap, low-quality knock-off of blog layouts like Zach Holman’s. While I’d made some efforts to make things a bit different and more readable, something wasn’t right about it - I just looked at it, and it didn’t look great.

So, I decided to go back to grassroots - I’d start from a basic blog layout - this time, based on Tom Preston-Werner’s site, but rather than trying to replicate what was there, I’d go my own way. In the end, I ended up changing just about everything from that original layout, but in a good way - it feels like my own now.

Early design decisions

I do a lot of drawing, when I go to ‘design’ something. Really, I’m just trying to fool myself into thinking I’m some sort of ‘designer’, doing ‘wireframes’ - but I’m not, because my drawings are a pile of rubbish (see example). What I do end up doing though, is thinking as I doodle, and take notes across the page. By the end of the process, I have a pretty good idea of what I change.

My 'wireframing' process Open Full Size

Some of the key things I decided were:

The body font had to go - what I had hoped would be a lovely, slightly old-fashioned-but-still-somehow-modern look turned into something that looked just a tad messy. Along with the copious use of sans-serif fonts elsewhere on the site, there was a lot of inconsistency here
My blog posts always end up long. I should add in more images to break up the content and make things easier to read
I didn’t like the listing of posts on the homepage - I felt that it was an unnecessary impediment to visitors seeing what content I have for them.
I refer to Github projects a lot - I should be able to ‘feature’ them somehow for those readers who don’t know what I’m talking about.

Typographical Inspiration

As I’ve said, I liked the way that mojombo’s blog had a nice, slicked-back look to it, and decided that this was the sort of thing I wanted to pursue. I also decided that since consistency and readability was a priority for me, I would focus on trying to get post body copy looking the best.

I had stumbled across Mark Boulton’s blog a few days previously, and took some of his lessons on board from there - in particular, I took the measure of my old blog, and found that the lines were a bit too long - between 80 and 90 characters. I also found that the line-spacing (Sorry, leading), didn’t quite match the word-spacing.

Once I had some ideas for basic typographic improvements, I then had a look around Google Web Fonts to find some feature fonts for headings and important copy. I really liked the look of Tom Ward’s blog, which used a nice clean serif font for the headings that contrasted well with the sans-serif body copy. Eventually, I stumbled across Arvo, which fitted my needs perfectly.

Since my blog is powered by Jekyll, I also took a look through the list of Jekyll sites that are maintained on the wiki. This had some typographical and design stuff, but most of all, it allowed me to explore what might be possible to add to my posts and my homepage. It’s here I got the idea to add my ‘Featured Projects’ section to the bottom of some of my posts, and to show a ‘snippet’ of content for the latest 5 blog posts on the homepage, along with a list of all posts further down the page.

Implementation and Experimentation

I used LESS for implementing the CSS for the new blog layout and typography. I’d originally begun using LESS with Twitter Bootstrap (which enhances ‘my’ design capabilities significantly).

I set the base font-size on the body element, so that I could use relative font sizes from there on, and instructed the page to use that workhorse of fonts, 'Arial', 'Helvetica', sans-serif. I did try a few other fonts, but in the end settled on Helvetica, largely due to:

Standardization - it’s an expected and familiar font for readers
Adaptability - it’s suitable for a broad range of contexts and uses
Legibility - Helvetica/Arial remains legible at a range of sizes

To fix up the measure (which should be about 70 characters across), I used a smaller single-column layout. I used a width of 50%, which at a width of 1280px, yields an average measure of just over 70 characters. I prefered a fluid width for this column, so that it can scale a bit more gracefully between resolutions, and I added a media query or two to change the width to 90% once the resolution drops below 767px, so that at that size, I’m not wasting any page.

After these few improvements, everything was looking way better. As is the way with these things though, discovering the best fit for all of these, along with some other minor tweaks, took the best part of a day.

What I Like

I do like the refactoring work I’ve done overall. I feel like everything is a bit more utilitarian and back-to-basics, but in a good, clean way. It still looks professional and makes it clear that I can wrangle CSS, without being too much for what it is.

I’ve placed more images on a couple of my posts, and I also like how this is working. What was previously mostly drab posts (whose content is fascinating, I assure you!), has now been broken up a wee bit.

I also think that the way I’m featuring Github projects is working well, and will hopefully help people out a bit if they’ve just scanned through a post and want to find out more about a specific project I’ve mentioned.

The last thing I’m happy with is the homepage. I was a bit worried about achieving the right balance here, but I think that it’s about right. I now ‘feature’ the latest 5 posts on the homepage, with, hopefully, just enough content to get visitors interested in clicking through to the full article, but just below that I also include a full listing of all posts. This means that for an casual visitor, there’s info right there for them to find something interesting, and for the return visitor, there’s a way for them to refer back to a previous post without having to leave the homepage to dig through an archive.

What I don’t like

There’s not actually too much - what I don’t like are things that I can fix in improvements later down the track.

The main thing I’d like to have is captions underneath images. As I’ve mentioned in previous posts, I use Markdown for writing posts, and markdown happens to not support the <figure> or <figcaption> tags, which are perfect for an image/caption scenario. I write this markup in HTML for some blog images, but I’d really like it to happen straight off a Markdown image tag. I should be able to fix this by adding some javascript to wrap each image tag in a figure element, with the caption being based on the image’s title attribute.

Another thing I’d like to do is properly attribute images and resources I use. A lot of the images that I use are in the public domain (usually out of Wikimedia Commons), but some do require proper attribution, which I’m not currently giving. I’m still thinking about the proper way of doing this - I may be able to just add it to the image caption, once I’ve got my figure tags appearing properly, or I may do something like the ‘Featured Project’ boxes I’ve got at the bottom of posts that reference Github projects. Hm.

Overall, I’m quite pleased with this. There’s certainly some improvements to be made, but I feel like I’ve built up something that is a bit less proprietary and based on previous work, and a bit more tailored to me. Along the way, I’ve even managed to learn a few things about typography, LESS, and advanced Jekyll functionality! Win!

Got any comments about more improvements I could make? I welcome tweets or private messages on Twitter - @sudojosh

Process configuration management with Foreman

Fri, 31 Aug 2012 00:00:00 +1200

Often in the Rails projects I build, I have a need to store configuration that I don’t really want to have in my code. The most obvious way to do this is to either store the configuration in a YAML file (e.g. config/settings.yml), or in environment variables.

Foreman is Heroku’s answer to these types of configuration challenges. It is used internally at Heroku to manage processes and their configuration.

Managing Configuration

If you’ve ever worked with Heroku, you’ve probably seen the heroku config command. When you run this command on an active Heroku project, you see output like this:

=== latter Config Vars
DATABASE_URL:                 postgres://xxxxxxxxx:5432/db
GEM_PATH:                     vendor/bundle/ruby/1.9.1
HEROKU_POSTGRESQL_PURPLE_URL: postgres://xxxxxxxxx:5432/db
PGBACKUPS_URL:                https://backups.heroku.com/client
RACK_ENV:                     production
RAILS_ENV:                    production
SHARED_DATABASE_URL:          postgres://xxxxxxx

This configuration is actually managed by Foreman, and is stored in a file called ‘.env’ in the root of the project. The .env file for the configuration above would look like this:

DATABASE_URL=postgres://xxxxxxxxx:5432/d3o4ndc7b0g39o
GEM_PATH=vendor/bundle/ruby/1.9.1
HEROKU_POSTGRESQL_PURPLE_URL=postgres://xxxxxxxxx:5432/db
LANG=en_US.UTF-8
PGBACKUPS_URL=https://backups.heroku.com/client
RACK_ENV=production
RAILS_ENV=production
SHARED_DATABASE_URL=postgres://xxxxxxx

The cool thing about Foreman is that you can use this .env file just as easily locally. All it does is reads the file when you run a command, and loads these variables into your shell as environment variables.

Managing Processes

The other functionality that Foreman provides is the ability to run and manage multiple processes. This is something that you may or may not have run into in Heroku, but it’s still pretty cool. Without Foreman, if you had a Rails server, a PubSub server (Faye, for example), and a daemon to run, you might have to do the following:

New terminal tab, run bundle exec rails s
New terminal tab, run bundle exec rackup faye.ru
New terminal tab, run bundle exec script/daemon start

…and then you need to keep an eye on each of these tabs.

With Foreman, you create a file called a Procfile. This stands for ‘Process file’, and is a simple listing of the processes you would like to run. For example, on-the-spot’s Procfile looks like this:

web: bundle exec rails s 
pubsub: bundle exec rackup ./private_pub.ru -s thin -E production
daemon: bundle exec ./lib/daemons/spotify_controller_ctl run

When you run foreman start, Foreman will boot up each of these processes, and output the combined logs for all in the window. When you stop Foreman (By hitting Ctrl-C), it will shutdown each of the processes before exiting. This process is much easier to handle than multiple terminal tabs.

Installing Foreman

There’s two ways to install Foreman:

Via the Heroku Toolbelt. Heroku publishes a ‘Toolbelt’ - a collection of tools required to develop applications that run on the Heroku platform. I don’t use this method myself, as I prefer to have control over what’s installed and where, but you can install this from https://toolbelt.heroku.com/ if you’d like
As a gem. You can just run gem install foreman to get it installed - easy as that. If you’re using RVM, then you may want to follow my pattern - I normally place Foreman in my global gemset along with the heroku gem, so that it accessible to all my apps. If you’re just experimenting with Foreman though, you may want to place it in it’s own gemset.

Using Foreman

Foreman is actually pretty easy to use on a day-to-day basis - there’s really only two commands you need to run:

foreman run [cmd] - You can run any command with Foreman, doing all the environment variable setup, etc. beforehand. If you require any variables in your application’s setup routine, then you will need to run all your rails * and rake * commands with this, otherwise you can just run it when you need access to these variables.
foreman start - Use this command to start all the processes listed in your Procfile, and just Ctrl-C when you want to stop them.

Easy as that really. There’s a bunch of other things you can do with Foreman (such as exporting launchd and init.d scripts to run your application automatically), but they’re not really something you’ll need to know to get started.

Debugging Ruby with Foreman

A common complaint I hear about Foreman is that it’s quite complex to debug application’s in. This is a consequence of the way that Foreman combines the running of all of the processes in your Procfile into a single output flow - debugger’s can’t easily make it through. I’ve found there’s a couple of ways to debug with Foreman though:

Run a Rails debugging server with foreman run. Since foreman run doesn’t do any of the fancy Procfile stuff, you can just run foreman run rails s --debugger when you do need to debug - it will behave just as it always has.
Use a debugging console directly in Foreman. I’ve had mixed results with this, but if you do run with Foreman start and have debuggers in your code, I have noticed that these breakpoints do get hit - it’s just that Foreman doesn’t always output the prompt characters for the debugger. You can still type in irb to drop into an IRB session, and everything will behave as it usually does from here - it just requires a keen eye to spot when you’ve landed on a debugger in Foreman (since there won’t be any output)
Run a remote instance of ruby-debug and connect that way. Ruby’s debugging support includes the ability to run a debugging server from your application, that you can connect to from another terminal window or tab. To do this, you need to do a bit of setup in an initializer, but once it’s there, it’s pretty seamless:
- In config/environments/development.rb, add the following lines:

    # Wait for remote connections from rdebug
    Debugger.wait_connection = true
    
    # Listen for remote connections
	Debugger.start_remote

In a new terminal window or tab, run the rdebug program to connect to your remote debugging instance:

 	rdebug -c

(More info on this Github issue)

I do prefer the first two options to the last, as I find that code changes seem unnecessary for this, and I normally aren’t needing to debug all the time. Largely though, this is up to personal preference, and I wanted to present all options.

Conclusion

Foreman is a pretty awesome tool for development, especially for those apps that may have some complex configuration and process management needs. I don’t use it for all my applications (since I’m a strong believer in picking just enough of the right tools for the job), but I do find it incredibly handy to extract all my configuration and processes into external files that I can easily update and generally DRY up my code.

Information: This blog post is based on a lightning talk I delivered at 3months for the dev team. 3months is an agile-based company based in Wellington, New Zealand. We have fun. Interested in working for us? Give us a shout!

Building On-The-Spot: A Spotify play queue

Sun, 12 Aug 2012 00:00:00 +1200

About the Application

The idea behind On the Spot was to allow everyone in the office at work to control the playlist, so that there was some semblence of fairness about what we listen to. Probably the simplest way of showing you what it is is to see some pictures, right?

Part 1: You can search for music Open Full-Size

Part 2: You can see what's playing, and what's about to play Open Full-Size

So, we basically have a two-page app - one page for finding music, another for playing it. This whole thing runs on a server in the office, and is DNS-d to http://spotify.office.3months.com, making it accessible to all staff.

Technology Choices

The technology choices for this app ended up being pretty clear cut, although as with all projects I do in Rails, I do sense that I could have found a lighter approach.

In the end, the core technologies are:

Rails 3.2
No database
Redis
Hallon
Daemons Rails
Bootstrap for UI (customized)
Font Awesome for Icons

When I started the project, I considered a couple of projects that I thought might make the ‘real-time’ part of the application, mostly (Sinatra and Cramp). I figured though, that as I wanted to add extra features, the drawbacks of these frameworks were going to get in the way - I opted to use Rails, accept the performance hit, in favor of not having to risk the framework getting in the way too much.

Searching

Searching was the first feature I worked on - here’s a reminder of what a search looks like:

A reminder about searching Open Full-Size

I actually tried two different ways of searching Spotify - the first was Meta Spotify, and the second was Hallon. I ended up sticking with Hallon, as I was using this library to also play the files, however I did that Meta Spotify’s results seemed more accurate.

The way I implemented the search was actually very simple - I have a SearchController, with a new action which looks a bit like this:

  def new
    @results = Rails.cache.fetch "search-for-#{params[:q]}" do
      Hallon::Search.new(params[:q], :tracks => 25, :tracks_offset => 10).load.tracks.to_a.map do |track|
        {
          :name => track.name, 
          :artist => track.artists.first.name, 
          :album => track.album.name, 
          :popularity => track.popularity, 
          :uri => track.to_link.to_str
        }
      end
    end

    respond_to do |format|
      format.json { render :json => @results }
    end
  end

In human terms, I perform the search, restricting the number of results to 25 for performance, and then pull a subset of the information available to me out, and return the data as JSON.

The other side of this is the Javascript - once again, quite a simple and common implementation of a typeahead search. The code for this is a bit more convoluted, so I’m going to post a shortened and annotated version of the main bit of it:

# This function takes results that are returned from an AJAX getJSON() call to the search action
handleSearchResults = (results) ->
	tracks = []

	# We only want to add unique tracks to our array
	for track in results
		tracks.push track unless _.include(tracks, track)

	# Now that we have the raw data, we iterate through the
	# collection and build an HTML list item for
	# each track
	tracks = _.map tracks, (track) ->
		$("<li></li>")
		.append($('<span></span>').addClass('track').text(track.name))
		.append($('<span></span>').addClass('artist').text(" - #{track.artist}")) 
		.append($('<span></span>').addClass('album').text(" (#{track.album})"))
		.data('popularity', track.popularity)
		.data('uri', track.uri)
		.prepend($('<i></i>').addClass('icon-music'))
		.append($("<button></button>")
			.addClass('btn btn-success play-btn')
			.attr('title', "Play: #{track.name}")
			.append($('<i />').addClass('icon-plus')))


	# We want to sort the collection by the 'popularity'
	# that Spotify records. This helps pull
	# popular songs to the top, and push Karaoke/Knock off
	# versions to the bottom
	collection = _.sortBy tracks, (item) ->
		return 1000 unless item.data('popularity')
		console.log "Track: #{item.find('.track').text()}, popularity: #{(100 - item.data('popularity'))}"
		return 100 - item.data('popularity')

    # Don't add anything if we have nothing in our collection
	return if collection.length == 0
	
	# Empty the last set of search results and add 
	# the new ones.
	$('#tracks').empty()
	_.each collection, (item) -> 
		$('#tracks').append(item)

So, the combination of these two code blocks is enough to give us a typeahead search of tracks on Spotify. The search itself is trivialized by the Hallon library, and so the only real concern for us here is performance - each search takes about 500ms, and each key stroke performs a search.

Queueing Tracks

Once we have our search results, clicking on the ‘+’ icon next to the track queues it. The queue is a simple Redis list, which I push onto from the ‘left’ side, and pop items off from the ‘right’ side. This makes the controller action to queue a track as simple as this:

  def create
    position = $redis.lpush "tracks", params[:uri]
    flash.now[:notice] = I18n.t('queue.create.success', :position => position.ordinalize)
    render :status => :created
  end

Queueing is one thing, but we still need a way to interact with this queue, and play tracks through Spotify. This is where the daemons-rails gem comes in!

Playing Tracks

‘On the Spot’ has two components. The first, we’ve discussed - it’s the Rails application, that we use for searching and queueing tracks. The second is a daemon, which runs in a loop, grabbing the next track to play, and playing it using Hallon - here’s the main bit of this code:

  # Keep running while there are tracks to play
  while uri = $redis.rpop("tracks")
  
    # Only continue if the URI looks like a spotify one
    if uri =~ /\Aspotify\:/


      # Set the currently-playing track
      $redis.set "currently_playing", uri

      # Load the track
      track = Hallon::Track.new(uri).load

      begin
        # Pre-load the track
        $player.load(track)
        
        # Play the track (blocks until complete)
        $player.play!(track)
      ensure
        $redis.del "currently_playing"
      end
    end
  end

In a similar way to searching, much of the work is done for us. Really, all this code is doing is pulling a Spotify URI (pretty much an ID) from our Redis list (called “tracks”), storing it as the value of a Redis key (called “currently_playing”), so that we can display which track is playing, and then actually plays the track (using Hallon). Once the track has finished playing, we un-set that currently playing value.

Future Features

Really, all I’ve done so far, then, is built a web-interface around the functionality afforded by the Hallon gem (which is itself a wrapper around Spotify’s API library, libspotify). Despite this though, it does work exactly do design. The next thing that needs doing is the ability to control the player - right now, there is no way to pause or stop the player, except by actually logging into the Redis console and clearing the list by hand. I’d also like to add some more realtime capability using Faye or Pusher, and provide browser notifications of playing tracks.

Conclusion

This has ended up a bit more long-winded than I meant it to be, but has hopefully touched on the process I’ve gone through to build this app - in total, I’ve spent around 12-15 hours getting this to its current state, including writing this blog post, and altogether, it’s working pretty well. In the future, I hope to write further about adding realtime notifications and features to this application, so stay tuned!

VirtualBox Debians with additional adapters

Tue, 31 Jul 2012 00:00:00 +1200

I get stung by this one all the time, and I can never remember how I fix it. I’m noting it down here, in the hope that next time I’m googling for a fix I’ll stumble across my own blog post.

Adding network adapters (or refreshing the MAC address) of a virtual machine running a Debian-based OS on VirtualBox causes a strange problem - the new interface is not listed in the guest after booting, and the network interface device is simply not present.

The fix for this is very simple. Effectively, Debian ‘caches’ some interface information like the MAC address (which isn’t really supposed to change), in a file that gets loaded on boot. When the MAC address DOES change, Debian effectively ignores that interface.

If you see errors that seem to be causing a missing network interface after adjusting Network adapter settings for the VM in VirtualBox, try the following fix:

sudo rm /etc/udev/rules.d/70-persistent-net.rules

This file gets recreated on boot, so it’s perfectly safe to delete - if you’re feeling cautious though, you can also just move it somewhere else. Once you’ve deleted it, just restart the VM, and you should have the correct number of networking interfaces ready to go!

Whakamāori te Latter

Mon, 30 Jul 2012 00:00:00 +1200

Introduction

I’ve spent the last week translating Latter (really badly) into Māori using Rails’ I18n support. Since this is the first time I’ve done a full site translation, I wanted to detail how I did this here.

First, I’ll quickly explain how Rails I18n support works. Essentially, it’s a key-value store - you specify a ‘key’ which represents a bit of text like this:

I18n.t('player.create.success')

The I18n module then takes this key, and looks up the ‘value’ that is stored in a YAML file, that, for this string, might contain a snippet like this:

player:
  create:
    success: 'Player was created successfully'

It then substitutes the request for the string value for the current language, with the string value in the translation file (in this case, ‘Player was created successfully’)

The main advantage of I18n is centralization. Moving all the messages and small bits of text used all over the site makes it much easier to change content, try out different wording in certain areas, but most significantly, to offer support for multiple languages globally by changing just a couple of files.

Adding Te Reo Māori to Latter

The process of adding Te Reo to Latter was pretty simple - just time consuming. Initally, the bulk of the work was just finding translatable strings of text, and adding English translations for them, so that I could later add support for Māori. This process goes much more smoothly when you know where to look:

Flash messsages in the controller
Links and buttons
Model-based forms
Navigation and Layout

The next challenging bit that I didn’t realize would be difficult is the organization of keys into a logical structure where particular strings are going to be easy to find and maintain. The strategy that I chose to stick with is the following:

Resource - the name of the resource, for example, Player, Game, Score. I preferred singular forms of names, as these more closely relate to the source class or model
Action/Category - the controller action to which the translation belongs, or, if the string is for a specific area of the layout or a page, then an identifier for this idea
Identifier - a key that identifies the string uniquely.

For example:

# Resource
player:
  # Action
  update:
    # Key
    success: 'Player updated successfully'

By this point, I had the bulk of the application translated into English - that is, rather than having key messages, labels and actions specified in the HTML markup, I had moved it into config/locales/en.yml - the file that Rails loads when the application starts, and where it looks for translations.

The Current State of Play: Latter, translated to English Open Full-Size

New state of play: Latter with Te Reo Māori translations Open Full-Size

The next step, then is to add another language. This process in itself is actually very simple. Simply copy the en.yml file to mi.yml, and begin translating!

cp config/locales/en.yml config/locales/mi.yml

I found Māori quite a hard language to translate, for a few reasons. I had no prior experience with translating content, and a fairly weak grasp on Māori language constructs and syntax. Because of this, I wouldn’t be at all surprised to discover a range of minor errors in the translations, however I did find that having a central file to refer back to, as well as a fairly standard and simple vocabulary made this process much more simple.

The translation process itself is very simple. Given a translation within config/locales/en.yml like the following:

en:
  game:
    index:
      heading: All Games

then the Māori equaivalent in config/locales/mi.yml would look like the following:

mi:
  game:
    index:
      heading: Katoa Kēmu

– the keys for the translation never change, just the content, typically with one file being added for each new language.

Latter is the first Rails app that I’ve added full support for translations to. Māori was a really important language for me to add, as I feel that it’s a key part of the New Zealand identity and culture. Conveniently, it also tied in with Māori Language Week.

Overall, the translation project took about a week’s worth of evenings and parts of the weekend - maybe about 15 hours all told, but now that the initial setup work is complete, adding additional languages is simply a matter of translating the strings already present. This process could have been sped up further, had Rails had build-in support for Māori - while the rails-i18n project has translations for many languages, the absence of Māori meant that I had to provide my own translations for some areas. This is an area where I see a key opportunity to add I18n translations for Māori to Rails, however I don’t feel that my Māori language skills are adequate enough to provide a quality contribution.

Internationalization is an often overlooked part of most web sites and applications, especially in New Zealand, where Te Reo Māori is one of our three official languages. It’s worth noting that only a fraction of the non-government websites and applications I have seen that are intended for a New Zealand audience offer Māori as an alternative language, which is really something I’d like to see in the future.

I hope that this post has explained the translation process clearly, and perhaps may even help drive more internationalization efforts with Rails in the future.

Compiling with Homebrew without XCode

Sun, 29 Jul 2012 00:00:00 +1200

I’m a big fan of not installing XCode on new Macs. I just don’t think it’s necessary, unless you’re genuinely building OS X or iOS applications.

Instead, I prefer to download the [Apple Command Line Tools] straight from Apple’s Developer Center, and install everything else I need to with Homebrew. Here’s how to do that.

First of all, download and install those command line tools. You’re going to need those first of all.

Next, install Homebrew. There are instructions on the site, but the quick version is:

/usr/bin/ruby -e "$(/usr/bin/curl -fsSL https://raw.github.com/mxcl/homebrew/master/Library/Contributions/install_homebrew.rb)"

Once homebrew install installed, you need to install another version of gcc (A compiler). There is a version of this that already comes with OS X, but it’s pretty old.

Enable the dupes homebrew package, and install an up-to-date version of GCC:

brew tap homebrew/dupes
brew install apple-gcc42

Theoretically, you now have the compilers, utilities, and tools you need to set up a range of packages on your Mac. One last problem that I’ve run into installing packages such as zsh and node, however, are programs that are run from the compilers to try and find XCode - and fail when they find it is not installed. The error looks something like this:

xcode-select: Error: No Xcode is selected. Use xcode-select -switch <path-to-xcode>, or see the xcode-select manpage (man xcode-select) for further information.

The solution for this is to tell these tools that Xcode is ‘installed’ in /usr/bin - where all the compilers are installed:

sudo xcode-select -switch /usr/bin

This stops xcode-select from erroring out and halting the compile process, by ensuring that it tries to find the compilers that it requires by looking in the directory where they are already installed.

Once all these steps are completed, you should have a full set of developer tools installed - all you need to compile almost all of the many Homebrew recipes that are available.

Note: The original instructions for setting up command-line tools and gcc-4.2 come from: http://robots.thoughtbot.com/post/27985816073/the-hitchhikers-guide-to-riding-a-mountain-lion. The xcode-select fix comes from this issue report on the Homebrew repository

Sublime Text 2: Do not reopen files

Thu, 19 Jul 2012 00:00:00 +1200

For quite some time now, I’ve been using alloy’s fork of Macvim as my primary editor, along with janus, and it’s been working out really well.

I’ve just started trying out Sublime Text 2 though, and it’s been pretty nice (although I still have reservations). Something I can’t stand in a developer application though, is for an application to re-open all the files you had open last time the application was used. Sure enough, this is something that Sublime Text 2 does.

To turn it off, you can change the following settings in your user preferences file. To open these preferences, open Sublime, click the application menu, and then go to ‘Preferences’ -> ‘Settings - User’, and add the following keys:

"hot_exit": false,
"remember_open_files": false

If you haven’t already added any user settings, these two lines will need to be surrounded by curly braces - { and }.

I don’t understand why any application needs to re-open files - if I close an application, that means I don’t need it anymore. If I needed it again, I would just leave it open, or suspend my computer and come back to it later.

Hopefully these settings make things a bit easier on somebody else though - I couldn’t find this documented anywhere on the internet from a quick Google, so had to jot it down here.

Document your code with Tomdoc

Mon, 25 Jun 2012 00:00:00 +1200

Documentation in Ruby on Rails apps tends to be somewhat of a hit-and-miss affair, particularly within non-product organizations. In this blog post, I’ll run through how Tomdoc has helped my code become clearer and easier to maintain, with very little overhead.

First, a small explanation about Tomdoc. It is effectively a spec that dictates a way of writing inline documentation in comment blocks, in a similar way to ri and rdoc, but without some of the overhead. It was created by Tom Preston-Werner of Github fame, who felt that tools such as RDoc required too much verbosity and effect to properly document code.

Tomdoc has been written specifically to be read by humans, unlike other documentation systems that are generally optimized for machine parsing and formatted display. Tomdoc makes the very sensible assumption that in most cases, the person reading the documentation will be looking at the code, and so removes the need for any special kind of markup.

The Tomdoc spec has a detailed example of what a Tomdoc documentation section may look like, so I’ll skip over the big example, and instead drill down to how I use it to write more maintainable code.

Effectively, I use Tomdoc the same way as I would an RSpec integration test - within my documentatation, I not only explain the API of the method, but also the context of how and where this method is used - for example:

  # Public - Record that this deal has been viewed by a user
  #
  # This method facilitates a given user having 'read' and 'unread'
  # deals. When a deal is displayed to the user, it should call this method,
  # which will add a DealView record for the card and user id, if one
  # does not already exist.
  #
  # user - The user who viewed this deal
  #
  # Returns the found or created dealview
  def viewed_by!(user)
    DealView.find_or_create_by_deal_id_and_user_id(:deal_id => self.id, :user_id => user.id)
  end

Within this method, I not only define the visibility (Public) of the method, and provide a short description of what the method does, but I also describe briefly where and how the method is used. I then go on to define the parameters that the method accepts, and what the method returns.

By commenting my code in this format, I achieve benefits in number of areas:

It becomes easier for new developers to understand the relationships between different areas of code
The documentation forms a contract which future changes to the method should conform to
Writing the documentation forces me to think about the best way to code the method, and how it may need to interact with other areas of the application

Overall, I’ve noticed that I’ve had a much better understanding of how components of applications I’ve been working on interact since I started documenting the code I write, and have been able to pass on this understanding to colleagues and others - that for me, indicates that this type of documentation works really well with an opinionated framework like Rails, and a untyped language like Ruby, providing a flexible specification to build system knowledge without getting in the way or slowing things down.

Just as a sidenote, I’ve also forked and made some minor improvements to an existing fork of tomdoc, that adds the ability to generate formatted documentation from a source tree. My change adds the ability to pass in a folder to parse documentation from, and adds nicer formatting and navigation controls. You can see my fork at https://github.com/joshmcarthur/tomdoc

Import Trello cards from a CSV file

Thu, 31 May 2012 00:00:00 +1200

This morning I wrote a quick script that I found quite handy - it takes a CSV file, and adds cards to Trello from it’s contents. At 3months, we quite regularly get long spreadsheets with requirements, so this script saves us a lot of time when we need to import these lists into cards so we can estimate and expand on them. Now, with a bit of reformatting, replacing proper quotes with double quotes, etc., I can just import in a couple of minutes.

I’ve documented the script pretty well, but basically it uses the ruby-trello gem along with Foreman (for managing quite a few ENV variables that get passed around) to use an existing Trello OAuth authorization to add cards to a board and label them (since you may want to apply a ‘from-spreadsheet’ label or something like that). It probably won’t handle special characters wonderfully, but the main thing is just to make sure that your CSV is a valid CSV file - escaped quotes, quoted columns etc.

I hope somebody else may find this useful!

Jekyll Content Liquid Tags

Fri, 18 May 2012 00:00:00 +1200

For the last week after deploying my new blog to joshmcarthur.com instead of a ‘blog’ subdomain, and implementing a new design and back-to-basics blogging engine, I’ve noticed that my markdown wasn’t being parsed properly (/at all).

Originally this was hard to spot - my posts had been migrated from Blogger, to Tumblr, out of Tumblr, transformed from HTML to Markdown, imported into an Octopress blog, and then transferred to this blog. After this process, most of them weren’t in very good shape anyway, but it was still hard to spot because my embedded gists were still being rendered - in fact anywhere where I had used actual HTML markup was fine.

After transferring markdown engines, a lot of Google and Stackoverflow searching and all-round frustration, I noticed that Jekyll mentioned two Liquid tags that could be used to interpolate content into a page:

content, and
page.content

I had considered these to be the same - one was just a namespaced one, like a convenience method. I was actually wrong about this, as they perform two slightly different functions:

The content tag outputs the contents of the file being built, and also processes the contents of that file - i.e. if it is a Markdown file, it parses the Markdown, if it is Textile, it parses the Textile, etc. etc. It also scans the output for other Liquid tags and does any further processing needed for these.

The page.content tag, on the other hand, just outputs whatever the content is. It’s basically just the first part of the content tag - the file reading bit, without any of the processing.

I had been using page.content in my layouts, as I liked the idea of the namespaced tag better - as soon as I changed this to use content everything worked. I just wish it hadn’t taken me a week to sit down and figure that out, instead of blaming everything else but me!

Bootstrap Tablesorter Styles

Mon, 14 May 2012 00:00:00 +1200

When Twitter Bootstrap first shipped, it came with a couple of handy integrations with external jQuery plugins. Once of the more popular of these integrations was styles that were compatible with the classes, elements and attributes added by jQuery Tablesorter.

Recently, however, these integrations have been removed from Bootstrap, as it was felt that they were distracting from the core of what the project was trying to achieve, and that as most plugins were themeable anyway, users of Bootstrap could just build a theme.

This is all well and good, but sometimes it’s just a quick fix that is necessary. Here’s the CSS needed to provide Bootstrap look-and-feel in the table sorter plugin:

table .header {
	cursor: pointer;
}
table .header:after {
  content: "";
  float: right;
  margin-top: 7px;
  border-width: 0 4px 4px;
  border-style: solid;
  border-color: #000000 transparent;
  visibility: hidden;
}
table .headerSortUp, table .headerSortDown {
  background-color: #f7f7f9;
  text-shadow: 0 1px 1px rgba(255, 255, 255, 0.75);
}
table .header:hover:after {
  visibility: visible;
}
table .headerSortDown:after, table .headerSortDown:hover:after {
  visibility: visible;
  filter: alpha(opacity=60);
  -moz-opacity: 0.6;
  opacity: 0.6;
}
table .headerSortUp:after {
  border-bottom: none;
  border-left: 4px solid transparent;
  border-right: 4px solid transparent;
  border-top: 4px solid #000000;
  visibility: visible;
  -webkit-box-shadow: none;
  -moz-box-shadow: none;
  box-shadow: none;
  filter: alpha(opacity=60);
  -moz-opacity: 0.6;
  opacity: 0.6;
}

Add this stylesheet into your project to add the tablesorter integrations from Bootstrap 1, to Bootstrap 2. It doesn’t mess with the default table styles or anything else, but does make the tablesorter feel much more integrated.

(The original discussion on this issue is from: this Google Group thread)

Git-browse for Quick Repo Viewing

Tue, 28 Feb 2012 08:50:00 +1300

Last night I quickly patched together a command called git-browse - it’s a small but handy extension to git that looks at the remotes you have set up inside a repository, and opens up the first Github repository it finds - to give an example:

-> git remote -v
origin git@github.com:joshmcarthur/WriteIdeally.git (fetch)
origin git@github.com:joshmcarthur/WriteIdeally.git (push)
heroku git@heroku.com:writeideally.git (fetch)
heroku git@heroku.com:writeideally.git (push)
-> git browse
(Opens https://github.com/joshmcarthur/WriteIdeally in default browser)

l The source is just below - at the moment, though it’s only compatible with OS X and Linux - Ubuntu that I know of, but I believe that RedHat won’t work without installing packages - see this StackOverflow thread

Installation is pretty basic - just download, make it executable by running chmod +x git-browse, and then copy it somewhere that is referenced by your $PATH variable (you can see what’s in this variable by running echo $PATH in a Terminal). Once you’ve done this, you should be able to enter any repository with Github remotes, and run git browse to have the repository open in your default browser.

Extensions to this idea should be pretty easy - something I would be keen to see in a pull request, for example, would be to support opening arbitrary files - i.e. git browse lib/writeideally/api.rb would open https://github.com/joshmcarthur/WriteIdeally/blob/master/lib/writeideally/api.rb. Having said that, my gsubbing is pretty gross, so that needs refactoring as well.

In any case, enjoy!

Treatme Lite: Part One - the Backend

Wed, 08 Feb 2012 21:07:00 +1300

The Backend

This article first is in a series of three blog posts documenting how I created TreatMe Lite, an HTMl5 web app using Zepto.js, Coffeescript, and a range of other frameworks and tools, all backed by an Express JS web service. See my previous post for more information about the application.

Last post (the intro), I left off explaining why I set out the build the application, and the technologies I had selected to build each part. This post is going to detail the steps I took to create the web service using NodeJS and Express.

First of all, I needed a way to serve the deal JSON. In actuality, this is something that I could have done just as easily directly from the front-end Javascript, however, I’d started off with a NodeJS application, and it seemed just as sensible to request the JSON from TreatMe using Node’s support for HTTP requests.

To kick off the node app, I cloned a starter application from Github:

git clone git://github.com/twilson63/express-coffee.git treatmelite

This application gives us an easy base with support for Coffeescript, Jade templating and a bunch of other CSS stuff, like the Skeleton grid (which I’m a big fan of).

To satisfy the requirements for the front-end, my NodeJS application needed to perform two tasks:

/deals/:latitude/:longitude: Retrieve a list of nearby deals from TreatMe

The code:

Explanation:

First, we make the HTTP request, passing a callback that will get a response object as an argument
We listen for certain events while we receive the data from the response - if we hit an error, we return an error message. We listen for when we receive some data, and append it to the string we want to return to the front-end, and when we reach the end of the data, we return this string.

/geocode: Geocode an address, and pass back the information we need to display to the user

The Code:

Explanation:

We build the path to the Google Geocoder using either the latitude and longitude from the user’s current location (i.e. reverse geocoding), or the address (typically a city name like ‘Wellington, New Zealand’)
We follow a similar pattern to the Deals action, requesting a geocoding result, waiting for a response, and parsing the data we need out of that response. In this case, we build a JSON structure containing the geocoded address, it’s latitude, and it’s longitude.

A note on Geocoding: We use Google’s Geocoding service to retrieve the information we need from either the name of a location, or the latitude and longitude (either way, it fills in the information we need). Once again, this is something we ideally would do from the front-end, however, I ran into some cross domain issues, and so simplified matters by performing the request on the server

Treatme Lite: Adventures in Javascript

Mon, 06 Feb 2012 21:01:00 +1300

For a long time, I’ve been looking to move into the NodeJS trend that’s been taking the development community by storm. The problem is that coming from a Rails development background, and my inability to follow callbacks through more than 2 levels, has lead to my previous efforts to end in broken code and frustration.

Happily, however, I took another look at express, a Sinatra-like framework for NodeJS, and it clicked with me on a level that vanilla Node had not in the past. It supported middleware, was reasonably well-documented, and I was able to find a handy project on Github which allowed me to start from a solid base of proven middleware and NodeJS modules to help me out (I typically now develop in Coffeescript for my personal projects - I enjoy the ability to code in something that takes my missing knowledge of how to write beautiful Javascript into account).

Next, I needed something to build - ideally something I could become reasonably passionate about finishing, but that also would not consume more than a week or so of working in the evenings. Something I’ve been using recently is TreatMe Now - a sort of customers-on-demand voucher system for retailers that is updated throughout the day with some really good deals - typically, these last only a couple of hours though, so it’s important to check quite often. Something else I had lamented is that, while there is an iPhone app available, there was no mobile web support, no support for other devices, and the normal web interface is chock full of data-heavy maps and unfriendly panels that are, frankly, downright annoying to use on a smartphone.

Using Chrome’s web inspector, I was able to find a JSON feed of deals, and work out how to filter the results to a particular region. Given this datasource, I was ready to begin building out this experiment, using a number of resources I’ve wanted to try out in more detail - NodeJS, Express, more in-depth HTML5 things, responsive design, and an adaptive grid system (Normally I use Twitter Bootstrap, but I made a conscious effort to keep things light, using Skeleton’s adaptive grid system, and Zepto.js in place of jQuery). This blog post is going to detail how I went about building backend and fronted of the application, to the point where I’ve got The Final Product, an express-backed single-page application supporting offline access, local storage and a fully responsive design.

Over the next three posts, I’m going to be documenting how I went about building a responsive and HTML5-based web application to serve TreatMe Deals, using NodeJS, Express, ZeptoJS, Coffeescript and the Skeleton CSS framework, including difficulties I ran into, and how I solved them:

Part One: The Backend - an examination of my basic web service written in Express which serves a JSON datasource of deals, and makes calls to Google’s Geocoding service
Part Two: The Frontend - a look at the Javascript and CSS tools I used to develop the frontend of the application
Part Three: Responsiveness and Extras - some detail about how I added support for offline asset caching and local storage of deals, and made additional tweaks to make the application work fully offline.

Upcoming: RateMyCourses

Mon, 14 Nov 2011 20:16:00 +1300

Tonight I’d like to talk about a project I’ve been working on in my spare time for the last few months. I’m super excited to see it coming together, and it’s nearly ready to go live.

The application is called RateMyCourses - put simply, it’s a way for University students to explore, rate, and comment on courses they’ve done to help out others. Inspired by such projects as Fix my Street and Fix my Transport, RateMyCourses aims to increase the transparency of quality courses, lecturers, and tertiary education institutes within New Zealand. Over the last few months I’ve been not only building up this application, but also generating the dataset of courses thats at the heart of the application from scratch, pulling course details from all major Universities within New Zealand, including:

Auckland University
Massey University
Waikato University
Victoria University of Wellington
University of Canterbury
Lincoln University
Otago University

All up, I’ve got nearly 20,000 courses indexed, from these seven institutions. This is a huge opportunity for students to really engage with each other, with their Universities, and with new entrants. Over time, I’m hoping RateMyCourses will become a valuable tool for course planning, as it uncovers the best courses, institutions, and educators in New Zealand (and, eventually, other countries).

This is just a preliminary post while I polish up some areas, and make sure I’m really happy with things, but I welcome feedback and comments as I move into the launch and adoption phases. I’m planning to launch in the upcoming weeks, and spend the summer with a core group of users ironing out the major issues, before really pushing it at Victoria next year.

Add jQuery to Any Page Really, Really Easily

Sat, 12 Nov 2011 21:27:00 +1300

When I’m working on a site, or analysing somebody else’s, I often wish that jQuery was loaded into that page - it’s an unbeatable tool for really digging into the site’s source to debug something or work out how something has been done.

To help out with this, I’ve put together a quick bookmarket, similar to the more fun one I’ve done a while ago. It works extremely simply - it just creates a script element, adds jQuery 1.7.0 (served from Google), and appends it to the page body. As soon as the script has loaded, jQuery can be used from within Web Inspector, Firebug, or whatever other Javascript console you use.

Here’s the bookmarklet:

Add jQuery

To add this to your browser, you can either: right-click on this link, select ‘Bookmark Link’, or just drag-and-drop onto your Bookmarks toolbar. Once it’s added, you can simply click on the bookmark on any page, and jQuery will be loaded into the page for you to use.

Here’s the source:

// Create the script tag
var s = document.createElement('script');

// Set the necessary attributes on the tag
s.type = "text/javascript";
s.src = "https://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js";

// Add to the body tag (assume here we're dealing with HTML, and there IS a body tag)
document.getElementsByTagName('body')[0].appendChild(s);

Generating 'Gem Install' Commands From 'Gem List'

Thu, 03 Nov 2011 16:34:00 +1300

Here at 3Months, we have a couple of monolithic projects that have been around for yonks - because of this, they don’t have bundler set up, and many of them have incomplete or out-of-date gem requirements. Yesterday I needed to get one of these projects set up locally for the first time, so I was kindly lent the output of gem list with which to install the gems at the correct versions required.

To make this problem a little easier in the future, I wrote a quick script to generate a massive shell command to install all the gems recorded in gem list. Next time I, or someone else, needs to set up one of these types of project, I can run this script, and hand them the generated shell script. They can run this in their gemset of choice, and install all gems required - much easier!

Here’s the script - alternatively, you can check it out the Gist:

#!/usr/bin/env ruby


### gem_to_command.rb ###############################################################
# Produce a command to install the gems you currently have installed (using gem list)
# Makes a simple Shell script that can be run on Linux or Mac OS X
#
# Author: @sudojosh
######################################################################################

gems = `gem list`
File.open("install_gems.sh", "wb") do |script|
  script << "#!/usr/bin/env ruby\n"
  gems = gems.split("\n").map { |gem|
    gem = gem.split(' ').map { |part|
      part.gsub(/[^\w\.\-]/, '')
    };
    [gem[0], gem[1..-1].min]

  }
  gems.each { |gem|
    script << "gem install #{gem[0]} -v=#{gem[1]}"
    script << " && " unless gem[0] == gems.last[0]
  }
end
puts "Wrote install_gems.sh"

Here’s what it does:

Get the output of gem list
Open a .sh file for writing
Iterate through the gem list output, and parse from the file the name of the gem and the lowest version number (lowest is safer than highest, even though there should only ever be one)
Assemble a gem install command with the --no-rdoc --no-ri arguments, and write this to the file.

Enjoy!

A Fun Little Bookmarklet

Thu, 20 Oct 2011 13:28:00 +1300

I’m sure this has been done before, but after noticing 1-day’s ‘Look Busy’ feature, I just had to write a bookmarklet to load this up on any site!

If you just want to try it out, here’s the link Look Busy Bookmarklet - either right-click on the link and ‘Copy URL’ and paste into a new bookmark, or drag to your bookmarks bar or folder. It should work in Chrome, Safari, Firefox and IE7-9

Here’s the source code for anyone interested in seeing how this works:

// Create a new div tag to contain our image
var busy = document.createElement('div');
var body = document.getElementsByTagName('body')[0];
var max_width_cache = body.style.maxWidth;
body.style.maxWidth = '100%';
busy.setAttribute('id', 'lookbusy');

// Add a background image, and position the div tag above all other content and make it fill the screen
busy.setAttribute('style', 'position: absolute; z-index: 1000; width: 100%; height: 100%; top: 0px; left: 0px; right: 0px; bottom: 0px; background: #FFF url(http://www.1-day.co.nz/images/2010_mission_critical_development_strategy.png) no-repeat 0 0;');

// Add a inconspicuous link to close the image
var close = document.createElement('a');
close.setAttribute('class', 'busy close');
close.setAttribute('style', 'position: fixed; z-index: 1001; right: 10px; bottom: 10px; background-color: #FFF; color: #000; font-size:10px');
close.setAttribute('href', '#');
close.innerText = "OK, Clear";
close.setAttribute('onclick', "javascript:body.removeChild(document.getElementById('lookbusy'));body.style.maxWidth = " + max_width_cache);

// Add the close image to the look busy div tag
busy.appendChild(close);

// Add the look busy div tag to the body of the document
body.appendChild(busy);

Give it a go! It’s super handy for when you’re checking out TradeMe, Failblog, or anything similar!

Safely Start and Stop VirtualBox VMs with init.d

Wed, 19 Oct 2011 17:05:00 +1300

Recently I’ve rolled out a virtual machine host box to run headless VMs (headless means that there is no display, keyboard etc plugged into it), as these make great test and experiment machines for trying new things out. As part of this rollout, I mashed a couple of blogs and some documentation together and came up with this script:

#!/bin/sh

# virtual_machines	Start and stop virtual machines when the host changes state
#
#
VMUSER=administrator

case "$1" in
	start)
		echo "Starting VirtualBox VMs"
		if [ -f /etc/virtualbox/machines_enabled ]; then
			cat /etc/virtualbox/machines_enabled | while read VM; do
			  sudo -H -b -u $VMUSER /usr/bin/VBoxHeadless -startvm "$VM"
			done
		fi
		;;
	stop)
		echo "Saving state of VirtualBox VM..."
		cat /etc/virtualbox/machines_enabled | while read VM; do
		  sudo -H -u $VMUSER /usr/bin/VBoxManage controlvm "$VM" savestate
		done
		;;
	*)
		echo "Usage: /etc/init.d/virtual_machines {start|stop}"
		exit 1
		;;
esac

exit 0

I wrote this because this isn’t something that is supported directly by VirtualBox, and it’s essential that on a headless server that these virtual machines be able to go up and down happily without harm and reliably. It supports start and stop explanations and uses savestate, rather than poweroff - essentially, as the host server goes down, it will ‘hibernate’ each VM, and then restore when the server starts back up again.

Another feature that I built into this was the use of a file in which a list of VMs can be specified to start and stop - for example, if you only want this script to deal with a subset of the virtual machines you have set up on the host server.

An example of this file might look like this:

winxp_development
win7_development
ubuntu_11_10_development

It works really well for me, and since it’s something I had trouble tracking down, hopefully I can make someone’s life a little easier with this solution.

Action Mailer Interceptors

Thu, 06 Oct 2011 16:51:00 +1300

ActionMailer Interceptors are a great way to test the full stack of your mailing in Rails from the generation from data through to receiving the email in your client. They are similar to ActiveRecord’s before_x-type callbacks, and let you change something about the message being sent right before it’s actually dispatched.

The most common purpose I use these for is to redirect mail being sent from the application to either my Inbox, or some shared account (if it’s on a project where I’m not the only developer). This lets me make sure that the HTML in the message is displayed properly, and that the data gets injected into the content the way I expect it to be.

Here’s a quick example of how to use a Mail interceptor for this purpose:

# lib/development_mail_interceptor.rb
class DevelopmentMailInterceptor
  def self.delivering_email(mail)
    mail.subject = "<#{mail.to}> #{mail.subject}"
    mail.to = "me@mywork.co.nz"
  end
end

# config/initializers/mail_interceptors.rb
require 'development_mail_interceptor'
ActionMailer::Base.register_interceptor(DevelopmentMailInterceptor) if Rails.env.development?

…and give one of your mailers a go.

This is a very simple example, of course, but there are a lot of uses for this type of thing - logging & auditing, checking against quotas, analysis, etc. etc.

What really triggered this post though, was a odd problem I came across when trying to get this interceptor to work. An ActionMailer method can be triggered using one of two methods: either deliver or deliver! - the main difference between the two is that the second will throw exceptions if it cannot be sent, which is why I tend to prefer using it. Something to keep in mind though, is that using deliver! will call any registered Mail Observers, but not Interceptors - meaning that your mail will be sent unaltered.

It really was a frustrating process to debug, but after looking at the Mail gem source code (ActionMailer back-ends onto this gem for it’s mail setup and delivery processes), in particular the Message class, I noticed this crucial difference between the two. Resolving the issue is, of course, as simple as using deliver - without the exclamation mark. After that, your Interceptors will be triggered just as they should be.

Overriding Action Caches

Fri, 30 Sep 2011 11:48:00 +1300

Recently, I have been working on a web application that is quite media rich, and is expected to run into quite a bit of traffic. I’ve been working on building an API for a front end system, using JSON to handle passing this data back and forth.

Obviously with this amount of traffic, and the size of some of the JSON collections we were having to marshall and store via Rails, we were going to need some pretty intense caching to reduce the load on Rails, and our database server. We had to balance this need, however, with the requirement that data should be live, or close to live (i.e. around 5-10 minutes) - in particular, statistics, which are calculated on-demand for several responses.

The strategy we chose to manage this balance was to use Rails’ provided caches_action method to cache our JSON responses, building up a cache key from certain parameters, as well as some meta-data, for example, the user’s logged-in status. Because we were using memcached, we could use the :expires_in option to tell the memcached store to expire the cached value after x minutes.

This approach worked for a while, but we found we had a pretty major problem - while the data was cached, it went alright, but as soon as the cache expired we were having loads of users hitting a response that took way to long to build (before we optimized queries, 30+ seconds). So, we needed another fix.

To fix this problem, we tried out adding some cron tasks that used curl to ping the cached URLs, to try and preload the cache so that less users would be hitting the DB. This only partially fixed the problem though, so we identified a solution that would work a little better for us.

What we wanted to do was to leave our existing caching in place - aside from the expiry, it was working fine, and we didn’t want to rework everything. With this in mind though, we needed a way to force a refresh of the data in the cache external from the controller. What we ended up implementing was a monkeypatch on Rails’ caches_action-related methods, that allows us to pass in an :overwrite option - this can be a Proc, or just a boolean - basically, when the value of :overwrite is true, Rails will bypass the cached value, grab the new value, and load this into the cache - effectively refreshing the value without a user having to trigger the process.

Here’s the monkeypatch code - have a scan through it, and I’ll explain it below:

    require 'set'

    module ActionController #:nodoc:
      module Caching

	    module Actions
        extend ActiveSupport::Concern

          protected
          class ActionCacheFilter #:nodoc:
            def initialize(options, &block)
              @cache_path, @store_options, @cache_layout =
                options.values_at(:cache_path, :store_options, :layout)
            end

            def filter(controller)
              path_options = if @cache_path.respond_to?(:call)
                controller.instance_exec(controller, &@cache_path)
              else
                @cache_path
              end

              cache_path = ActionCachePath.new(controller, path_options || {})
              overwrite = if @overwrite = @store_options.fetch(:overwrite, nil)
                @overwrite.respond_to?(:call) ? controller.instance_exec(controller, &@overwrite) : @overwrite
              else
                false
              end

              body = overwrite ? nil : controller.read_fragment(cache_path.path, @store_options)

              unless body
                controller.action_has_layout = false unless @cache_layout
                yield
                controller.action_has_layout = true
                body = controller._save_fragment(cache_path.path, @store_options)
              end

              body = controller.render_to_string(:text => body, :layout => true) unless @cache_layout
              controller.response_body = body
              controller.content_type = Mime[cache_path.extension || :html]
            end
          end
        end
      end
    end

By dropping this code into config/initializers, this code gets patched into the ActionController::Caching::Actions::ActionCacheFilter class, and overrides the initalize and filter methods to let us a) pass in an override option, and b) choose to refresh the cache if the override option is set.

The filter method performs as normal until it has finished generating the cache path - at this point, it would normally return the cached response if it was there, and if it had not expired. Instead, my colleague James Moriaty replaced some code here:

First, it retrieves the value of the :overwrite option passed in to the caches_action method from the @store_options hash - if it’s a Proc, it executes it here to get the value, otherwise assumes it’s a boolean variable.
If the :overwrite option has not been passed in, it returns false - i.e. don’t overwrite the cache.
If the overwrite value is true, it sets the body to nil, so that it will be re-built. Otherwise, it does the usual and returns the cached response from Memcache.
From here, it more or less goes back to the default class, rebuilding the response from the database.

In our application’s case, we use this functionality by tweaking our cron jobs a little to pass in a particular parameter - we then added the :overwrite option to our caches_action methods, with a Proc that returns true if this parameter equals the correct value.

So far, this solution has worked fantastically - now, hardly any of our users hit the database - instead, they are heading to memcache to grab that response, while our background cron jobs rebuild the data that will get returned to them. Using a parameter for refreshing the cache also lets us easily refresh manually for testing or to check for a value.

This solution is clean, simple and easy to implement. I suggest that if you are facing similar problems, that you give it a go - it’s really adaptable, and requires few changes if you are already using action caching. Full credit to James for thinking up and implementing this solution - I’m just documenting it.

Howto: Database Backup and Restore

Fri, 23 Sep 2011 13:26:00 +1200

An inherent part of developing web applications is managing your datastores - typically, a relational database such as MySQL or PostgreSQL. Today, I’m going to quickly cover off how to backup and restore for both of these databases.

What you’ll need

Either PostgreSQL or MySQL
Access to a database (preferably one with data in it)

Why this is useful to know

Lots of interactions with databases have the potential to destroy or modify data (in a bad way). When using frameworks such as Ruby on Rails, it’s even easier to, say, accidently delete all of your Users (Horribly easy in DataMapper, unfortunately). It’s important that before you do anything with data that’s destructive, you have a backup of your database that you can restore from quickly and easily.

PostgreSQL

Postgres databases are backed up using the pg_dump command - a command-line utility that comes packaged with the database server. Here’s the command:

    pg_dump --no-owner -U [username] -W [database_name] > [file to dump to].sql.dump

Let me explain these options and why I use them:

--no-owner: By default, Postgres dumps the database with lots of SET OWNER TO statements. I like to take these out of the dump, as I’m not necessarily restoring the dump to exactly the same server with exactly the same users. Using –no-owner means that ownership statements will be excluded.
-U [username]: This lets you pass in the database user name your web application usually uses to connect - using this is just good practise, as it ensures that what you’re dumping is exactly what the web application has.
-W: This option, used in conjunction with the -U flag, prompts for the password when you run the command
[database_name]: This is the name of the database that you want to dump
[file to dump to]: This is the file that the SQL script that pg_dump produces will be piped into. I normally name this file with the .sql.dump extension, so that I can see it’s a SQL dump straight off.

Restoring a database dump

Restoring a Postgresql dump is really easy, and involves using the standard psql client to connect to the database and execute the SQL script in your dump file.

First of all, make sure that you have created the database you want to load the data into. In this example, let’s say I’ve dumped from the facebook_production database to the file facebook_production_23092011.sql.dump, and I want to restore into the facebook_development database so that I can test out some code against some production data. I want to connect to the database using psql as the web application user, and load the dump in:

    psql -U facebook -W facebook_development < facebook_production_23092011.sql.dump`

Note how I am using the opposite of the less-than symbol I used above - this basically denotes the direction of the data - it’s coming from the file, going to the database.

Upon running this commmand (with your own database, of course), you will first be prompted for your database user’s password, and will then see a bunch of SQL statements being executed. Once it’s completed, your database has been loaded successfully - you can jump in using psql if you’d like, and query around a bit.

MySQL

MySQL databases are backed up with the mysqldump program - one I’m not as familiar with as Postgres, but I know the basics, and largely that’s all you need with this type of thing. The main thing to keep in mind is that the process is the same as for PostgreSQL above - use the dump program to write the database out to a file (in the form of SQL statements), and then use the database client program mysql in this case, to execute the commands in the file against the database being restored to. Here’s the command to dump a MySQL file to disk:

    mysqldump -U [username] -P [database_name] > [file to dump to].sql.dump

The options are more or less as I’ve described above, except that -P is substituted for -W, and I’ve still stuck with the .sql.dump naming scheme.

Restoring a database dump

This process is almost identical to the PostgreSQL restore process. Let’s stick with the same example format we already have - dumping from a database called facebook_production to file facebook_production_23092011.sql.dump, restoring into a database called facebook_development - here’s the command we need for that:

    mysql -U facebook -P facebook_development < facebook_production_23092011.sql.dump

Once again, you’ll be prompted for your password, but for this one you won’t see the output of the SQL statements - it will take a couple of seconds, and then the program will exit. This is normal however - if you’d like to see the output of the batch load, you can add -v -v -v before the < symbol to turn verbosity to level three, otherwise you can go ahead and jump into your database and make sure everything is there.

Tips:

If you don’t have a database user account yet, on Unix and Linux you can use sudo su postgres to login as the ‘postgres’ user - a root-like database user that gets created for you when Postgres is installed. MySQL has a root account, but it’s not a system user - you are normally prompted for a root username and password when you install MySQL. If you are using the Postgres user, you don’t need to pass in a username or password in the above commands, but you need to remember that your restored databases will be owned by ‘postgres’, not you web application database user.
A handy place to put database dumps that you are planning to use right away is /tmp (Only on Unix and Linux). This is a directly that is writeable by everyone, that will get ‘garbage collected’ periodically. This is especially handy if you are logging in as the ‘postgres’ user, as typically this user won’t have write permissions on many other directories

Introducing Blog Broadcaster

Wed, 14 Sep 2011 19:04:00 +1200

I’ve just completed a Blog Broadcaster for this blog. It had a couple of interesting technical things, and I needed to test it properly, so here’s this post!

I recently migrated this blog from Tumblr, and while Tumblr was pretty awesome and easy to use, it didn’t have great support for blocks of code and preformatted comment, and, like Radiant CMS, it stored layouts, stylesheets, and all of that kind of thing in a database somewhere - it wasn’t in source control, and making changes to it was dangerous.

One thing that I immediately missed from Tumblr, however, was it’s ability to post to Facebook and Twitter automatically whenever I published a post. As I don’t post that often, it’s really important to me that I market my blog as much as I can - like my Github profile, the content on my blog is a reflection of my knowledge and skill as a developer, and so I want to get that in front of people as much as possible - broadcasting to social networks is a great way of achieving that.

With my blog on Github, I needed a way to broadcast new posts to these social networks. Github has a built-in post-receive hook to post to Twitter, but I really needed something more than that. Here’s my list of requirements:

It should automatically post without me needing to do any special task
It should be conditional - i.e. it shouldn’t post everytime I change something
It should support both Facebook and Twitter
It should be able to be triggered by a commit

What I ended up building was a Sinatra app whose sole purpose was to receive POST’ed commit information, parse it into a Facebook and Twitter post, and broadcast it. It’s hosted on Heroku, and gets triggered by a Github Post-Receive Hooks. I did run into a couple of problems along the way - the main one was sorting out being able to post Facebook updates without being logged in, or even needing to be involved at the process at all. This wasn’t too difficult, but I did need to get an access token that was long-lived, and have the ability to update this if necessary. I overcame this problem by adding some methods to my Sinatra app that will allow me to update the access token if it ever expires, or change the Facebook account used if necessary.

The second problem I ran into wasn’t really a problem, but it was a challenge to try and think of a nice way of doing it. Basically I needed to store the Facebook access token somewhere so that I could use it when I needed it - but I didn’t have a database, and I didn’t particularly want to add one just for storing a single string. Since this is running on Heroku (Bamboo stack, not Cedar), I was also on a read-only filesystem, so couldn’t store it in a simple text file either. In this end, I chose to store the value in Memcache using the Heroku add-on. This still isn’t necessarily a good solution, as this storage method isn’t guaranteed to be persistent, however it should suit my needs - it doesn’t particularly matter if I lose the access token, the application will gracefully degrade and just post to Twitter until I log in to Facebook via the app again.

So, I think I’ve come up with a good solution. It started off as a simple broadcaster for my blog, but I think it has a lot of potential for use with any open-source project that has social networking presence - I think it’s a much more elegant and flexible hook than that which Github provides, and I hope it get’s a bit of use.

Select Anything From Everything with Select

Fri, 09 Sep 2011 00:00:00 +1200

I was recently called upon to make a horrible select input for a Ruby on Rails project - essentially, there was this model, let’s call it a Snafu, and one Snafu could share an attachment to any number of models.

This select was difficult because I couldn’t just have a selection of record ID’s - I would have to use both the model type as well as the model ID in order to be able to track down these records when the form was submitted. Here’s how I did it:

**Step 1: Rendering the select ** First of all, we need to render a selection box for the ‘New Snafu’ form. This selection box should be populated by a number of model collections, keyed by an identifier that specifies both the model name and the model ID. Time for a model method, and a helper!

    def options_for_select_anything(selected = nil)
     #Make a hash of the things we want to select from
     options = {
      'Foo' => Foo.active.map(&item_for_select_anything),
      'Bar' => Bar.active.map(&item_for_select_anything)
     }
     #Use a Rails helper to actually get the tags
     grouped_options_for_select(options, selected)
    end

    def item_for_select_anything
     # Return an array of the record name and the identifier we want to use
     # The record name in this case is the display value, while the identifier
     # is the data value
     lambda { |record| [record.name, "#{record.id}-#{record.class.name}"] }
    end

Now that we have these helpers, we have a grouped collection of options, in which each option tag within the select will look something like ‘<option value="1-Foo">Foo #1</option>’ - let’s render our select.

	<% form_for @snafu do |form| %>
		<p>
			<%= form.label :item_identifier, 'Item' %><br />
			<%= form.select :item_identifier, options_for_select_anything %>
		</p>
		<p>
			<%= form.submit 'Create' %>
		</p>
	<% end %>

This is a basic form of course - your form will have all the other attributes you need, but the key thing to notice here is that we aren’t trying to load against our Polymorphic ‘item’ association within the form - instead, we’re just going to send through a ‘item_identifier’ parameter that we can use to find the item in our model. Let’s take a look at the sections we need.

First of all, we need to have the Polymorphic association in our model, if you haven’t already done this.

	# Snafu.rb

	class Snafu < ActiveRecord::Base
		# Snip
		belongs_to :item, :polymorphic => true
		# Snip
	end

	# Migration
	add_column :snafus, :item_id, :integer, :null => false
	add_column :snafus, :item_type, :string, :null => false

Next of all, we need to add an attr_writer to our model - this will hold the ‘item_identifier’ from our form submission until we are ready to use it.

	# Snafu.rb

	attr_writer :item_identifier

Finally, we need to add a before_validation filter to associate the record identified by our item_identifier with our model instance:

	# Snafu.rb
	class Snafu < ActiveRecord::Base
		# Snip
		belongs_to :item, :polymorphic => true
		before_validation :set_item

		attr_writer :item_identifier

		# Snip

		private

		def set_item
			if self.item_identifier
			id, model = self.item_identifier.split('-')
			self.item = Kernel.const_get(model).send(:find, id) rescue nil
		end
	 end

Let’s take a look at what we’ve done there - really, the meat of it is in our set_item method, that gets called before validations run (so that we will have a real item set before we may need to run validations on that item). First of all, set_item checks that we have set a item_identifier - if we haven’t we don’t want to run into Nil exceptions! Given an item_identifier is present, we want to split our identifier (Remember, this is in the format id-class name), into two parts. Finally, we do a little Ruby magic to get the class using Kernel.const_get, and then call find on it with the ID that we want. If anything goes wrong with this bit (The class not existing, for example), then we just set item to nil.

There we have it then - it’s a horrible situation, but I feel like it’s a pretty good approach. The logic is where it should be (models and helpers), the views and controllers feel clean, and it’s flexible to be reused pretty easily.

As a final tweak, there’s one more change you may want to make - that is setting the selected item when we return to our form. If you take a look at the helper method we defined above, you’ll notice that we already support a selected option - we just need to pass this in. To do this, we want to add a method to our ‘Snafu’ class that will return a string of the item id and the item name concatenated with a dash - i.e., the format that our select box in the form is expecting. Go ahead and add the method now:

	# Snafu.rb

	def item_identifier
		# Return the owner_identifier set using the attr_writer, if it exists
		return @owner_identifier if @owner_identifier

		# Otherwise, try and build it from the current item saved against the model
		[self.item.id, self.item.class.name].join('-') if self.item.present?
	end

With that method, can can just change our select input in the form to make use of this value:

	# new.html.erb
	# Snip
	<%= form.select :item_identifier, options_for_select_anything(form.object.item_identifier) %>

There we are! Now when we show the form, the selected item will appear, just as we wanted.

Capistrano rvm-shell: command not found error

Mon, 22 Aug 2011 00:00:00 +1200

I’ve just come across this problem, and I had to share it here - I can’t find anywhere else on the the Internet where the solution is specifically stated - it’s just alluded towards. If you are using RVM’s Capistrano integration, you may come across a CommandNotFoundError to do with rvm-shell not being under / usr/local/rvm/bin (Which is exactly where it should be). Upon searching the internet, you will find that you have to upgrade RVM (you don’t), and that when installed for a local user, RVM puts rvm-shell under ~/bin (But, you know, it’s a system-wide install). The solution is really simple - rvm-shell is under / usr/local/bin - use set :rvm_bin_path, "/usr/local/bin" in your deploy script, and you’re away. Clearly this is a bug with RVM putting things where it shouldn’t, but that’s the way of things. And, if I sound a little short, it’s because I had to all but reinstall RVM and break everything before realizing this.

Achievements on Coderwall

Sat, 23 Jul 2011 00:00:00 +1200

Well, it’s taken weeks for Coderwall to finally get it’s crawler to hit my Github Profile, but I’ve finally got more badges. I’ve been moving some old PHP stuff of mine onto Github for people to use, so that’s where I’ve gotten the PHP badge from. As part of my personal campaign to master at least one other language apart from Ruby, I’ve also been working on some Django tutorials (Django is a Python web framework), so have earned another badge there. A happy accident was the walrus badge - I didn’t realize I had projects up in so many different languages. Probably next on this list might be the ‘Forked 20’ achievement for my Spree extension, [spree-import-products] (https://github.com/joshmcarthur/spree-import-products) - I’m not so sure this is a good thing, though - the forks-to-pull-requests ratio is too low on this project - most likely indicating that I need to work more on making the extension easier to use for my general audience - forks without pull requests mean that people are taking their own copies to change them for their needs, rather than to work on bugfixes and general improvements. We’ll have to see. But… yay, more achievements! If you’ve got a Github account, I highly recommend you check out Coderwall - the achievements are largely irrelevant to any sort of reputation, but it’s a really nice way of finding interesting Github members with interesting projects.

Arbitrary Ordering in PostgreSQL when Rails + ENUM = No.

Thu, 14 Jul 2011 00:00:00 +1200

I’ve recently had to do a custom sort for a work project that has required a sort on something that is not naturally sortable correctly (For example, alphabetical or numerical sorting). While searching for a completely different solution, I came across this post that outlined a nice technique. Basically, when you have a field in your Rails model with a predefined set of possible values, you can use a CASE statement in PostgreSQL to perform the sort in whichever order these values should appear. Here’s a sample of how this could be achieved using Rails:

Result.order("CASE " + "WHEN medal='gold'
THEN 1 " + "WHEN medal='silver' THEN 2 " + "WHEN medal='bronze' THEN 3 " +
"ELSE 4 " + + "END,name")

It’s messy of course - how you want to format the SQL string is up to you, but it’s a great solution when you don’t have the normal sorting capabilities of a ENUM datatype available to you.

Quick: Clear Gemset

Sun, 10 Jul 2011 00:00:00 +1200

If you’re finding that you have to change something fairly significant in your bundler dependencies, it’s usually a good idea to get rid of what you’ve got loaded in an RVM gemset so that you don’t end up with different versions of gems fighting with each other. To do this, simply run rvm gemset empty - it’ll delete all the gems currently in your gemset, giving you a blank slate.

The Dictionary of New Zealand Sign Language

Mon, 27 Jun 2011 00:00:00 +1200

The Dictionary of New Zealand Sign Language went live on Friday - it’s a project that I have worked on with Daniel, Chris and James R, at 3Months. I attended a launch event at Victoria University on the Friday, and it really was quite a humbling experience - this dictionary has been the effort of a team of people for years and years - really, what 3Months has done has just been the tip of the iceberg. It really does seem like this project has been rewarding - we’ve taken a comprehensive database of sign information and images (Built with what seems to be a lot of time and effort by Dave Moskozitz), and given it a public presence - hopefully, something that everyone can use - whether it’s to learn New Zealand Sign Language, or just to learn how to finger spell names and words. I’m mostly posting this in the hope that everyone who reads this goes to check out the site, and to learn something of this language - give it a spin! http:// nzsl.vuw.ac.nz

'Password' or 'Passphrase'

Thu, 23 Jun 2011 00:00:00 +1200

So apparently pass phrases are the new ‘secure password’ - kinda the step you get to when you finally accept that your users are going to use something like ‘password’ for their account password. The natural step here is to reinforce a secure password strategy by requiring x numbers, x special characters and a certain length - but I find this really annoying when I just want to get signed up, and that means that other users do as well. Something I’ve just been thinking about is the naming semantics of password field - labeling it ‘password’ immediately prompts users to think of an actual word - if they are computer-savvy, then they might throw a symbol or number in, but most likely it will still be based on an actual word. I wonder what would happen if you labelled this field ‘Passphrase’ though? I think it is inevitable that many users will recognize the pattern of the form rather than the labelling of the fields and still enter their ‘password’, but just maybe there will be some users who get the semantics of the label, and enter a sentence, instead of a word. Even though there may not be special characters in that sentence, it’s still just as, if not more secure from dictionary attacks - guessing one word is pretty easy, but it’s much, much harder to guess a string of words, in the correct order - especially if one or two of those words are obfuscated with some special characters or numbers. Just a thought…. but interesting nonetheless.

Rails HABTM relationships on a non-standard connection

Tue, 21 Jun 2011 00:00:00 +1200

Recently, I’ve been implementing an admin interface for a system that I want to make more secure than the main application. The way I’ve chosen to do this is to run some models that relate solely to the admin application (Authentication and Authorization in particular), on a different database - let’s call it ‘login’. This seems to be a reasonably common thing to do for security purposes, and also for things like moving data from one database to another. Once again though, I’ve been tempted into the potential nest of bugs that is has_and_belongs_to_many - let me explain the schema first though: * Administrators table - holds email, encrypted passwords and other data about administrators * Roles table - holds the name of the role - used to authorizing an administrator when performing an action. Each of these tables uses the line establish_connection 'login' to connect to a different database than the other models - this is the secure database that I want to leave purely for the administrative application. So, given that I had an administrator that should be given multiple roles, and obviously each role could have many administrators, has_and_belongs_to_many seemed the obvious candidate - I didn’t really want a model just for the association, and I would just need to write a migration to create the join table. So, off, I went, and here’s what happened: ` ERROR: relation “administrators_roles” does not exist` i.e. - the Administrator table exists, and the Role table exists, but the join table just isn’t there. The first call for me was to take a look in the database, and make sure that the table was there - which it was - and that migrations had definitely run correctly and the schema was correct - which they were. After much frustration, I found this thread which described the problems I had been having - and I was vaguely satisfied to see that the problem wasn’t really my fault. It seems that in some versions of Rails, ActiveRecord’s has_and_belongs_to_many_association class doesn’t respect the database connections that the models are trying to use - instead, it uses the universal database connection to try and look up the join table - so, what was going wrong was that ActiveRecord was looking in the development environment’s database, when it should have been looking in the login database. Unfortunately, short of updating your version of ActiveRecord/Rails or patching this class, it seems that there isn’t really any way of avoiding this problem - you have to drop back to using has_many :through with a Model representing your join table. I can, though, at least vouch that once you have done this, it does work as expected, which, in the end, is what we want. It still feels a bit hacky though….

Quick: .rvmrc

Tue, 21 Jun 2011 00:00:00 +1200

This post is probably something more experienced RVM users will already know, but I wanted to post this as it’s definitely my discovery of the week. When throwing an .rvmrc file into a project, it’s a nice thing to do to write the script correctly so that it will just work for other developers (As well as telling you what gemset you’re using when you jump into the directory). In your .rvmrc file, put something like this: rvm use 1.9.2@gemset --create …this will attempt to use that gemset (Printing out a nice message telling you it’s using that one as it does so), and will create it if it doesn’t exist.

Ubuntu: Quick ImageMagick Install

Thu, 16 Jun 2011 00:00:00 +1200

This is cross-posted from a tweet I posted a while back - I think it’s a nice bit of advice, and I wanted to store it in a more persistent form

Installing ImageMagick is one of the things that Rails developers need to do reasonably often when provisioning new servers - basically, if you’re doing any sort of image processing in your application (including the popularPaperclip gem), ImageMagick is what you’ll be using.

The problem is that there is a bit of a magical formula I have needed to use in the past - if you just install ImageMagick, it will most likely not work, as it needs to have support for different image formats you want to use compiled it in right from the get go. Previously, I have just looked up the various libraries I have needed (For PNG, JPEG, etc.), and then either found the libraries in the Ubuntu package repositories or built them from source.Â A nice quick way of doing it though, is to use an ImageMagick meta-package in the Ubuntu repositories named libmagick-9-dev - it is just a collection of popular image format libraries, as well as a couple of additional utilities for ImageMagick. You can install it on any Ubuntu system by running this command: sudo apt-get install libmagick-9-dev ImageMagick itself still needs to be installed, of course. The best option here is just to build from source - packages in the repositories are horribly out of date, and I have found Paperclip, RMagick and Minimagick all require a fairly recent version of ImageMagick. Building from source sounds really intimidating, but it really isn’t - just follow these steps:

First of all, download a tarball of the ImageMagick source onto your computer: wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
Next, extract the tarball: tar -xvzf ImageMagick.tar.gz
Now configure the package - note especially the end of the output (There is a lot of output) - it tells you which Image libraries you have installed - any with ‘yes’ next to them it will happily format and convert - because you’ve installed the package above, all the common formats should have a ‘yes’ next to them, but it’s worth checking: cd ImageMagick-[VERSION] (VERSION will be a series of numbers like '6.7.0-8') && ./configure ‘ Now all the hard work (By you) is done - you just need to compile the packages:

make && sudo make install

All done! - ImageMagick should be all installed. To check, run the command which identify, and check it returns a path to the identify command. Note: If you have trouble compiling, make sure Ubunutu’s build tools are installed: sudo apt-get install build-essential

Integration tests with Devise and RSpec

Sat, 11 Jun 2011 00:00:00 +1200

RSpec 2 has supported integration tests for a while now, and I’ve chosen to use these for a project I’m working on at the moment instead of Cucumber (I don’t feel that I need the verbosity andÂ English-like structure Cucumber provides given that it a more complex process to write tests). A bit of a problem I’ve come across recently is how to get a Devise user signed in to your application in your tests, given that integration tests don’t really give you any access to either the session or the controller (This rules out manually setting ID’s in the session, or stubbing out current_user. As it turns out, the implementation of it is pretty simple, but I did have to do a bit of browsing and piece a few bits together to work out a nice way of doing it. Here is the code you can use (You would normally place this within a before(: each) filter in your routes spec (Which is what an integration test in RSpec is called):

# At top of spec file, after require 'spec_helper'
include Warden::Test::Helpers

# In a before(:each) block
@user = Factory.create(:user)
@user.confirm!
login_as @user, :scope => :user

Now, what is this doing? Well, first of all, you include some test helpers that Warden (Which devise back-ends onto), provides. I tried out using Devise’s Devise::TestHelpers here, but it looks like Devise haven’t really designed their helpers with integration tests in mind - they didn’t really work.Â Within our before(:each) block, it now gets pretty simple. We create a User, using Factory Girl (This part of the implementation doesn’t really matter, you can use fixtures if you prefer, or even a plain old User.create. Next, we confirm the user. This isn’t necessary if your users haven’t been marked as :confirmable in your User model, but obviously our new user needs to be confirmed and active in order to log in. Last of all, we use a helper method Warden’s test helpers has provided us to log in the user, which sets us up for any more requests we need to make. Have fun speccing nicely with Devise/Warden and RSpec!

Quick: Get random record efficiently in Rails

Tue, 07 Jun 2011 00:00:00 +1200

You could use SQL’s random function (RAND() or RANDOM() depending on database engine) - but this isn’t database agnostic, so isn’t really very quick. Instead you can use @nzkoz’s suggested method: Widget.first(:offset => Widget.count) …. the count() method is fast, and the first() method will limit it to the first result in the SQL as well.

ImageMagick: Cropping then Resizing a PNG

Fri, 06 May 2011 00:00:00 +1200

I’ve been working on an image processor class for work, and recently ran into this issue. I thought I would post it up here as normally I need to be quite desperate before I start trawling through email mirrors - hopefully somebody comes across this post first. If you use ImageMagick (in particular, in conjunction with MiniMagick), then you may come across this issue, and there is actually a quick fix to it. The issue itself is as follows: my image processor retrieves an image from an online source, but the image has a 1 or 2px border, and is thousands of pixels wide - too wide for web use. I therefore wrote this class to first crop the border off the image, and then resize it. If you do actually do this though, there is an important step that needs to go in-between the cropping and the resizing. If you won’t do this, you will basically get a PNG layer that is offset from the image itself - i.e. some or all of the Â actual image content is not visible. This happens because when the image is cropped, the origin of the image changes. It needs to be reset back to coordinates 0, 0 in order to not offset the layer itself when the image is resized. Here’s how to do it.Â For Ruby/Minimagick: image.set(“page”, “#{image[‘width’]}x#{image[‘height’]}+0+0”) For ImageMagick directly (i.e. in console): convert [image sequence]: -set page [width]x[height]+0+0 Simple! The image should save correctly.

Using setInterval to handle scroll() events

Tue, 12 Apr 2011 00:00:00 +1200

I’ve just added a nice unobtrusive scroll to top feature to my blog, and learnt an interesting tip in the process I thought I would share, originating from one of the many problems Twitter has had with it’s jQuery fanciness. The scroll to top stuff isn’t overly complicated - just detect when the user has scrolled x pixels down the screen, and then show a link to go back to the top (Which can be animated if you want) - for a nice tutorial on this, check out_this_tutorial. The thing is that this technique uses the scroll event (bound to window, but can be bound to basically anything with a scrollbar). The scroll event gets fired any time the window is scrolled - but any time the scroll amount changes at all. What this means is that the scroll event is fired any time scroll position changes - so if you grab the scrollbar, and pull, the scroll event is fired every time the bar moves, not when you release the mouse. John_Resig_reports_that_Twitter_ran_into_some_problems_with_thisÂ - they had a function bound directly to the scroll event being fired every time anyone scrolled (at all!). Resig has also reported a nice solution though, which I’ll pass on here. It’s actually quite simple. Instead of binding a complicated function to the scroll event, you simply set a flag variable to let something else know that the window has been scrolled. The second piece of the solution is a function running via setInterval - that is, a function being called on a scheduled basis. The first task of this function is to check this flag - if it is set, it can perform any task (Such as showing a ‘Back to Top’ link, and set the flag variable back to false.Â The end result of this solution is that you basically have a polling function, rather than an event-driven one. This is actually a good thing though, when it comes to this type of event. Instead of having a complex function called every time the window is scrolled, a function is called every quarter second or so, and only executes if the window has been scrolledÂ - much better! I personally used an object variable, rather than just a flag - this object was populated with the object that was scrolled on, and I then checked if this object was populated in my polling function (rather than just is true), and could then use the context it provided in this function. Since then, however, I have realized that in my case, this object will alwaysbe the window object - so I may as well have a cheaper variable assignment and just directly refer to the object in my polling function.

Accessing controller instance variables in model (Urgh?)

Tue, 12 Apr 2011 00:00:00 +1200

I can tell that this title alone will irritate a lot of developers out there. It irritated me as well, until I figured out that sometimes, doing things the ‘wrong’ way is the best/only way. But let me get on with things. Every now and then, you will come across a scenario where you need to access something in the controller, from the model. This is so obviously non-MVC that many will (should) immediately shy away from it, but let me suggest a scenario. Let’s say you have users, who should be scoped to a particular subdomain. You would place a default_scope on the User model (with a Proc, once an issue with default_scope is resolved), that would automatically add conditions to any database lookup calls to restrict the found users to those belonging to the current subdomain. The tricky bit is, of course to get this subdomain. Because it is a default scope, you don’t actually ‘call’ it - so that rules out passing parameters. I’ve come across other suggestions as well, such as using cattr_accessor (NO! It’s shared across requests), and storing a value in Thread.current (Relying on the application using threads in the way you expect them to = not safe). So far though, the best hint I have seen here, although extremely anti-doing what I am suggesting, would seem to work.Â Basically, the jist of this post was that you could dynamically add methods to ActiveRecord::Base using an around_filter to access sessions, cookies, params and the request. Now, I agree that this might be a bit too extreme - if you really need to access the entire session or cookie hash, for example, you probably aredoing something wrong. Something that I think isÂ suitable though, given an appropriate situation, is to use this technique to add an ‘accessor’ to ActiveRecord::Base that returns the value you need in your model - the current subdomain for instance. This is still non-MVC, but is, I think, a far more elegant solution that anything else I’ve seen suggested.

I18n ALWAYS escapes dots in chained backends :-(

Fri, 08 Apr 2011 00:00:00 +1200

Just a quick tip I’ve come across while browsing through the comments on a Railscast_I’ve_been_watching. It looks like the I18n gems that get automatically installed with Rails 3 have a teensy bug. When storing a translation to the backend (be it YAML, Redis, whatever), there is an :escape option that can be passed to enable (or prevent) the key from containing the dot character (.) (Among others, I’m sure). This seems to work fine when dealing with a single backend, but unfortunately when you are using chained backends (For example I am using Redis preferentially, but falling back on YAML when necessary), the options hash that you would pass the :escape option in is mistakenly initialized to an empty hash. It’s a simple fix - there is already a closed pull request on the issue here. It looks like the latest stable version is only 0.50 though, so the fix isn’t going to filter through to the actual gem just yet. For now, the best option is probably to class_eval into that particular section, or just change the file yourself.

Spree Hosted Gateway

Thu, 10 Mar 2011 00:00:00 +1300

Spree Hosted Gateway is my second ‘big’ extension - one of the ones that isn’t just adding one or two bits of nice functionality, but actually and end-to-end solution to add something that I think would be useful to a broad range of Spree developers (My first was spree-import-products, see my post on it here). Spree (for those of you unfamiliar with it), is an open-source eCommerce framework written in Ruby on Rails, and has recently been upgraded to Rails 3, which has enabled extensions to now be packaged as Rails engines, making them far easier to create, implement and share. Out of the box, Spree is able to provide much of what a basic online store requires, but something I have felt is missing is support for alternative payment methods that are much more rudimentary than those catered for by Railsdog’s_fork_of_ActiveShipping.Â Inspired by a project I was working on at 3Months, I decided to develop an extension that provided a new type of Payment Method (Alongside Cheque and Creditcard) - External (Or Hosted) Gateway - this type of payment method was characterized in the following ways: * Redirection - the customer was required by the gateway to be redirected to an off-site form to make payment. * Tracing the payment and order: attributes required for payment (Amount, etc.) were gathered by the payment gateway from POSTed form data. * Communications: there was no type of complex back-and-forth between Spree and the gateway provider behind the scenes - information was transmitted to the gateway, and all Spree is able to do from that point is listen for a postback and collect any information from that. These type of gateways are (unfortunately) quite prevalent, especially around small-to-medium independent companies - they tend to be cheaper than more widely-known solutions, at the expense of a more reliable and secure transaction for the customer. Spree Hosted Gateway, as I have mentioned, hooks into Spree in the form of registering itself as a Payment Method that can be configured from the admin interface. After seeing all theÂ arbitraryÂ information that I was required to POST to the payment gateway I was testing against, I also made it easily extendable, and used Spree’s preferences system to allow any number of key- value pairs to be created (Which could be modified in the Spree Admin UI), which are automatically posted to the payment gateway when I customer is redirected (Unless the preference key is added to the exclusion list, of course!). I conjunction to the ‘transmit’ function of the extension, I have also implemented a kind of ‘landing pad’ for postbacks from the Payment Gateway to hit. When this happens, the extension is able to detect whether the transaction was successful or not, and continue processing the order. If the customer has made an payment that is not sufficient to cover the cost of the order they are redirected back to the payment page, and if the transaction was not successful, an error message is presented to them. All in all, I’m happy with this extension - it was a great way to learn about how Spree itself handles payment methods, different types of gateway and payment validation and handling, and I’m hoping this extension will help out some of the Spree developers out there working on smaller stores that don’t need such a large (expensive) gateway.

Running Rake tasks with Cron (RVM)

Wed, 09 Mar 2011 00:00:00 +1300

Recently I’ve had to deal with a strange problem with rake tasks being run using Cron, a UNIX tool for running commands on a scheduled basis. The problem was basically the server being slowly strangled of resources across a forty to fifty minute time period, as each time the rake task was run it would leave an entire bash process sitting there consuming resources. I had assumed that this problem was relating to the rake task exiting in a non- normal fashion - an exception maybe, or just not returning something that bash was requiring to say ‘OK, I can close now’. As it turned out, the solution was even more basic than that. The server is question is a bit of a prototype, as it’s using a system-wide installation of RVM to mange two very different sets of gems. To help integration with Passenger, each project directory has an .rvmrc file that automatically sets the correct Ruby version and Gemset when that directory is entered. RVM has a neat security feature however, that prompts you to view and approve the file before it is executed for the first time - and, you guessed it, this is what was tripping up Cron. What was happening, is that Cron was changing directory to the release path in order to execute the rake task - but not getting as far as actually executing the Rake task. Instead, Cron was being presented with a security warning from RVM, which was very cleverly sitting there waiting for input to approve the file - causing Bash to sit there… for ever. Fortunately, I did find a solution - it is possible to turn this security mechanism file off, to stop it from prompting for approval of the .rvmrc file, by adding a configuration flag to the default rvmrc file in /etc/rvmrc - here’s the configuration flag: export rvm_trust_rvmrcs_flag=1 (It goes inside what should be an if block). From now on, when Cron runs the rake tasks, it won’t be presented with this prompt - approval will be assumed. No more server resources limits exceeded alerts!

OK, So I Was Wrong

Mon, 07 Mar 2011 00:00:00 +1300

OK,_So_I_Was_Wrong So I was just flipping through my archives of blog posts, when I found this_one

my first ever post to Tumblr, where I vowed to stop posting enormously long essays on silly ideas I have to change the world. Oops. Sorry everyone!

Dynamically Serving Inline Images with Rack

Fri, 04 Mar 2011 00:00:00 +1300

Following an insanely interesting presentation by Steve Souders at Webstock, I began thinking about ways to simplify (i.e. automate) certain processes to optimize performance of sites. Something that immediately jumped out at me from early on in Steve’s workshop (So early, I was able to start coding in the halfway break), was that inline images were an ideal candidate for something that is relatively easy to do, but could have a significant effect on overall page load speed. The concept of inline images has been around for a while, and basically involves embedding Base64-encoded image data within the image tag like so:

So, what impact does this have on performance vs. the normal technique? Well, the normal technique involves several more requests - one per image, in fact. As Steve pointed out in his workshop, server response time is only a small part of the problem - the reality is that it doesn’t make a huge difference how fast a piece of content is processed on the server - unless you are doing something really wrong, of course. Where you can really hit performance bottlenecks is actually in the request-response transaction - waiting for data to be transmitted, received, processed, assembled, and rendered on screen. Using inline images can improve web performance when used appropriately, but as with any other performance technique, it has it’s drawbacks. First of all, because the images are actually embedded into the page, caching is affected - browsers may cache images longer than static pages, and if your web server is not set up correctly, dynamic pages may not be cached at all. Next, the relative file size of the page is not equal to the page size without images plus the combined images file size. The encoding process tends to result in about a 30% increase in size. Notice above I have said when used appropriatelyÂ - this means that, despite the drawbacks above, it really can help web performance. The key to not overusing this technique is to carefully consider the boundaries of what images should be embedded - the best candidates are small to medium sized images that do not generally change much - especially if there are quite a few of them on the page and the server is slow - this means that your page load speed might be slower, but there will also be no need to make several requests once the page hasÂ loaded to draw in all of these assets. Large images, on the other hand, should not generally be encoded. There is a fine balance between embedding images into a page to save on requests, and overloading the page with embedded images and ending up with a 2 or 3 megabyte page for the client to load. Large images will shove up this page size, and in addition to this, they are usually particular files that you want to apply a different caching policy to anyway. In order to help out and generally feel good about my contributions to web performance and open source, I have actually translated what I’ve written down here into a piece of Rack middleware that creates inline images (Since I’ve come across rack-pagespeedÂ since then, this middleware is probably superseded), and embeds them into a page. This technique was nice because a) It gave me a chance to learn how Rack middleware works with a practical project, and b) because when myself or others wish to use it, all that needs to be done is to drop the Gem into my Gemfile, run bundle install, and register the middleware in my Application’s configuration - from there on, the middleware will automatically parse responses from the Rails (Or any Rack-compatible application, actually), and replace the with encoded data. Overall, this technique is not perfect - it’s definitely not something to be overused, but with careful consideration and appropriate selection of images it’s a good stepping stone to improved web performance.

Creating a Scalable Academic Signup System

Fri, 04 Mar 2011 00:00:00 +1300

While working at 3Months as a developer, I am also completing a Bachelor of Business Information Systems at Victoria University. One of the things that rolls around twice a year and always causes students headaches is the signups systemÂ - it’s one of the bragging points of the School of Information Management at Victoria, as it was build as a project by a group of students there (Hence the name ‘S-Cubed = SSS = SIM Signups System). In itself, it’s a good application - it fulfills it’s brief of managing signups for tutorials and workshops, and lecturers and students alike seem to have no huge issues using is. The problem is though, is that its performance under load is crap - it simply was not build for the load that it receives when tutorial signups for 2 courses of 700 people each opens at 9am, and each of those 1400 people trying to get the exact tutorial they want are continually refereshing the page and so exacerbating the server load even more.

This server load doesn’t just extend to slow page load speed - since the site has not been build with scalability in mind, there have been ongoing issues with sessions expiring as people try and signup for a tutorial or workshop, people stuck halfway through a transaction, and the server even goes down every now and then. What I am proposing is an open framework that takes the best aspects of this type of system, and builds upon it techniques to optimize page load speed, and increase scalability. It seems to me that Universities in general are lacking a system that allows them to smoothly run signups (usually they are dependent on modified forum software, paper signups, emails or proprietary in-house systems), and so this isÂ definitelyÂ something that is marketable around the world. As I see it, here are the base requirements for such a system: * LDAP and OAuth Authentication so that students do not need to ‘register’ to sign up for a tutorial * Accurate time-based releases of tutorials and workshops that do not require human intervention * Automatic server load monitoring with the ability to increase allocated resources if necessary (i.e. this application is an ideal candidate for Rackspace Cloud or Heroku) * Per-course and per-University analytics for better scheduling of tutorial and workshop release times and how students are accessing the system. * The ability for lecturers and course administrators to export and print class lists, tutorial lists, and workshops list with all student data or just a subset. This project isÂ definitelyÂ going on theÂ work pileÂ for my personal projects

it’s got a couple of interesting aspects to it which I haven’t approached before (Such as LDAP from Rails), and generally seems to follow along with my ethos of community-oriented ‘responsible web applications’.

Radiant as a Service-delivery Platform

Thu, 23 Dec 2010 00:00:00 +1300

Tonight I’ve been playing round with Radiant - it looks really impressive as a content-management system, and I’ve been thinking about ways I can change it round to support vendors - that is, one person or organization offering Radiant as a service to multiple clients - basically, it would be one big Radiant application (to simplify administration and server maintenance), but would be scoped (for administrative users) just to that organization - nothing would be different from them, aside from the fact that the vendor can use one instance of the application to create, modify and maintain many organizations’ Radiant content management systems.Â My motivation for doing this is to try and create something I thought of a bit earlier today - a service to help low-level, low-budget schools (Primary to Intermediate) to move services and content online in a way that they can manage themselves. Radiant seems to be an excellent option in this regard - supporting diverse and flexible themes, and with plenty of extensions to offer additional functionality for little work. What I want, though, is a sort of dashboard for myself - I want to have a nice interface to have overall control over each installation, and for me to administer the site myself if that is a client’s preference. In addition, by using the method of scoping a ‘site’ to an organization, some cool things are possible, like an online signup wizard that would allow a client to set up a site without any initial involvement from me - because it is just records being inserted into a database, and not an entire application being created, this type of process becomes very easy. It’s early days yet, of course, but something in the educational area is what I’ve been wanting to tackle for a while now - let’s see how it goes. For more information about Radiant CMS, seeÂ http://radiantcms.org/ For more information about what I am doing, follow my github repository: Â https://github.com/joshmcarthur/radiant-for-vendor

So I decided this evening to devote a couple of hours to giv...

Mon, 13 Dec 2010 00:00:00 +1300

So I decided this evening to devote a couple of hours to giving back to Spree, after using the framework for a couple of work projects, and appreciating it for it’s flexibility and extensibility.Â I’ve developed an extension for Spree called Import Products. The source is based on a few bits of code that I have used for 2 projects now to import products into a Spree store from a comma-seperated values file (.CSV). It’s not perfect, but I’m happy with it as my first ever full contribution back to the Ruby on Rails community - I hope I have done a good enough job that it is useful to a couple of people. If you want to check out the source code, it’s on github. If you want to check out a description of the extension, check out the Spree extension_page.

RVM, Ruby-Debug, and hours of frustration

Tue, 14 Sep 2010 00:00:00 +1200

Every now and then, you run into one of THOSE problems - inexplicable, and something it seems no one else has solved before you. This afternoon, I hit one of these. Basically, I had just installed RVM, Ruby_1.8.7 and Rails_2.3.8 - everything was working great - I could run a server just fine, and use the running site. Eventually though, I found something broken. Easy enough to diagnose, I thought

I’ll just start up a debugging server, and work out whats going on. Wrong. Wrong, wrong, wrong. What followed was a continuation of this error: You need to install ruby-debug to run the server in debugging mode. So I had a Google - lots of people have run into this problem before (here and here specifically) - the problem is, most of these problems came down to a missing library called libreadline - for them, installing the library corrected all of their problems. In the end, I stumbled across this_post, from a guy having issues with Ruby after upgrading his Mac OS X - finally - he suggested something I hadn’t already tried. And, as it turns out - it worked! I have no idea why - but there we go. If you are having a problem getting Ruby 1.8.7 to acknowledge ruby-debug is there when running a debugging server, try this: gem uninstall ruby-debug gem install ruby-debug linecache –no-ri –no-rdoc script/server –debugger … It might just work for you!

Rubbish Domain Names

Fri, 20 Aug 2010 00:00:00 +1200

Finding appropriate domain names is a chore. There are so many rubbish parking sites out there that just … shouldn’t be. I dare sayjoshmcarthur.me could be considered one of them, since I could easily offload this onto Tumblr, but I enjoy having the unlimited capabilities a hosted site allows. I think that domains should be treated in a similar way to Patents when they are registered. Patents require the applicant to justify to an overall authority (the Intellectual_Property_Office in NZ), why they should be allowed to own a particular patent. If there was a justification scheme in place, it would not restrict the people who need domains - they would simply have to specify why they need the domain. What it would do is provide a decision framework on which to base decisions on registering domains to domain agents. Why tie up a perfectly good address? Leave them open, and truly make it first- in, first-served, instead of the ‘richest wins’ model that exists at the moment.

Time Problems

Wed, 18 Aug 2010 00:00:00 +1200

Fixing time problems is probably one of the most satisfying things to fix programmatically for me. It is one of those things where you may spend ages stabbing in the dark with different time zones, formats, etc etc. - but there it really is satisfying when you do get it right and what you see in your code is what you see on your clock. Weird, but true.

Interesting Ideas

Wed, 11 Aug 2010 00:00:00 +1200

As part of my research for my last-minute essay rewrite, I happened to pick up an article covering an interview with David_Heinemeier_Hansson, partner at 37signals.com and inventor of Ruby_on_Rails. Hansson had some very interesting ideas, including the concept of reducing working hours in favour of creative quality, and the value of constraints over capital: “Excellent decisions come out of constraints…You must make compromises and decisions to cut down and release something that’s simple and easy to use”. Hansson also had some interesting thoughts regarding the virtual workplace, suggesting that employees are happier in an environment of their choice, and a relaxed managerial style that reduces overhead and emphasises quality over quantity. Definitely somebody to keep an eye on!

A handy tip

Tue, 10 Aug 2010 00:00:00 +1200

Just FYI… See 2 posts ago - I am modifying this theme to make it work a bit nicer. Part of this included wanting to change some images. I was trying to just link to the images in CSS, but I found a much nicer way of doing it. If you want to change your theme currently, you can often change a range of colors. There is a technique that allows you to prompt the user to upload an image file, rather than choosing a file: First, set up the parameter that the settings drop down will look for by making a meta tag in the document head that has it’s name set as “image: YourTemplateImage”, and the content set as the link to the image that should be used as default. You can then use this image by referring to it in your CSS or HTML - for example: #content { background: #FFF url({image:YourTemplateImage}) no-repeat top left; } Very nice! Cheers to this_blog_post for the link.

My new look...

Mon, 09 Aug 2010 00:00:00 +1200

Do you like my revamped theme? I think it’s totally cool to have this type of timeline in a microblog like Tumblr - it makes things really easy to flip through… don’t you want to keep pressing the arrows? It’s this_theme, but I tweaked the CSS a bit and added some jQuery to bind left/right scrolling to the arrow keys as well as the buttons.

Here's how this works...

Sun, 08 Aug 2010 00:00:00 +1200

Right team, listen up. See, I’m quite a lazy blogger - it’s just too much effort, and it never seems to be shiny enough (Just check out my old_blog out for proof. I’m going to make this easy for all of us - I’ll find cool stuff on the Internet, or even in the Real_World - and I’ll share it here. It might be a sentence, it might be a couple of paragraphs. No more pages-long blog posts for me!

Notes: Josh McArthur

How to handle Cloudflare Challenge pages when making Turbo requests

How to test new Github Actions workflows

Talk on Plain Language Law at the AI Engineers meetup in Wellington, NZ

Handy access to OS environment variables on Debian

How to set up Rails directories for MCP servers

Bikeshed #470: All about Queues

Connecting to a locally Running app with hotwire native

A simple redirect to next pattern for Rails

How to run a command via AWS SSM with live output

A Stimulus controller to auto-submit authentication codes

Open-sourcing a reference implementation of native iOS Google sign-in flow with Hotwire

About Trove Project

OpenNext on Cloudflare

Hutt Forks/Eastern Hutt Hut

Restart rails quickly with bin/rails restart

Check that an enumerable satisfies matchers with args in RSpec

Detecting the origin of an ActiveRecord record's destruction

How to use assigns() in a Rails request spec without a dependency

Generating ActionMailer Previews inside a transaction

Asserting method calls with Minitest

Converting geofiles to WKT using ogr2ogr

Tapping targets with Stimulus

iOS Action Sheet with Strada

Implementing an around callback with Minitest using Ruby

Auto-resizing images for .ico files using ImageMagick

Generating responsive utility classes with Bootstrap

How to rearrange menu bar icons in Mac OS

Logging the DOM state of an element with testing-library

Stretching interactive elements

Passion projects

Procs can be passed to Rails' collection form helpers

Avoiding script injection when rendering ERB

Streamlining daily blog posts

Adding elevation to DOC track dataset

Rails supports transforming aria hashes into dasherized attributes

Create a trix editor with a hidden toolbar

Asserting a partial was rendered in an RSpec view spec

CLI tool entr for running a command when a file changes

Assert window.location properties with Jest and jsdom

References

Mid King Biv Trip Report

Day 1: Holdsworth carpark to Mid-King Biv via Baldy

Day 2: Mid King Biv to Holdsworth Carpark

Retrospective

Quickly access Ruby Mail object parts

Access ActionView context in an RSpec test

Access parent node with Capybara

ActiveRecord::Base.update actually calls model setter methods

Handy Ruby shorthand for testing class ancestors

Go offline with Capybara, Selenium and Chromedriver

Access browser console messages from a Capybara test

Use Foundation data attributes in a React functional component

What happens when I run 'rake db:migrate' in Ruby on Rails?

How is the ‘db:migrate’ rake task defined?

The Rails gem includes ‘activerecord’ as a gem dependency:

The activerecord gem adds it’s own ‘lib/` directory to Ruby’s load path:

The ‘activerecord’ gem ‘railtie’ file is require-d by Rails

ActiveRecord Railtie is evaluated

ActiveRecord Database Rake tasks are defined

How is my migration run when I run bundle exec rake db:migrate?

ActiveRecord::Tasks::DatabaseTasks.migrate

ActiveRecordBase.connection.migration_context

ActiveRecord::MigrationContext

ActiveRecord::Migrator

Your migration! (ActiveRecord::Migration)

Updating db/schema.rb

Preview Devise mailers

Docker: You must use Bundler 2 or greater with this lockfile

Strip newlines from terminal output

Inspect zip file contents

See recipients of a GPG encrypted message

Cool utilities I've discovered this weekend

bat

jira

entr

TLDR pages

UNIX find case insensitive

My favourite HTTP client

Faraday

The ‘activerecord’ gem ‘railtie’ file is `require`-d by Rails

How is my migration run when I run `bundle exec rake db:migrate`?

`ActiveRecordBase.connection.migration_context`

`bat`

`jira`

`entr`

First implementation: Use other Enum methods, just not `filter`