Console productivity hack: Discover the frequent; then make it the easy

[article index] [] [@mattmight] [+mattmight] [rss]

For the expert user, nothing matches the efficacy of the command line.

It's embarrassing that this is still true.

Aside from multiple consoles and tab completion, the console interface hasn't evolved much in 30 years.

The continued dominance of the command line among experts is a testament to the power of linguistic abstraction: when it comes to computing, a word is worth a thousand pictures.

Yet, research in human-computer interaction barely acknowledges the command line's existence. It's a strange omission, since the core principles of human factors engineering still apply to the console.

For the moment, let's consider the principle of frequency:

The ease of performing a task
should be proportional
to its frequency.

If there's something you have to do a lot, you should make it easy to do.

With a cockpit, this means that the design of a control panel should put commonly accessed controls nearest the pilot.

In the kitchen, it means that the most frequently used cooking implements and ingredients should be within arms reach of the chef (and especially not hidden in drawers).

In this article, I'll give a few examples of how I apply the principle of frequency to my console computing, including:

  • silently logging console activity to MySQL;
  • damping frequencies to eliminate defunct activities;
  • mining these logs to find frequent tasks; and
  • employing frequency-adaptive commands.

Update: Readers have pointed out easier ways to implement the technical bits with shell scripts. I've included these at the bottom for comparison.

More resources

  • The best book I know of for console hacks is Linux Server Hacks: 100 Industrial-Strength Tips and Tools. It's loaded with fiendishly creative scripts and shortcuts. (Almost all of the hacks apply to any Unix-derived OS, including Mac OS X.)
  • It turns out that my colleague Eric Eide did a masters thesis on an adaptive interface to a Unix shell, Valet. Valet uses a mixture of system knowledge and heuristics to "detect and correct the kinds of mistakes that experienced users make most frequently: typographical errors, file location errors, and minor syntactic errors." Neat!

Logging and mining console activity

Before you can exploit the principle of frequency, you need an unbiased record of what it is that you do most frequently. You might even be surprised by what your most frequent tasks are.

Fortunately, shells like bash already have some of that data in the form of the history command. The history command prints a list of recent commands. bash stores this history (by default) in ~/.bash_history.

You should periodically examine your frequently used commands, and find ways to execute them quickly. In many cases, a one-letter alias (e.g. e for emacs) is surprisingly effective.

To find frequent commands, you'll need a script to compute counts for the contents of the history file. For this, I created a perl script called frequency:

my %counts = () ;

while (my $cmd = <STDIN>) {
    chomp $cmd ;
    if (!$counts{$cmd}) {
        $counts{$cmd} = 1 ;
    } else {
        $counts{$cmd}++ ;
    }
}

foreach $k (keys %counts) {
    my $count = $counts{$k} ;
    print "$count $k\n" ;
}

It counts the number of times that each line occurs on stdin, and then dumps out a report.

For example:

$ cat ~/.bash_history | frequency | sort -rn | head
78 ls
25 recent
17 ls -l
16 mutt
12 cd matt.might.net
12 cd
10 echo hi
9 emacs .bash_profile
7 cd articles/
5 sudo perl -MCPAN -e 'install DBD::mysql' ;

This example is clearly representative of about a day's worth of my recent activity, since my history only goes back for 500 lines. To get a truly representative sample, you'll probably want to increase your history size. Inside your .bash_profile, you should add:

export HISTFILESIZE=10000 # Record last 10,000 commands
export HISTSIZE=10000 # Record last 10,000 commands per session

At the same time, while it's useful to look at the full commands, it's more instructive to look at just the commands themselves. For this, you can use cut:

$ cat ~/.bash_history | cut -d" " -f1 | frequency | sort -rn | head
84 ls
67 cd
48 recent
47 history
37 sudo
26 emacs
22 echo
15 mutt
15 ecmd
13 r

It shouldn't be any surprise to an experienced console user that nearly a third of all my commands are to list files and change directories. Since these commands are so common, I'll focus on optimizing those two. That's where my biggest "bang for buck" comes from.

Logging directories to MySQL

Switching directories is a common console task. In any given week, however, perhaps 80-90% of a your time will be spent in a small working set directories. This frequency provides a chance to optimize.

What you should do is log the directories visited, and then create a script for quickly switching to one of the most visited directories.

To log directories, you can use (1) a MySQL database to store the data, and (2) the PROMPT_COMMAND environment variable to insert a script into the console that runs every time a console prompt displays.

(Yes, this is an abuse of the PROMPT_COMMAND variable, but that's OK.)

Setting up the table is straightforward:

 CREATE TABLE dircounts
        (path VARCHAR(255) PRIMARY KEY NOT NULL,
         count INT NOT NULL DEFAULT 0,
         time TIMESTAMP NOT NULL)

Then, you need a script, markdir.pl, to mark the current directory:

use strict;
use warnings;

use DBI;

# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;

my $dbh = DBI->connect("DBI:mysql:$database",
                       "$user",
                       "$password")
    || die "Connection failed: $DBI::errstr";

my $qpwd = $dbh->quote($ENV{'PWD'}) ;

$dbh->do("INSERT INTO dircounts(path,count) ".
              "VALUES ($qpwd,1) ".
           "ON DUPLICATE KEY UPDATE count=count+1") ;

$dbh->disconnect();

Before each time the prompt is displayed, bash evaluates the contents of the environment variable PROMPT_COMMAND. This variable is just the right hook to monitor the current directory; for instance, in ~/.bash_profile, we can add:

export LASTDIR="/"

function prompt_command {

  # Record new directory on change.
  newdir=`pwd`
  if [ ! "$LASTDIR" = "$newdir" ]; then 
    /path/to/markdir.pl
  fi

  export LASTDIR=$newdir
}

export PROMPT_COMMAND="prompt_command"

Damping the frequency

You should damp the counts over time, so that if you spend one week accessing a directory a lot, but never again thereafter, it disappears from the frequent directories set quickly. Otherwise, you could have a defunct directory that sticks around for a long time after you've finished a major project.

To do this, I recommend a cronjob that runs a damping script once a week:

use strict;
use warnings;

use DBI;

# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;

my $dbh = DBI->connect("DBI:mysql:$database",
                       "$user",
                       "$password")
    || die "Could not connect to database: $DBI::errstr";

my $qpwd = $dbh->quote($ENV{'PWD'}) ;

# Cut all counts in half:
$dbh->do("UPDATE dircounts SET count = count/2");

$dbh->disconnect();

The script exponentially decays counts, so that as soon as you stop using a directory regularly, its relative importance fades quickly.

Run crontab -e and then add:

0 0 * * 0 /path/to/damping/script

A frequency-adaptive change directory

With a database of all directory visits, it's easy to write a perl script, f, that lists these directories and permits changing to them with just a couple keystrokes:

#!/usr/bin/env perl
use strict;
use warnings;

use DBI;

my $target = shift ;

# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;

my $dbh = DBI->connect("DBI:mysql:$database",
                       "$user", 
                       "$password")
    || die "Could not connect to database: $DBI::errstr";

my $sth = $dbh->prepare("SELECT path ".
                        "FROM dircounts ".
                        "ORDER BY count DESC LIMIT 7") ;
$sth->execute() ;

my $count = 1 ;

if (!$target) {
    print "+ change to directory with: f <number>\n" ;
}

while (my @row = $sth->fetchrow_array) {
  if ($target) {
    if ($count == $target) {
      $sth->finish() ;
      $dbh->disconnect();
      system(". cdto '@row'") ;
      exit ;
    }
  } else {
    my $home = $ENV{'HOME'} ;
    my $path = "@row" ;
    $path =~ s/$home/~/ ;
    print "${count}: $path\n" ;
  }
  $count++ ;
} 

$dbh->disconnect();

For example:

$ f
+ change to recent directory with: f <number>
1: ~
2: ~/family/bertrand
3: ~/ucombinator/private/grants
4: ~/ucombinator/private/papers
5: ~/ucombinator/private/projects
6: ~/matt.might.net
7: ~/talks

A recency-adaptive change directory

Another useful perl script, r, lists the most recently accessed directories.

#!/usr/bin/env perl
use strict;
use warnings;

use DBI;

my $target = shift ;

# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;

my $dbh = DBI->connect("DBI:mysql:$database",
                       "$user", 
                       "$password")
    || die "Could not connect to database: $DBI::errstr";

my $span = 1000 * 60 * 60 * 7 ;

my $q = "SELECT path ".
          "FROM dircounts ".
         "WHERE UNIX_TIMESTAMP(time) > UNIX_TIMESTAMP(now()) - $span ".
      "ORDER BY time DESC LIMIT 7" ;

my $sth = $dbh->prepare($q) ;

$sth->execute() ;

my $count = 1 ;

if (!$target) {
  print "+ change to directory with: r <number>\n" ;
}

while (my @row = $sth->fetchrow_array) {
  if ($target) {
    if ($count == $target) {
      $sth->finish() ;
      $dbh->disconnect();
      system(". cdto '@row'") ;
      exit ;
    }
  } else {
    my $home = $ENV{'HOME'} ;
    my $path = "@row" ;
    $path =~ s/$home/~/ ;
    print "${count}: $path\n" ;
  }
  $count++ ;
} 

$dbh->disconnect();

List recent files on directory change

In addition to changing directories, my other most common task was listing directory contents. Looking at the history file, I saw that an ls usually comes right after a cd.

This indicates that automatically listing directory contents after a directory switch should save time.

But, instead of listing all of the directory contents, it's more useful list only the most recently used files.

To do this, we need to augment PROMPT_COMMAND to check if the directory has changed, and if so, to run a command like ls -t | head -7, which tells us the 7 most recently modified files in the directory:

export LASTDIR="/"

function prompt_command {

  # Record new directory on change.
  newdir=`pwd`
  if [ ! "$LASTDIR" = "$newdir" ]; then 
    /path/to/markdir.pl

    # List contents:
    ls -t | head -7
  fi

  export LASTDIR=$newdir
}

export PROMPT_COMMAND="prompt_command"

Recency is actually a good approximation for frequency. Seeing the seven most recently modified files in a directory tends to quickly jog my mind and remind me where I left off.

Switching to the last directory on login

Another comman pattern in my work-flow is to open a second terminal window and then move to the directory I'm currently in. Once again, this change directory step should be eliminated by having new terminals open to the most recently used directory. Modifying ~/.bash_profile yet again provides the desired behavior.

# Change to most recently used directory:
if [ -f ~/.lastdir ]; then
    cd "`cat ~/.lastdir`"
fi

export LASTDIR="/"

function prompt_command {

  # Remember where we are:
  pwd > ~/.lastdir

  # Record new directory on change.
  newdir=`pwd`
  if [ ! "$LASTDIR" = "$newdir" ]; then 
    /path/to/markdir.pl

    # List contents:
    ls -t | head -7
  fi

  export LASTDIR=$newdir
}

export PROMPT_COMMAND="prompt_command"

The cdto script

The scripts above use a cdto command, which changes to a new directory from within a script. To accomplish this, cdto actually writes to the file ~/.lastdir, and starts a new shell.

#!/bin/bash

echo + Switching to $*
echo + Press CTRL-D to return to `pwd`

echo $* > ~/.lastdir
bash --login

The advantage of this is that a user can quickly change to a new directory, do work and "pop" back to the old directory by exiting the shell.

Recent files in emacs

Emacs can remember recent files as well. Add the following to ~/.emacs:

(require 'recentf)
(recentf-mode 1)
(setq recentf-max-menu-items 25)
(global-set-key "\C-x\ \C-r" 'recentf-open-files)

Then C-x C-r to see recently opened files.

Conclusion

The principle of frequency isn't profound; but it is effective. Just because the console is a powerful way to compute doesn't mean we shouldn't look for ways to make it more efficient. Logging activity and then mining that data is a low-cost way for console users to discover their most frequent tasks. And, once those tasks are discovered, users can devise methods (or additional shell scripts) to make them go faster.

Update: Easier ways

One of the beautiful things about Unix is that you can use it for a decade and still find new things to learn every day.

Multiple readers have pointed out that much of these scripts could be replaced with simple, elegant shell hacks.

For instance, you can implement frequency as:

 $ sort | uniq -c | sort -g

And, you can count commands with:

 $ history | cut -c8- | sort | uniq -c | sort -rn | head

For jumping between directories, try autojump.

Related pages


[article index] [] [@mattmight] [+mattmight] [rss]