More resources
-
The best book I know of for console hacks is
Linux Server Hacks: 100 Industrial-Strength Tips and Tools
. It's loaded with fiendishly creative scripts and shortcuts. (Almost all of the hacks apply to any Unix-derived OS, including Mac OS X.)
- It turns out that my colleague Eric Eide did a masters thesis on an adaptive interface to a Unix shell, Valet. Valet uses a mixture of system knowledge and heuristics to "detect and correct the kinds of mistakes that experienced users make most frequently: typographical errors, file location errors, and minor syntactic errors." Neat!
Logging and mining console activity
Before you can exploit the principle of frequency, you need an unbiased record of what it is that you do most frequently. You might even be surprised by what your most frequent tasks are.
Fortunately, shells like bash already have some of that data
in the form of the history command.
The history command prints a list of recent commands.
bash stores this history (by default) in ~/.bash_history.
You should periodically examine your frequently used commands, and
find ways to execute them quickly.
In many cases, a one-letter alias (e.g. e for
emacs) is surprisingly effective.
To find frequent commands, you'll need a script to compute
counts for the contents of the history file.
For this, I created a perl script called frequency:
my %counts = () ;
while (my $cmd = <STDIN>) {
chomp $cmd ;
if (!$counts{$cmd}) {
$counts{$cmd} = 1 ;
} else {
$counts{$cmd}++ ;
}
}
foreach $k (keys %counts) {
my $count = $counts{$k} ;
print "$count $k\n" ;
}
It counts the number of times that each line occurs on stdin, and then dumps out a report.
For example:
$ cat ~/.bash_history | frequency | sort -rn | head 78 ls 25 recent 17 ls -l 16 mutt 12 cd matt.might.net 12 cd 10 echo hi 9 emacs .bash_profile 7 cd articles/ 5 sudo perl -MCPAN -e 'install DBD::mysql' ;
This example is clearly representative of about a day's worth of
my recent activity, since my history only goes back for 500 lines.
To get a truly representative sample, you'll probably want to increase
your history size.
Inside your .bash_profile, you should add:
export HISTFILESIZE=10000 # Record last 10,000 commands export HISTSIZE=10000 # Record last 10,000 commands per session
At the same time, while it's useful to look at the full commands,
it's more instructive to look at just the commands themselves. For
this, you can use cut:
$ cat ~/.bash_history | cut -d" " -f1 | frequency | sort -rn | head 84 ls 67 cd 48 recent 47 history 37 sudo 26 emacs 22 echo 15 mutt 15 ecmd 13 r
It shouldn't be any surprise to an experienced console user that nearly a third of all my commands are to list files and change directories. Since these commands are so common, I'll focus on optimizing those two. That's where my biggest "bang for buck" comes from.
Logging directories to MySQL
Switching directories is a common console task. In any given week, however, perhaps 80-90% of a your time will be spent in a small working set directories. This frequency provides a chance to optimize.
What you should do is log the directories visited, and then create a script for quickly switching to one of the most visited directories.
To log directories, you can use (1) a MySQL database to store the
data, and (2) the PROMPT_COMMAND environment variable to
insert a script into the console that runs every time a console prompt
displays.
(Yes, this is an abuse of the PROMPT_COMMAND variable, but that's OK.)
Setting up the table is straightforward:
CREATE TABLE dircounts
(path VARCHAR(255) PRIMARY KEY NOT NULL,
count INT NOT NULL DEFAULT 0,
time TIMESTAMP NOT NULL)
Then, you need a script, markdir.pl, to mark the current directory:
use strict;
use warnings;
use DBI;
# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;
my $dbh = DBI->connect("DBI:mysql:$database",
"$user",
"$password")
|| die "Connection failed: $DBI::errstr";
my $qpwd = $dbh->quote($ENV{'PWD'}) ;
$dbh->do("INSERT INTO dircounts(path,count) ".
"VALUES ($qpwd,1) ".
"ON DUPLICATE KEY UPDATE count=count+1") ;
$dbh->disconnect();
Before each time the prompt is displayed, bash evaluates the
contents of the environment variable PROMPT_COMMAND.
This variable is just the right hook to monitor the current directory;
for instance, in ~/.bash_profile, we can add:
export LASTDIR="/"
function prompt_command {
# Record new directory on change.
newdir=`pwd`
if [ ! "$LASTDIR" = "$newdir" ]; then
/path/to/markdir.pl
fi
export LASTDIR=$newdir
}
export PROMPT_COMMAND="prompt_command"
Damping the frequency
You should damp the counts over time, so that if you spend one week accessing a directory a lot, but never again thereafter, it disappears from the frequent directories set quickly. Otherwise, you could have a defunct directory that sticks around for a long time after you've finished a major project.
To do this, I recommend a cronjob that runs a damping script once a week:
use strict;
use warnings;
use DBI;
# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;
my $dbh = DBI->connect("DBI:mysql:$database",
"$user",
"$password")
|| die "Could not connect to database: $DBI::errstr";
my $qpwd = $dbh->quote($ENV{'PWD'}) ;
# Cut all counts in half:
$dbh->do("UPDATE dircounts SET count = count/2");
$dbh->disconnect();
The script exponentially decays counts, so that as soon as you stop using a directory regularly, its relative importance fades quickly.
Run crontab -e and then add:
0 0 * * 0 /path/to/damping/script
A frequency-adaptive change directory
With a database of all directory visits, it's easy to write a perl script, f,
that lists these directories and permits changing to them with just a couple keystrokes:
#!/usr/bin/env perl
use strict;
use warnings;
use DBI;
my $target = shift ;
# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;
my $dbh = DBI->connect("DBI:mysql:$database",
"$user",
"$password")
|| die "Could not connect to database: $DBI::errstr";
my $sth = $dbh->prepare("SELECT path ".
"FROM dircounts ".
"ORDER BY count DESC LIMIT 7") ;
$sth->execute() ;
my $count = 1 ;
if (!$target) {
print "+ change to directory with: f <number>\n" ;
}
while (my @row = $sth->fetchrow_array) {
if ($target) {
if ($count == $target) {
$sth->finish() ;
$dbh->disconnect();
system(". cdto '@row'") ;
exit ;
}
} else {
my $home = $ENV{'HOME'} ;
my $path = "@row" ;
$path =~ s/$home/~/ ;
print "${count}: $path\n" ;
}
$count++ ;
}
$dbh->disconnect();
For example:
$ f + change to recent directory with: f <number> 1: ~ 2: ~/family/bertrand 3: ~/ucombinator/private/grants 4: ~/ucombinator/private/papers 5: ~/ucombinator/private/projects 6: ~/matt.might.net 7: ~/talks
A recency-adaptive change directory
Another useful perl script, r, lists the most
recently accessed directories.
#!/usr/bin/env perl
use strict;
use warnings;
use DBI;
my $target = shift ;
# DB settings
my $host = "localhost" ;
my $database = "..." ;
my $user = "..." ;
my $password = "..." ;
my $dbh = DBI->connect("DBI:mysql:$database",
"$user",
"$password")
|| die "Could not connect to database: $DBI::errstr";
my $span = 1000 * 60 * 60 * 7 ;
my $q = "SELECT path ".
"FROM dircounts ".
"WHERE UNIX_TIMESTAMP(time) > UNIX_TIMESTAMP(now()) - $span ".
"ORDER BY time DESC LIMIT 7" ;
my $sth = $dbh->prepare($q) ;
$sth->execute() ;
my $count = 1 ;
if (!$target) {
print "+ change to directory with: r <number>\n" ;
}
while (my @row = $sth->fetchrow_array) {
if ($target) {
if ($count == $target) {
$sth->finish() ;
$dbh->disconnect();
system(". cdto '@row'") ;
exit ;
}
} else {
my $home = $ENV{'HOME'} ;
my $path = "@row" ;
$path =~ s/$home/~/ ;
print "${count}: $path\n" ;
}
$count++ ;
}
$dbh->disconnect();
List recent files on directory change
In addition to changing directories, my other most common task
was listing directory contents.
Looking at the history file, I saw that an ls usually
comes right after a cd.
This indicates that automatically listing directory contents after a directory switch should save time.
But, instead of listing all of the directory contents, it's more useful list only the most recently used files.
To do this, we need to augment PROMPT_COMMAND to check if
the directory has changed, and if so, to run a command like ls
-t | head -7, which tells us the 7 most recently modified files
in the directory:
export LASTDIR="/"
function prompt_command {
# Record new directory on change.
newdir=`pwd`
if [ ! "$LASTDIR" = "$newdir" ]; then
/path/to/markdir.pl
# List contents:
ls -t | head -7
fi
export LASTDIR=$newdir
}
export PROMPT_COMMAND="prompt_command"
Recency is actually a good approximation for frequency. Seeing the seven most recently modified files in a directory tends to quickly jog my mind and remind me where I left off.
Switching to the last directory on login
Another comman pattern in my work-flow is to open a second
terminal window and then move to the directory I'm currently in.
Once again, this change directory step should be eliminated
by having new terminals open to the most recently used directory.
Modifying ~/.bash_profile yet again provides the desired
behavior.
# Change to most recently used directory:
if [ -f ~/.lastdir ]; then
cd "`cat ~/.lastdir`"
fi
export LASTDIR="/"
function prompt_command {
# Remember where we are:
pwd > ~/.lastdir
# Record new directory on change.
newdir=`pwd`
if [ ! "$LASTDIR" = "$newdir" ]; then
/path/to/markdir.pl
# List contents:
ls -t | head -7
fi
export LASTDIR=$newdir
}
export PROMPT_COMMAND="prompt_command"
The cdto script
The scripts above use a cdto command, which changes to a new directory from within a script.
To accomplish this, cdto actually writes to the file
~/.lastdir, and starts a new shell.
#!/bin/bash echo + Switching to $* echo + Press CTRL-D to return to `pwd` echo $* > ~/.lastdir bash --login
The advantage of this is that a user can quickly change to a new directory, do work and "pop" back to the old directory by exiting the shell.
Recent files in emacs
Emacs can remember recent files as well.
Add the following to ~/.emacs:
(require 'recentf) (recentf-mode 1) (setq recentf-max-menu-items 25) (global-set-key "\C-x\ \C-r" 'recentf-open-files)
Then C-x C-r to see recently opened files.
Conclusion
The principle of frequency isn't profound; but it is effective. Just because the console is a powerful way to compute doesn't mean we shouldn't look for ways to make it more efficient. Logging activity and then mining that data is a low-cost way for console users to discover their most frequent tasks. And, once those tasks are discovered, users can devise methods (or additional shell scripts) to make them go faster.
Update: Easier ways
One of the beautiful things about Unix is that you can use it for a decade and still find new things to learn every day.
Multiple readers have pointed out that much of these scripts could be replaced with simple, elegant shell hacks.
For instance, you can implement frequency as:
$ sort | uniq -c | sort -g
And, you can count commands with:
$ history | cut -c8- | sort | uniq -c | sort -rn | head
For jumping between directories, try autojump.