Sunday, February 24, 2013

Handy bash substitution

Working on the command line is awesome. But there are times where all the typing gets unwieldy. No one likes to type a long file name a lot of times. Sure, using the up arrow and tab keys can make things faster sometimes, but I'll show you some tricks to make your life easier.

Let's start with a really common operation. You download a tarball, extract, cd and have fun. Usually, you'd do this:
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf linux-3.7.9.tar.bz2
$ cd linux-3.7.9
$ bob's your uncle
>
Using the Tab key, you can save a lot of typing:
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf l<TAB>
$ cd l<TAB>
$ bob's your uncle
>
However, this is highly dependant on the contents of your directory, names in your path and auto-complete configurations. I'm not being picky here. While testing these commands, I had to press Tab twice. First, I had a directory called "libs", so it stopped at "li". Second, I had an episode of the Linux Action Show in a file called linuxactionshowep248.mp4, so it stopped at "linux". And, after you decompress the file, it stops at "linux-3.7.9". Also, if your auto-completion isn't smart enough to filter names that aren't files or directories when you type Tab after cd and tar, you could have many more conflicts.

I don't mean to bash (ha!) on auto-completion, just point you to more efficient alternatives: bash (get it?) has a nice feature called substitutions. You may have come across it before if you ever tried to add an unescaped exclamation mark (henceforth mentioned using the niftier name "bang"):
$ echo "Bang!"
bash: !": event not found
Or got surprised by it:
$ echo "balrog:!you shall not pass:/nonexistent:/bin/false" >> /etc/passwd
echo "balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false" >> /tmp/test

$ sleep 3d; grep balrog /etc/passwd
balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false
That is because the bang is used to reference previous commands. The simplest is substitution of the last command:
$ echo billy
billy
$ !!
echo billy
billy
One thing I should mention: to avoid chaos and destruction (or at least inform you that chaos and destruction have happened) the command with all substitutions made is printed before its output. That is the fourth line in the example, showing that !! was substituted by echo billy.

This substitution is simple and needs no explanation, but it raises a question: what is it useful for? Have you ever typed a (possibly long) command, just to get the following message after running it?
$ cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
cat: /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_: Permission denied
You can solve this by typing <UP><HOME>sudo<SPACE> (if you type <UP><LEFT>{201}sudo<SPACE>, I will personally punch you in the face). But you can save a lot of typing just using
$ sudo !!
sudo cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
[sudo] password for billy:
Here, "!!" got substituted by the last command, as you can see on the echo. Some other uses I found (which you may not understand if you're not or is on the path to enlightenment becoming a bash enthusiast) were:
$ cat my_files
file1
file2
file3
$ cat $(!!)
cat $(cat my_files)
contents_of_file1
contents_of_file2
contents_of_file3
$ perl -e 'bang your head on the keyboard'
# god only knows
$ echo "!!" > file_to_save_command # This could be done by clever use of Ctrl-x Ctrl-e, but sometimes you're in a hurry. That's alright, as long as you do it carefully.
When ! is followed by a number n, it executes the nth command in the history (which you can check with the history command). More useful, though, is that, if n is negative, it will execute the last nth command:
$ echo some
some
$ echo thing
thing
$ !-2
echo some
some
$ !-3 other thing
echo some other thing
some other thing
Without thinking much, you can discover that !! is just an easier way to spell !-1.

Now, these are useful techniques, but we are just getting started! Usually, you don't need the whole command, but just one or a few arguments. Common scenario:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd ef
or how about:
$ mv some_directory a_diferent_name_for_some_directory
$ cd a_diferent_name_for_some_directory
It is very common to address the same file on multiple, successive commands. You can, again, save some typing:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd !$

cd ef
$ mv some_directory a_diferent_name_for_some_directory
$ cd !$

cd some_directory a_diferent_name_for_some_directory
As you can see, !$ is substituted by the last argument of the last command. If you know your regular expressions, you can make a link with the "end of line" symbol. But there's more. This is just a nice shortcut to the more general form !n:m. This means: from the nth, get the mth argument.
$ tar -f my_tar -cz file1 file2 file3
$ du -sh !:2
4.0K    my_tar
$ du -sh !-2:4-6
4.0K    file1
4.0K    file2
4.0K    file3
What goes before the colon can be any of the previous substitutions: !!, !n or !-n. Note that you can specify a range using :n-m, which will get substituted by the parameters n to m. If you need them in a different order, or some non-contiguous arguments, just use more than one substitution:
$ diff ours theirs
$ # gah
$ diff !:2 !:1
diff theirs ours
Alright, I guess this is enough information for a single post. Have fun playing with The Bash and see you soon for more advanced uses of history substitution.

P.S.: I couldn't leave without some closing notes.

  • I learned everything shown here from scattered information on the internet. I decided to compile them because I never found a document which explained them in detail AND without burying them on other bash-related commands.
  • That said, a good reference (not so good for self-learning, but, if you read the text, you should be okay) is bash's man page. Search for "Event Designators" or "\!\!" (escaping is mandatory):
$ man bash
/Event Designators<CR>
  • While all this may seem hard to use practically,  I recommend you just keep it in a dark corner of your mind. I promise you'll start to find places where you can apply it. They can be used to write commands quickly, especially several one-shot commands in a row. Just don't try forcing the use when it's not necessary. After all, they were created to save time.
  • If you ever got stabbed by not knowing how the bang works (like the examples I showed in the beginning), you can avoid getting stabbed again by escaping the bangs, either with a backslash or single quotes:
$ echo Bang\!
Bang!
$ echo 'balrog:!you shall not pass:/noexistent:/bin/false' >> /etc/passwd
  • More of a side note, but to get that humongous file name, I discovered a new (string of) command(s) to get the longest line from a text.
$ find /sys/ | awk '{print length, $0;}' | sort -nr | head -n 1
  • You can probably do the same thing (although it might be a little slower) using a while loop and another bash trick: while $variable (or ${variable}) will give you the contents of a variable, ${#variable} will give you the length of that content. 
$ find /sys/ | while read i; do echo "${#i} $i" | sort -nr | head -n 1
  • Apparently, Jethro Tull's Too Old to Rock 'n' Roll, Too Old to Die generates pretty long file names (the title song is the second only to a really long cache file name on my file system).

Wednesday, February 6, 2013

An image is worth a thousand megabytes

Now we have all the components we need to run our Raspberry Pi, we can start preparing them to actually work together. The first thing we need to do is get the boot record, partitions and operating system on the SD card, so our little friend can boot and run a true system. Thankfully, some nice folks have already done the hard work, so we can jump many of the complex steps.

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem


I'm going to use the Raspbian distribution, which is based on the famous Debian (GNU/)Linux distribution, and includes many of the standard software packages already prepared to run on the Pi. If you haven't already, grab Raspbian's image, which you can find on Raspberry Pi Foundation's downloads web page.
$ wget http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip
Note that this is the current version as I write this post. Visit the page to get the most recent version. An optional but recommended step after you download the image is to check its SHA-1 checksum. That will ensure nothing has been corrupted during the download, which is always a good thing to know, specially when dealing with sensitive data such as disk images. To do that, you can use the sha1sum program:
$ sha1sum 2012-12-16-wheezy-raspbian.zip
514974a5fcbbbea02151d79a715741c2159d4b0a 2012-12-16-wheezy-raspbian.zip
Compare the output with the hash on the SHA-1 section of the download page. Or, if you have the SHA-1 stored in a file:
$ sha1sum -c 2012-12-16-wheezy-raspbian.zip.sha1 2012-12-16-wheezy-raspbian.zip: OK
If you don't get the appropriate result, just download the file again and check it, until you get it right. Now we are going to copy the image to the SD card. Before you insert the SD card on your computer (you may need an external SD card reader, depending on the ports available on your machine), execute the command
$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sda6  /dev/sda8
/dev/sda1  /dev/sda3  /dev/sda5  /dev/sda7
That will list all the disks available. If you prefer a more friendly output, you can use
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6        46G   30G   14G  69% /
udev            2.9G  4.0K  2.9G   1% /dev
tmpfs           1.2G  1.1M  1.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.9G   76K  2.9G   1% /run/shm
/dev/sda4       107G   50G   53G  49% /mnt/data
This will list the devices used by the mounted filesystems. Now you can insert the SD card and run the command again. You will notice that new entries will appear on the output. It will usually be /dev/sdb* entries, if you have only one disk. That is the device file for the card. If the card is automatically mounted, you can see the size on df's output, so you can be sure that it is the right device.

A really important note here: you need to get the correct device name. We are going to use a program that does raw block copying, which can destroy your data if you target the wrong device. The device mounted on root ('/') is probably not the device you want, for example.

Another important thing: if the card already has partitions, some extra device files may appear, such as /dev/sdb1, /dev/sdb2 and so on. We need the device, not any partition, so be sure to use the device file with no number suffix. When you have the device name, make sure the device is unmounted so we can write data to it:
$ umount /dev/sdb*
Then use good old dd to copy the image:
$ dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M
A couple things deserve mention. It is always good to check (and re-check) the parameters, to make sure the image is the if (input file) value and the device is the of (output file) parameter. The bs parameter stands for block size, and is the size of each chunk of data copied at a time. Larger values make the process faster. Usually, 4mb is a reasonable Following UNIX's philosophy, dd doesn't output anything while it is working. Since it can take quite some time to finish, you may think it's froze. But don't worry, it hasn't. If you want to be absolutely sure, send the program a signal and it will output the progress:
$ ps aux | grep dd
  4821 pts/2    R+     0:33 dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M
$ kill -USR1 4821
When it is done, run
$ sync
$
to make sure everything is properly written and remove the card. Now insert the card on the Raspberry, plug the power supply and, if everything worked, you will get a nice show of lights as it boots. You can plug the HDMI to a monitor and see the boot messages. Here's a (somewhat light-impaired) gallery:




I know I promised we'd get it to work! Well, if you (unlike me) think this isn't actual work, check the next posts, where we will put the Pi to good use.