Uploading files the unix way

I just did something which I thought was worth sharing. It shows a couple of fundamental techniques in unix systems used in a context where most people would not think of.

The context: my mom sent me a recipe of potato souffle (I hope google translate is doing the right thing here, it seems like it is). But as most of e-mails coming from non-tech people, there was a lot of noise along the actual recipe text. Since I wanted to view it in my cell phone while I cooked, I decided to clean it up a bit.

So did I: copied the text to vim (which got rid of most of the noise immediately), edited a bit and it was ready:

    Ingredientes
        450 gramas: Batata cozida
        200 gramas: Queijo ricota caseiro – Guloso e Saudável
        4 colheres de sopa: Queijo gorgonzola
        3 colheres de sopa: Farinha de trigo branca
        300 mililitros: Leite desnatado
        2 unidades: Gema de ovo
        3 unidades: Clara de ovo
        1 colher de sopa: Fermento em pó
        1/2 unidade: Cebola ralada
        4 colheres de sopa: Salsa fresca cortada

    Modo de preparo
        Amasse os queijos, reserve
        Amasse as batatas, reserve
        Preaqueça o forno a 200ºC
        Bata as claras em neve com uma pitada de sal, reserve
        Numa vasilha misture as gemas, a farinha, o leite, a cebola, a batata, os queijos, a salsinha
        Junte delicadamente o fermento e as claras em neve
        Coloque o suflê de batata em ramequins e leve ao forno por 30 minutos ou até assar
        Sirva de imediato o suflê de batata com queijos, acompanhado de salada

Now all I needed was a way to get it on my cellphone. I could use dropbox or any of the related services, but that didn't seem exciting enough[1]. So I thought: well, I'll just put it in my webserver and access it in my mobile browser.

Fair enough. But this is a file opened in vim[2]. After a little thinking (which lasted for approximately 423ms), this is what I did. From inside vim, I executed the following command:

:!ssh raspberrypi 'nc -l 31415 > /srv/http/sufle.txt' &

Which launched an ssh client connecting to my webserver[3] running the netcat command. If you are not familiar with netcat, think of it as the traditional cat command which reads and writes to sockets instead of files. Here, I'm telling it to listen (-l) on port 31415 and output anything read to the file /srv/http/sufle.txt, where /srv/http is the document root of the webserver.

Since I added the ampersand to the end of the command, the command is launched in the background and just stays there waiting for input, releasing the shell immediately so I can continue using vim. Now that we're back, another vim command:

:w !cat > /dev/tcp/raspberrypi/31415

Here we are telling vim to write the contents of the buffer (:w). Normally, we would give it the name of a file to write, but the :write command accepts a special syntax, where instead of a file name, we put a ! and the rest of the line is interpreted as a command[4]. This command is executed in a standard unix shell, but with stdin redirected to read the contents of the buffer. As an example, try opening a new empty buffer in vim (c-w n), writing something and executing the command[5]:

:w !cowsay

Continuing the analysis of the command, we are using the :w ! command to write to the program run by the shell, which is

$ cat > /dev/tcp/raspberrypi/31415

Well, you don't need to be a unix wizard to know what this is doing. cat is a command that reads the contents from a list of files (or stdin if it's invoked without arguments, as it is here) and writes them in the same order to its stdout. Since its stdin will read the contents of the vim buffer, we already know the effect of this command: the contents of the buffer will get written to a file.

But what is that funny file starting with /dev/tcp/? That is feature of bash (I don't know about other shells). You can read about using the command[7]:

$ man -P 'less -p /dev/tcp' bash

What we are doing here is telling bash to write the data coming from cat's output to port 31415 on the host raspberrypi. Remember our little ssh friend we left running in the background a while ago? He's not done much since[8], but now there is data coming from the socket it's listening on, so the os wakes him up to do its job.

And so it does, reading the data from the socket and writing it to stdout. Remember what that is? That's the file we passed to the shell, /srv/http/sufle.txt. And so our long journey comes to an end. When nc finishes reading and writing the data, it dies[9] and so does the ssh client process, which was also sitting there, waiting for it's child process do die[10]. The shell that spawned the ssh client has long died: it had nothing more to do because we started the process asynchronously (using the & on the command line), so that's the end of that (process) family tree[11].

The result? The contents of the buffer have now been transfered and written to the file /src/http/sufle.txt on the remote host raspberrypi.

Conclusion

You might be thinking: "What the hell? You are telling me all this is simpler than dropbox?". And I would be lying if I said "yes". But the point here is not that this is simple. In fact, it's the base of one of the most complex systems in computer history: the operating system. And even though I have detailed some parts of the processes, there are a lot, and I really mean *a lot* more things going on behind the scenes here[12].

But if you step back, you will realize that all that was needed were two commands:

ssh raspberrypi 'nc -l 31415 > /srv/http/sufle.txt'
cat > /dev/tcp/raspberrypi/31415

What I'm trying to present here is the incredible complexity and richness that can be achieved by using shell commands and unix concepts like input and output redirections and socket communication. Another goal was to show how a powerful text editor incorporates this concept in its design, taking advantage of of the powerful features of the operating system to extend its capabilities[13].

And I bet I can type these two commands faster than your dropbox client can transfer the file to your cellphone[14]. Now, I you'll excuse me, there is a potato souffle that needs to be cooked.

Bonus

While doing some scientific tests on the cowsay program (trying to make the cow say the text of this post), I discovered perl has a special mention of my blog on its source code[6]:

:w !cowsay
This shouldn't happen at /usr/share/perl5/core_perl/Text/Wrap.pm line 84, <stdin> line 63.

shell returned 255

Notes

1. What sane geek uses dropbox, anyway?
2. Actually just a buffer, since I didn't even write it to a file.
3. A rπ.
4. The space after :w is important here. If we wrote :w!cat ..., it would be interpret as the :write! command, which is not what we want here. See :help :write! on vim.
5. What do you mean "I don't have cowsay installed"?
6. You can check it here (thanks sources.debian.net for their awesome service).
7. Don't worry, that will only open a man page passing a special flag to the pager telling it to go directly to the right section.
8. He hasn't been doing anything, actually, thanks to the magic of process scheduling and blocking io syscalls, so no resources are wasted (except for a few bytes on the process and file tables).
9. Ok, it terminates its execution, "dies" sounds so dramatic.
10. Just when you thought it could not get more dramatic.
11. Ok, that was on purpose =)
12. Including, for example, the whole file buffer cache algorithm that makes this whole process absurdly efficient, even though we seem to be copying data around dozens of times. That is the subject of whole book chapters (like this one).
13. My trademark phrase is: "do *that* in gedit".
14. After it has sent your data, maybe without using encryption, across the internet and sent a copy of the recipe to the NSA for analysis.

A post-mortem

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-morten

A lot has happened since my first rπ post. I've used that little thing to death (quite literally), as a ssh box, firewall, general server, web server, web browser, movie box and a whole lot more. Then, I've had the SD card die on me. For me, it was a perfect little linux computer that could stay on all the time, consuming little-to-no power, making absolutely no noise, occupying minimal space. In a few words: really, really awesome. And all that for 35 dollars!

As promised, I'll post here the steps to get from a rπ on a closed box to a little box of awesomeness. I know there a ton of posts like these, but hey, storage is free (for now) on blogger. If you haven't read the first post, this is a good time to do it. I will first make a few post-morten (again, literally) observations about my first assumptions.

SD card: let's start with the most important thing. When I first went shopping for the various attachments, I got the most inexpensive card I found. The only requisite was that it had at least a 4gb storage. After it died, I decided to spend a few extra bucks and get a decent one. Man, I was surprised. There isn't any way of comparison between the π before and now. Instead of moaning the old days, I'll just say this: get a decent, class 10 SD card and you'll have nothing to worry about. And since these things get cheaper and cheaper, get a 16gb one.

Power supply: again, when I started using the π, I just used my cell phone charger, which, by extreme luck, was compatible and provided almost exactly the amount of power needed. Then I started using a wireless adapter sometimes. Then I started using hdmi output sometimes. Then I tried to use both at the same time. And it died. Not that it does any harm to the board, but any moderate load will simply power-off the device. So, if you want to use more than the π itself, get an appropriate power supply. Those on the farnell website worked just fine for me. And just for completeness sake, mine is 1000mA.

HDMI: I did my first tests on my living room TV. But after a while (as addiction got worse), I felt the need to use it on my bedroom, where I have only an old computer monitor with VGA input. So I got a VGA-to-HDMI adapter and things worked fine (well, almost, more on that later).

OS: the first distribution I installed was raspbian, mainly because I was used to debian and, by the time I started playing, there weren't many alternatives. Now we have pidora and even riscos. But anyway, raspbian is a really good distribution and I never had a problem with it. But I am restless. Since I was going to reinstall the OS from /dev/zero, I decided to jump from the cliff and install arch. For no special reason, just to try something else. So, the instructions from now on will be arch centered, although many will be usable on other distributions.

Man, I write a lot. Let's stop right here and I'll get to the first installation steps later.

That moment you realize something really wrong is going on

python mocking

If you are into unit testing, you probably have been introduced to mocking. And if that is the case, you probably already have been bitten by it. Mocking requires some understanding of code execution, importing and name resolution that most people lack when first encountering such situations. In python, mocking is a relatively simple process, if you analyze carefully what needs to be done.

Mocking simply means replacing an object with another. This is usually done to avoid instantiating costly systems or to change the behaviour of a system. To begin with a simple example:

def some_function_in_your_code(a):
    if a.do_something():
        return 3.14159
    else:
        return 6.26318

def some_other_function_in_your_code():
    # ...
    a = create_complicated_object(*thousand_parameters)
    result = some_other_function_in_your_code(a)
    # ...

Here we have a function some_other_function_in_your_code that uses an object. Somewhere else in your code, some_other_function_in_your_code creates that object, which is a complicated process that involves hundreds of operations. If you just want to test some_other_function_in_your_code, you shouldn't need to go through this whole process[1].

To avoid that, we can use mocking. Notice that all we need to test on some_other_function_in_your_code is that the argument passed has a member called "do_something" that can be called with no arguments and returns something that can be converted to bool. There are many ways to do that, but to keep things short, I'll skip right to the library I use most of the times, mock.

The main component of the mock library is the Mock class, which is basically an empty object with a few useful characteristics (be sure to check the documentation, because they are really useful). Two of them are important for this discussion. First, every time you access an attribute of a Mock object, another Mock object is created and assigned to the attribute being accessed.

>>> import mock
>>> m = mock.Mock()
>>> print(m)
<Mock id='140095453080976'>
>>> print(id(m.some_attribute))
140095453131664
>>> print(m.some_attribute)
<Mock name='mock.some_attribute' id='140095453131664'>
>>> print(id(m.some_attribute))
140095453131664

As you can see, when we tried to access some_attribute, a Mock object was created for us. We could do it by hand with a single line of code, but it makes it makes the code easier, shorter and cleaner.

The other feature of Mock objects is the return_value attribute. Whatever is contained in this attribute gets returned when the object is called as a function object [2].

>>> import mock
>>> m = mock.Mock()
>>> m.return_value = u'the return value'
>>> m()
u'the return value'

Using this two techniques, we can now test our function:

import mock

def your_test():
    a = mock.Mock()
    a.do_something.return_value = True
    if some_function_in_your_code(a) != 3.14159:
        raise Exception(u'return wasn't 3.14159.')
    a.return_value = False
    if some_function_in_your_code(a) != 6.28318:
        raise Exception(u'return wasn't 6.28318.')

Let's break that down. We first create a Mock object to represent the argument passed to the function. The next line takes care of the two things we need to test the function: the do_something attribute and its return value. Then, all we have to do is call the function, passing the mocked argument, and check the return value. After that, repeat the process, this time with a different return value.

That was the easy part

This first section was easy, nothing you couldn't find out with a quick search on the internet. But the real world is not that pretty (at least mine isn't). The trickiest situation is when you have to change the behaviour of something inside a function, but you don't pass that something as an argument. Suppose we have this:

# some_file.py
import random

def f():
    if random.random() < 0.5:
        return 3.14159
    else:
        return 6.28318

How can you test that function if the value tested is conjured from oblivion in the middle of the function? Fear not, you can actually do it. The trick here is to mock the random function from the random module before the first line of f is executed. Let's start with this simple example, just to get the basic idea:

>>> import random
>>> random.random()
0.5212285734499994
>>> random.random()
0.40492920488281725
>>> import mock
>>> random.random = mock.Mock(return_value=0.5)
>>> random.random()
0.5

This seems pretty simple. But here is where everybody gets lost:

>>> import random
>>> import mock
>>> random.random = mock.Mock(return_value=0.5)
>>> import some_file
>>> some_file.f() == 6.28318 or raise ThisShouldNotBeHappeningException()

This may not explode the first time, but you have a 50% chance of getting an exception. To see why, you can put a print random.random inside f and see that the mock didn't work. To understand why, we have to dig a little deeper.

Import

What happens when you run import random? You can watch this video to know exactly what. Or you can just continue reading to get a summary. Or both. Anyway:

>>> locals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', '__doc__': None, '__package__': None}
>>> import random
>>> locals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', 'random': <module 'random' from '/usr/lib/python2.7/random.pyc'>, '__doc__': None, '__package__': None}

In python, locals is a built-in function that returns the local variables accessible on the current state of execution (don't believe me, run print(__builtins__.locals)). When you execute import random, the interpreter does its magic to find the module and load it, but more important, it creates an entry called "random" on the current namespace referring to the module loaded. The critical part here is "current namespace". Try this:

>>> def f(): print(locals())
>>> import random
>>> f()
{}

Here, importing random didn't affect the namespace on f. The same thing applies to namespaces of other modules. Our example fails because the namespace on the some_file module is different than the namespace where we run our tests. To change the namespace of some_file, we have to do it explicitly:

>>> import some_file
>>> some_file.random.random = lambda: 0.5
>>> some_file.f() == 6.28318 or raise ThisShouldNotBeHappeningException()

You can run that many times if you don't trust me, but that will always succeed. And it does because we now are changing the correct namespace. You can check it by putting a print(random.random) on f again.

Being nice

Now you know how to mock, but there is something I must say before you leave. Always, always, ALWAYS restore any mock you do. Seriously. Even if you're sure no one will use the mocked attribute. You don't want to loose an entire day of work just to find out that the problem was an undone mock.

And doing it is so simple: store the original value on a variable and restore it after the operation. I like to do it as soon as the operation is complete, before anything else is executed, but you don't need to, if you're not paranoid. Just to clear any doubt, here is exactly how to do it:

>>> import random
>>> original_random = random.random
>>> random.random = lambda: 0.5
>>> # do something
>>> random.random = original_random

Now you have no excuse. Better yet, you can use another feature of the mock library called patch. But that would be an extension to an already long post. Maybe I'll cover it in the future. Anyway, happy mocking!

Notes

1: You shouldn't have complicated processes that involve hundreds of operations anyway, but that is another problem.

2: Curious to know what happens when you access return_value without setting it first? No? Well, I'll show you anyway:

>>> import mock
>>> m = mock.Mock()
>>> m.return_value
<Mock name='mock()' id='140095453168080'>

Since we didn't set it, we get the default behaviour of __setattr__, which is to create another Mock.

Handy bash substitution

Working on the command line is awesome. But there are times where all the typing gets unwieldy. No one likes to type a long file name a lot of times. Sure, using the up arrow and tab keys can make things faster sometimes, but I'll show you some tricks to make your life easier.

Let's start with a really common operation. You download a tarball, extract, cd and have fun. Usually, you'd do this:

$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf linux-3.7.9.tar.bz2
$ cd linux-3.7.9
$ bob's your uncle
>

Using the Tab key, you can save a lot of typing:

$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf l<TAB>
$ cd l<TAB>
$ bob's your uncle
>

However, this is highly dependant on the contents of your directory, names in your path and auto-complete configurations. I'm not being picky here. While testing these commands, I had to press Tab twice. First, I had a directory called "libs", so it stopped at "li". Second, I had an episode of the Linux Action Show in a file called linuxactionshowep248.mp4, so it stopped at "linux". And, after you decompress the file, it stops at "linux-3.7.9". Also, if your auto-completion isn't smart enough to filter names that aren't files or directories when you type Tab after cd and tar, you could have many more conflicts.

I don't mean to bash (ha!) on auto-completion, just point you to more efficient alternatives: bash (get it?) has a nice feature called substitutions. You may have come across it before if you ever tried to add an unescaped exclamation mark (henceforth mentioned using the niftier name "bang"):

$ echo "Bang!"
bash: !": event not found

Or got surprised by it:

$ echo "balrog:!you shall not pass:/nonexistent:/bin/false" >> /etc/passwd
echo "balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false" >> /tmp/test
$ sleep 3d; grep balrog /etc/passwd
balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false

That is because the bang is used to reference previous commands. The simplest is substitution of the last command:

$ echo billy
billy
$ !!
echo billy
billy

One thing I should mention: to avoid chaos and destruction (or at least inform you that chaos and destruction have happened) the command with all substitutions made is printed before its output. That is the fourth line in the example, showing that !! was substituted by echo billy.

This substitution is simple and needs no explanation, but it raises a question: what is it useful for? Have you ever typed a (possibly long) command, just to get the following message after running it?

$ cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
cat: /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_: Permission denied

You can solve this by typing <UP><HOME>sudo<SPACE> (if you type <UP><LEFT>{201}sudo<SPACE>, I will personally punch you in the face). But you can save a lot of typing just using

$ sudo !!
sudo cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
[sudo] password for billy:

Here, "!!" got substituted by the last command, as you can see on the echo. Some other uses I found (which you may not understand if you're not or is on the path to enlightenment becoming a bash enthusiast) were:

$ cat my_files
file1
file2
file3
$ cat $(!!)
cat $(cat my_files)
contents_of_file1
contents_of_file2
contents_of_file3
$ perl -e 'bang your head on the keyboard'
# god only knows
$ echo "!!" > file_to_save_command # This could be done by clever use of Ctrl-x Ctrl-e, but sometimes you're in a hurry. That's alright, as long as you do it carefully.

When ! is followed by a number n, it executes the nth command in the history (which you can check with the history command). More useful, though, is that, if n is negative, it will execute the last nth command:

$ echo some
some
$ echo thing
thing
$ !-2
echo some
some
$ !-3 other thing
echo some other thing
some other thing

Without thinking much, you can discover that !! is just an easier way to spell !-1.

Now, these are useful techniques, but we are just getting started! Usually, you don't need the whole command, but just one or a few arguments. Common scenario:

$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd ef

or how about:

$ mv some_directory a_diferent_name_for_some_directory
$ cd a_diferent_name_for_some_directory

It is very common to address the same file on multiple, successive commands. You can, again, save some typing:

$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd !$
cd ef
$ mv some_directory a_diferent_name_for_some_directory
$ cd !$
cd some_directory a_diferent_name_for_some_directory

As you can see, !$ is substituted by the last argument of the last command. If you know your regular expressions, you can make a link with the "end of line" symbol. But there's more. This is just a nice shortcut to the more general form !n:m. This means: from the nth, get the mth argument.

$ tar -f my_tar -cz file1 file2 file3
$ du -sh !:2
4.0K    my_tar
$ du -sh !-2:4-6
4.0K    file1
4.0K    file2
4.0K    file3

What goes before the colon can be any of the previous substitutions: !!, !n or !-n. Note that you can specify a range using :n-m, which will get substituted by the parameters n to m. If you need them in a different order, or some non-contiguous arguments, just use more than one substitution:

$ diff ours theirs
$ # gah
$ diff !:2 !:1
diff theirs ours

Alright, I guess this is enough information for a single post. Have fun playing with The Bash and see you soon for more advanced uses of history substitution.

P.S.: I couldn't leave without some closing notes.

• I learned everything shown here from scattered information on the internet. I decided to compile them because I never found a document which explained them in detail AND without burying them on other bash-related commands.
• That said, a good reference (not so good for self-learning, but, if you read the text, you should be okay) is bash's man page. Search for "Event Designators" or "\!\!" (escaping is mandatory):

$ man bash
/Event Designators<CR>

• While all this may seem hard to use practically,  I recommend you just keep it in a dark corner of your mind. I promise you'll start to find places where you can apply it. They can be used to write commands quickly, especially several one-shot commands in a row. Just don't try forcing the use when it's not necessary. After all, they were created to save time.

• If you ever got stabbed by not knowing how the bang works (like the examples I showed in the beginning), you can avoid getting stabbed again by escaping the bangs, either with a backslash or single quotes:

$ echo Bang\!
Bang!
$ echo 'balrog:!you shall not pass:/noexistent:/bin/false' >> /etc/passwd

• More of a side note, but to get that humongous file name, I discovered a new (string of) command(s) to get the longest line from a text.

$ find /sys/ | awk '{print length, $0;}' | sort -nr | head -n 1

• You can probably do the same thing (although it might be a little slower) using a while loop and another bash trick: while $variable (or ${variable}) will give you the contents of a variable, ${#variable} will give you the length of that content. 

$ find /sys/ | while read i; do echo "${#i} $i" | sort -nr | head -n 1

• Apparently, Jethro Tull's Too Old to Rock 'n' Roll, Too Old to Die generates pretty long file names (the title song is the second only to a really long cache file name on my file system).

An image is worth a thousand megabytes

Now we have all the components we need to run our Raspberry Pi, we can start preparing them to actually work together. The first thing we need to do is get the boot record, partitions and operating system on the SD card, so our little friend can boot and run a true system. Thankfully, some nice folks have already done the hard work, so we can jump many of the complex steps.

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem


I'm going to use the Raspbian distribution, which is based on the famous Debian (GNU/)Linux distribution, and includes many of the standard software packages already prepared to run on the Pi. If you haven't already, grab Raspbian's image, which you can find on Raspberry Pi Foundation's downloads web page.

$ wget http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip

Note that this is the current version as I write this post. Visit the page to get the most recent version. An optional but recommended step after you download the image is to check its SHA-1 checksum. That will ensure nothing has been corrupted during the download, which is always a good thing to know, specially when dealing with sensitive data such as disk images. To do that, you can use the sha1sum program:

$ sha1sum 2012-12-16-wheezy-raspbian.zip
514974a5fcbbbea02151d79a715741c2159d4b0a 2012-12-16-wheezy-raspbian.zip

Compare the output with the hash on the SHA-1 section of the download page. Or, if you have the SHA-1 stored in a file:

$ sha1sum -c 2012-12-16-wheezy-raspbian.zip.sha1 2012-12-16-wheezy-raspbian.zip: OK

If you don't get the appropriate result, just download the file again and check it, until you get it right. Now we are going to copy the image to the SD card. Before you insert the SD card on your computer (you may need an external SD card reader, depending on the ports available on your machine), execute the command

$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sda6  /dev/sda8
/dev/sda1  /dev/sda3  /dev/sda5  /dev/sda7

That will list all the disks available. If you prefer a more friendly output, you can use

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6        46G   30G   14G  69% /
udev            2.9G  4.0K  2.9G   1% /dev
tmpfs           1.2G  1.1M  1.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.9G   76K  2.9G   1% /run/shm
/dev/sda4       107G   50G   53G  49% /mnt/data

This will list the devices used by the mounted filesystems. Now you can insert the SD card and run the command again. You will notice that new entries will appear on the output. It will usually be /dev/sdb* entries, if you have only one disk. That is the device file for the card. If the card is automatically mounted, you can see the size on df's output, so you can be sure that it is the right device.

A really important note here: you need to get the correct device name. We are going to use a program that does raw block copying, which can destroy your data if you target the wrong device. The device mounted on root ('/') is probably not the device you want, for example.

Another important thing: if the card already has partitions, some extra device files may appear, such as /dev/sdb1, /dev/sdb2 and so on. We need the device, not any partition, so be sure to use the device file with no number suffix. When you have the device name, make sure the device is unmounted so we can write data to it:

$ umount /dev/sdb*

Then use good old dd to copy the image:

$ dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M

A couple things deserve mention. It is always good to check (and re-check) the parameters, to make sure the image is the if (input file) value and the device is the of (output file) parameter. The bs parameter stands for block size, and is the size of each chunk of data copied at a time. Larger values make the process faster. Usually, 4mb is a reasonable Following UNIX's philosophy, dd doesn't output anything while it is working. Since it can take quite some time to finish, you may think it's froze. But don't worry, it hasn't. If you want to be absolutely sure, send the program a signal and it will output the progress:

$ ps aux | grep dd
  4821 pts/2    R+     0:33 dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M
$ kill -USR1 4821

When it is done, run

$ sync
$

to make sure everything is properly written and remove the card. Now insert the card on the Raspberry, plug the power supply and, if everything worked, you will get a nice show of lights as it boots. You can plug the HDMI to a monitor and see the boot messages. Here's a (somewhat light-impaired) gallery:

I know I promised we'd get it to work! Well, if you (unlike me) think this isn't actual work, check the next posts, where we will put the Pi to good use.

The raspberry

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem

My late Christmas gift (from me to myself) arrived on Monday, and it is a little Raspberry Pi. If you're not familiar with it (and you should be), it's a tiny board which contains everything a computer needs to run (or, as I explained to my perplexed mother, "it has everything a computer need to be a computer").

I won't go over the process of acquiring one, which is not as simple as it may seem, if you live in Brazil. I'll skip to the part just after you console yourself for paying the rough equivalent of 85 dollars, or 242% the original price (of course this includes shipping, but it still is absurd).

So, if you survive that step, you'll get this pretty box:

And inside it...

Prelude

If you're reading this before actually buying it, or before it arrives (and that is a smart thing to do), you'll probably want to get everything prepared so you can start working as soon as you get your hands on it. Here are the things you absolutely need to get started:

• SD card: this is where all the files are stored. If you'll be using Raspbian (and if you don't know what that is, it probably is what you'll be using), the card must be at least 2GB large. The size really depends on what you plan to do with it, but 8 GB and 16 GB suit most of the needs. If you don't want to buy it right away, you can use any SD card you have laying around, like those used in cameras.

• Power supply: this is the most delicate component. If you look at the Quick Start Guide, the Pi requires a "[...] good quality, micro USB power supply that can provide at least 700mA at 5V.". Depending on where you live, this may be easier or harder to find (apparently, I live in the hardest place on Earth). Remember to get the voltage right, but what's important is that 700mA is the minimum required for the Raspberry. If you want to plug other devices that need power supply to it, you're probably going to need more power. I was really lucky that my Motorola cell phone charger meets almost exactly the requirements (check the picture on the right).

• HDMI (or RCA) cable: in order to plug it to a monitor or TV, you'll need a HDMI cable. These can be found on any decent electronics store. Alternatively, you can use the RCA port to connect to older devices.

• USB keyboard: any keyboard with a USB cable will do. A mouse is not strictly necessary, but if you like, you can use any one with a USB cable.

So now we have all the components, we need to prepare our SD card. Check the next post for instructions on that, but meanwhile you can go ahead and download the Raspbian image from the Raspberry Pi foundation, which is the OS we'll use.

The super user is more powerful than god

$ useradd god
$ mkdir a
$ chmod 000 a
$ su god -c ls a
ls: cannot open directory a: Permission denied
$ sudo ls a
$

And it can even tell god what to do:

$ sudo su god -c "sed -i 's/\(-DWINDOWS\|-DMAC\)//' Makefile; make -B world"

Happy winter solstice. 

Hiding text in LaTeX

When writing my dissertation, I like to leave some text that I won't use at
that particular moment but know I will need in the (near) future. To remember
to correct it later, I surrounded it with the (pseudo-randomly chosen) \ldots
command, like this:

This is the actual text.

\dots

This is some text I'll need later.

\ldots

This is more of the actual text.

That solved my problem, I could just grep the files and find the parts that
needed refactoring. The problem is they ended up on the PDF, which is not the
end of the world, but it disrupted the reading and didn't look so good. I
finally decided to google the problem, and found a nice solution, using the if
command:

This is the actual text

\iffalse

This is some text I'll need latex.

\fi

This is more of the actual text.

Now I can still grep for \iffalse, and it doesn't pollute the PDF. And it even
advertises my blog. Yay!

Problem when adding SSH key on github

Recently, I had I problem with github. I couldn't add SSH keys anymore, the web page would just hang and present me a "Connection reset" error after a few (annoying) minutes. I struggled for a week, but have finally found a solution, which is not the best, but at least I can access github from my machine once again.

It all started two weeks ago, when I ran an everyday git push, and got a little surprise. It just sat there forever, and I had to SIGTERM it every time. What I did after was try to reset my key on github. I went to the SSH page and deleted my key, created a new one and tried to add it.

Yes I know it was stupid. The right way to do it would be:

• back up my key
• create a new one
• add it
• make absolutely sure it works
• remove the old one in github and on the machine

But I didn't, for one simple reason: adding keys has been such a natural operation, I didn't even think about it. Replacing a key (which I did a lot of times when I was learning how public-key encryption works) only takes a few seconds, and I never had to worry about it. Until now.

Haystacking

I am no ssh expert. So I turned to github's support. And I have to say, the response was really good, fast and to the point. Sadly, it didn't solve the problem. Here is the response:

Hi Bruno,

What size SSH key are you using? We have a known bug where the page will give a 500 error if the SSH key is too large.
I would recommend using the default size key for now, and hopefully we will have this bug fixed soon.
Sorry for the trouble.
Cheers,
Steve

It turns out the problem wasn't the key size, but you can see Steve gave a really polite response. Anyway, I started trying everything to make it work, but in a week I just couldn't do it. Then, yesterday, I decided to give the developer API a try. I didn't have much hope that it would work, because I had tried a lot of different things and because I thought the problem was exactly there. But I was wrong again.

With just a few commands, I was able to add a ssh key to my account and finally access my repositories. And, in the process, I discovered a handy command line utility, so I guess it wasn't all that bad.

curl (a.k.a. cURL)

curl is a program you can use to fetch web pages just as your browser would do. It is present on most distributions, so you can start right away and run 

$ curl iffalse.blogspot.com

And you'll get the HTML output of the page. You can use that to access the github API. To list all the keys of an account, you'd use

$ curl -u bbguimaraes https://api.github.com/user/keys

Use the -u option to set the username. You will be prompted for the password. The switch accepts the form -u user:password, but that just isn't very nice. You will get back a json output, listing all the keys registered for your account. There are several functions on the API, you can take a look at them on the docs page.

But the one that did it for me was the "create a puclib key". A simple post

$ curl -u bbguimaraes -d '{"title": "[...]", "key": "ssh-rsa AAA[...]"}' https://api.github.com/user/keys

and, after a week of awful manual syncing of code, I was finally able to use my beloved github repositories as I always had. I hope this text can save you some time if you ever happen to have the same problem.