Sunday, July 14, 2013

A post-mortem

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-morten

A lot has happened since my first rπ post. I've used that little thing to death (quite literally), as a ssh box, firewall, general server, web server, web browser, movie box and a whole lot more. Then, I've had the SD card die on me. For me, it was a perfect little linux computer that could stay on all the time, consuming little-to-no power, making absolutely no noise, occupying minimal space. In a few words: really, really awesome. And all that for 35 dollars!

As promised, I'll post here the steps to get from a rπ on a closed box to a little box of awesomeness. I know there a ton of posts like these, but hey, storage is free (for now) on blogger. If you haven't read the first post, this is a good time to do it. I will first make a few post-morten (again, literally) observations about my first assumptions.

SD card: let's start with the most important thing. When I first went shopping for the various attachments, I got the most inexpensive card I found. The only requisite was that it had at least a 4gb storage. After it died, I decided to spend a few extra bucks and get a decent one. Man, I was surprised. There isn't any way of comparison between the π before and now. Instead of moaning the old days, I'll just say this: get a decent, class 10 SD card and you'll have nothing to worry about. And since these things get cheaper and cheaper, get a 16gb one.

Power supply: again, when I started using the π, I just used my cell phone charger, which, by extreme luck, was compatible and provided almost exactly the amount of power needed. Then I started using a wireless adapter sometimes. Then I started using hdmi output sometimes. Then I tried to use both at the same time. And it died. Not that it does any harm to the board, but any moderate load will simply power-off the device. So, if you want to use more than the π itself, get an appropriate power supply. Those on the farnell website worked just fine for me. And just for completeness sake, mine is 1000mA.

HDMI: I did my first tests on my living room TV. But after a while (as addiction got worse), I felt the need to use it on my bedroom, where I have only an old computer monitor with VGA input. So I got a VGA-to-HDMI adapter and things worked fine (well, almost, more on that later).

OS: the first distribution I installed was raspbian, mainly because I was used to debian and, by the time I started playing, there weren't many alternatives. Now we have pidora and even riscos. But anyway, raspbian is a really good distribution and I never had a problem with it. But I am restless. Since I was going to reinstall the OS from /dev/zero, I decided to jump from the cliff and install arch. For no special reason, just to try something else. So, the instructions from now on will be arch centered, although many will be usable on other distributions.

Man, I write a lot. Let's stop right here and I'll get to the first installation steps later.

Saturday, May 11, 2013

python mocking

If you are into unit testing, you probably have been introduced to mocking. And if that is the case, you probably already have been bitten by it. Mocking requires some understanding of code execution, importing and name resolution that most people lack when first encountering such situations. In python, mocking is a relatively simple process, if you analyze carefully what needs to be done.

Mocking simply means replacing an object with another. This is usually done to avoid instantiating costly systems or to change the behaviour of a system. To begin with a simple example:
def some_function_in_your_code(a):
    if a.do_something():
        return 3.14159
        return 6.26318

def some_other_function_in_your_code():
    # ...
    a = create_complicated_object(*thousand_parameters)
    result = some_other_function_in_your_code(a)
    # ...
Here we have a function some_other_function_in_your_code that uses an object. Somewhere else in your code, some_other_function_in_your_code creates that object, which is a complicated process that involves hundreds of operations. If you just want to test some_other_function_in_your_code, you shouldn't need to go through this whole process[1].

To avoid that, we can use mocking. Notice that all we need to test on some_other_function_in_your_code is that the argument passed has a member called "do_something" that can be called with no arguments and returns something that can be converted to bool. There are many ways to do that, but to keep things short, I'll skip right to the library I use most of the times, mock.

The main component of the mock library is the Mock class, which is basically an empty object with a few useful characteristics (be sure to check the documentation, because they are really useful). Two of them are important for this discussion. First, every time you access an attribute of a Mock object, another Mock object is created and assigned to the attribute being accessed.
>>> import mock
>>> m = mock.Mock()
>>> print(m)
<Mock id='140095453080976'>
>>> print(id(m.some_attribute))
>>> print(m.some_attribute)
<Mock name='mock.some_attribute' id='140095453131664'>
>>> print(id(m.some_attribute))
As you can see, when we tried to access some_attribute, a Mock object was created for us. We could do it by hand with a single line of code, but it makes it makes the code easier, shorter and cleaner.

The other feature of Mock objects is the return_value attribute. Whatever is contained in this attribute gets returned when the object is called as a function object [2].
>>> import mock
>>> m = mock.Mock()
>>> m.return_value = u'the return value'
>>> m()
u'the return value'
Using this two techniques, we can now test our function:
import mock

def your_test():
    a = mock.Mock()
    a.do_something.return_value = True
    if some_function_in_your_code(a) != 3.14159:
        raise Exception(u'return wasn't 3.14159.')
    a.return_value = False
    if some_function_in_your_code(a) != 6.28318:
        raise Exception(u'return wasn't 6.28318.')
Let's break that down. We first create a Mock object to represent the argument passed to the function. The next line takes care of the two things we need to test the function: the do_something attribute and its return value. Then, all we have to do is call the function, passing the mocked argument, and check the return value. After that, repeat the process, this time with a different return value.

That was the easy part

This first section was easy, nothing you couldn't find out with a quick search on the internet. But the real world is not that pretty (at least mine isn't). The trickiest situation is when you have to change the behaviour of something inside a function, but you don't pass that something as an argument. Suppose we have this:
import random

def f():
    if random.random() < 0.5:
        return 3.14159
        return 6.28318
How can you test that function if the value tested is conjured from oblivion in the middle of the function? Fear not, you can actually do it. The trick here is to mock the random function from the random module before the first line of f is executed. Let's start with this simple example, just to get the basic idea:
>>> import random
>>> random.random()
>>> random.random()
>>> import mock
>>> random.random = mock.Mock(return_value=0.5)
>>> random.random()
This seems pretty simple. But here is where everybody gets lost:
>>> import random
>>> import mock
>>> random.random = mock.Mock(return_value=0.5)
>>> import some_file
>>> some_file.f() == 6.28318 or raise ThisShouldNotBeHappeningException()
This may not explode the first time, but you have a 50% chance of getting an exception. To see why, you can put a print random.random inside f and see that the mock didn't work. To understand why, we have to dig a little deeper.


What happens when you run import random? You can watch this video to know exactly what. Or you can just continue reading to get a summary. Or both. Anyway:
>>> locals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', '__doc__': None, '__package__': None}
>>> import random
>>> locals()
{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', 'random': <module 'random' from '/usr/lib/python2.7/random.pyc'>, '__doc__': None, '__package__': None}
In python, locals is a built-in function that returns the local variables accessible on the current state of execution (don't believe me, run print(__builtins__.locals)). When you execute import random, the interpreter does its magic to find the module and load it, but more important, it creates an entry called "random" on the current namespace referring to the module loaded. The critical part here is "current namespace". Try this:
>>> def f(): print(locals())
>>> import random
>>> f()
Here, importing random didn't affect the namespace on f. The same thing applies to namespaces of other modules. Our example fails because the namespace on the some_file module is different than the namespace where we run our tests. To change the namespace of some_file, we have to do it explicitly:
>>> import some_file
>>> some_file.random.random = lambda: 0.5
>>> some_file.f() == 6.28318 or raise ThisShouldNotBeHappeningException()
You can run that many times if you don't trust me, but that will always succeed. And it does because we now are changing the correct namespace. You can check it by putting a print(random.random) on f again.

Being nice

Now you know how to mock, but there is something I must say before you leave. Always, always, ALWAYS restore any mock you do. Seriously. Even if you're sure no one will use the mocked attribute. You don't want to loose an entire day of work just to find out that the problem was an undone mock.

And doing it is so simple: store the original value on a variable and restore it after the operation. I like to do it as soon as the operation is complete, before anything else is executed, but you don't need to, if you're not paranoid. Just to clear any doubt, here is exactly how to do it:
>>> import random
>>> original_random = random.random
>>> random.random = lambda: 0.5
>>> # do something
>>> random.random = original_random
Now you have no excuse. Better yet, you can use another feature of the mock library called patch. But that would be an extension to an already long post. Maybe I'll cover it in the future. Anyway, happy mocking!


1: You shouldn't have complicated processes that involve hundreds of operations anyway, but that is another problem.

2: Curious to know what happens when you access return_value without setting it first? No? Well, I'll show you anyway:
>>> import mock
>>> m = mock.Mock()
>>> m.return_value
<Mock name='mock()' id='140095453168080'>
Since we didn't set it, we get the default behaviour of __setattr__, which is to create another Mock.

Sunday, February 24, 2013

Handy bash substitution

Working on the command line is awesome. But there are times where all the typing gets unwieldy. No one likes to type a long file name a lot of times. Sure, using the up arrow and tab keys can make things faster sometimes, but I'll show you some tricks to make your life easier.

Let's start with a really common operation. You download a tarball, extract, cd and have fun. Usually, you'd do this:
$ wget
$ tar -xjf linux-3.7.9.tar.bz2
$ cd linux-3.7.9
$ bob's your uncle
Using the Tab key, you can save a lot of typing:
$ wget
$ tar -xjf l<TAB>
$ cd l<TAB>
$ bob's your uncle
However, this is highly dependant on the contents of your directory, names in your path and auto-complete configurations. I'm not being picky here. While testing these commands, I had to press Tab twice. First, I had a directory called "libs", so it stopped at "li". Second, I had an episode of the Linux Action Show in a file called linuxactionshowep248.mp4, so it stopped at "linux". And, after you decompress the file, it stops at "linux-3.7.9". Also, if your auto-completion isn't smart enough to filter names that aren't files or directories when you type Tab after cd and tar, you could have many more conflicts.

I don't mean to bash (ha!) on auto-completion, just point you to more efficient alternatives: bash (get it?) has a nice feature called substitutions. You may have come across it before if you ever tried to add an unescaped exclamation mark (henceforth mentioned using the niftier name "bang"):
$ echo "Bang!"
bash: !": event not found
Or got surprised by it:
$ echo "balrog:!you shall not pass:/nonexistent:/bin/false" >> /etc/passwd
echo "balrog:youtube-dl shall not pass:/nonexistent:/bin/false" >> /tmp/test

$ sleep 3d; grep balrog /etc/passwd
balrog:youtube-dl shall not pass:/nonexistent:/bin/false
That is because the bang is used to reference previous commands. The simplest is substitution of the last command:
$ echo billy
$ !!
echo billy
One thing I should mention: to avoid chaos and destruction (or at least inform you that chaos and destruction have happened) the command with all substitutions made is printed before its output. That is the fourth line in the example, showing that !! was substituted by echo billy.

This substitution is simple and needs no explanation, but it raises a question: what is it useful for? Have you ever typed a (possibly long) command, just to get the following message after running it?
cat: /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_: Permission denied
You can solve this by typing <UP><HOME>sudo<SPACE> (if you type <UP><LEFT>{201}sudo<SPACE>, I will personally punch you in the face). But you can save a lot of typing just using
$ sudo !!
sudo cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
[sudo] password for billy:
Here, "!!" got substituted by the last command, as you can see on the echo. Some other uses I found (which you may not understand if you're not or is on the path to enlightenment becoming a bash enthusiast) were:
$ cat my_files
$ cat $(!!)
cat $(cat my_files)
$ perl -e 'bang your head on the keyboard'
# god only knows
$ echo "!!" > file_to_save_command # This could be done by clever use of Ctrl-x Ctrl-e, but sometimes you're in a hurry. That's alright, as long as you do it carefully.
When ! is followed by a number n, it executes the nth command in the history (which you can check with the history command). More useful, though, is that, if n is negative, it will execute the last nth command:
$ echo some
$ echo thing
$ !-2
echo some
$ !-3 other thing
echo some other thing
some other thing
Without thinking much, you can discover that !! is just an easier way to spell !-1.

Now, these are useful techniques, but we are just getting started! Usually, you don't need the whole command, but just one or a few arguments. Common scenario:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd ef
or how about:
$ mv some_directory a_diferent_name_for_some_directory
$ cd a_diferent_name_for_some_directory
It is very common to address the same file on multiple, successive commands. You can, again, save some typing:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd !$

cd ef
$ mv some_directory a_diferent_name_for_some_directory
$ cd !$

cd some_directory a_diferent_name_for_some_directory
As you can see, !$ is substituted by the last argument of the last command. If you know your regular expressions, you can make a link with the "end of line" symbol. But there's more. This is just a nice shortcut to the more general form !n:m. This means: from the nth, get the mth argument.
$ tar -f my_tar -cz file1 file2 file3
$ du -sh !:2
4.0K    my_tar
$ du -sh !-2:4-6
4.0K    file1
4.0K    file2
4.0K    file3
What goes before the colon can be any of the previous substitutions: !!, !n or !-n. Note that you can specify a range using :n-m, which will get substituted by the parameters n to m. If you need them in a different order, or some non-contiguous arguments, just use more than one substitution:
$ diff ours theirs
$ # gah
$ diff !:2 !:1
diff theirs ours
Alright, I guess this is enough information for a single post. Have fun playing with The Bash and see you soon for more advanced uses of history substitution.

P.S.: I couldn't leave without some closing notes.

  • I learned everything shown here from scattered information on the internet. I decided to compile them because I never found a document which explained them in detail AND without burying them on other bash-related commands.
  • That said, a good reference (not so good for self-learning, but, if you read the text, you should be okay) is bash's man page. Search for "Event Designators" or "\!\!" (escaping is mandatory):
$ man bash
/Event Designators<CR>
  • While all this may seem hard to use practically,  I recommend you just keep it in a dark corner of your mind. I promise you'll start to find places where you can apply it. They can be used to write commands quickly, especially several one-shot commands in a row. Just don't try forcing the use when it's not necessary. After all, they were created to save time.
  • If you ever got stabbed by not knowing how the bang works (like the examples I showed in the beginning), you can avoid getting stabbed again by escaping the bangs, either with a backslash or single quotes:
$ echo Bang\!
$ echo 'balrog:!you shall not pass:/noexistent:/bin/false' >> /etc/passwd
  • More of a side note, but to get that humongous file name, I discovered a new (string of) command(s) to get the longest line from a text.
$ find /sys/ | awk '{print length, $0;}' | sort -nr | head -n 1
  • You can probably do the same thing (although it might be a little slower) using a while loop and another bash trick: while $variable (or ${variable}) will give you the contents of a variable, ${#variable} will give you the length of that content. 
$ find /sys/ | while read i; do echo "${#i} $i" | sort -nr | head -n 1
  • Apparently, Jethro Tull's Too Old to Rock 'n' Roll, Too Old to Die generates pretty long file names (the title song is the second only to a really long cache file name on my file system).

Wednesday, February 6, 2013

An image is worth a thousand megabytes

Now we have all the components we need to run our Raspberry Pi, we can start preparing them to actually work together. The first thing we need to do is get the boot record, partitions and operating system on the SD card, so our little friend can boot and run a true system. Thankfully, some nice folks have already done the hard work, so we can jump many of the complex steps.

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem

I'm going to use the Raspbian distribution, which is based on the famous Debian (GNU/)Linux distribution, and includes many of the standard software packages already prepared to run on the Pi. If you haven't already, grab Raspbian's image, which you can find on Raspberry Pi Foundation's downloads web page.
$ wget
Note that this is the current version as I write this post. Visit the page to get the most recent version. An optional but recommended step after you download the image is to check its SHA-1 checksum. That will ensure nothing has been corrupted during the download, which is always a good thing to know, specially when dealing with sensitive data such as disk images. To do that, you can use the sha1sum program:
$ sha1sum
Compare the output with the hash on the SHA-1 section of the download page. Or, if you have the SHA-1 stored in a file:
$ sha1sum -c OK
If you don't get the appropriate result, just download the file again and check it, until you get it right. Now we are going to copy the image to the SD card. Before you insert the SD card on your computer (you may need an external SD card reader, depending on the ports available on your machine), execute the command
$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sda6  /dev/sda8
/dev/sda1  /dev/sda3  /dev/sda5  /dev/sda7
That will list all the disks available. If you prefer a more friendly output, you can use
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6        46G   30G   14G  69% /
udev            2.9G  4.0K  2.9G   1% /dev
tmpfs           1.2G  1.1M  1.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.9G   76K  2.9G   1% /run/shm
/dev/sda4       107G   50G   53G  49% /mnt/data
This will list the devices used by the mounted filesystems. Now you can insert the SD card and run the command again. You will notice that new entries will appear on the output. It will usually be /dev/sdb* entries, if you have only one disk. That is the device file for the card. If the card is automatically mounted, you can see the size on df's output, so you can be sure that it is the right device.

A really important note here: you need to get the correct device name. We are going to use a program that does raw block copying, which can destroy your data if you target the wrong device. The device mounted on root ('/') is probably not the device you want, for example.

Another important thing: if the card already has partitions, some extra device files may appear, such as /dev/sdb1, /dev/sdb2 and so on. We need the device, not any partition, so be sure to use the device file with no number suffix. When you have the device name, make sure the device is unmounted so we can write data to it:
$ umount /dev/sdb*
Then use good old dd to copy the image:
$ dd if=./ of=/dev/sdb bs=4M
A couple things deserve mention. It is always good to check (and re-check) the parameters, to make sure the image is the if (input file) value and the device is the of (output file) parameter. The bs parameter stands for block size, and is the size of each chunk of data copied at a time. Larger values make the process faster. Usually, 4mb is a reasonable Following UNIX's philosophy, dd doesn't output anything while it is working. Since it can take quite some time to finish, you may think it's froze. But don't worry, it hasn't. If you want to be absolutely sure, send the program a signal and it will output the progress:
$ ps aux | grep dd
  4821 pts/2    R+     0:33 dd if=./ of=/dev/sdb bs=4M
$ kill -USR1 4821
When it is done, run
$ sync
to make sure everything is properly written and remove the card. Now insert the card on the Raspberry, plug the power supply and, if everything worked, you will get a nice show of lights as it boots. You can plug the HDMI to a monitor and see the boot messages. Here's a (somewhat light-impaired) gallery:

I know I promised we'd get it to work! Well, if you (unlike me) think this isn't actual work, check the next posts, where we will put the Pi to good use.

Wednesday, January 23, 2013

The raspberry

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem

My late Christmas gift (from me to myself) arrived on Monday, and it is a little Raspberry Pi. If you're not familiar with it (and you should be), it's a tiny board which contains everything a computer needs to run (or, as I explained to my perplexed mother, "it has everything a computer need to be a computer").

I won't go over the process of acquiring one, which is not as simple as it may seem, if you live in Brazil. I'll skip to the part just after you console yourself for paying the rough equivalent of 85 dollars, or 242% the original price (of course this includes shipping, but it still is absurd).

So, if you survive that step, you'll get this pretty box:

Raspberry Pi in-a-box

And inside it...

The Raspberry Pi unboxed


If you're reading this before actually buying it, or before it arrives (and that is a smart thing to do), you'll probably want to get everything prepared so you can start working as soon as you get your hands on it. Here are the things you absolutely need to get started:

  • SD card: this is where all the files are stored. If you'll be using Raspbian (and if you don't know what that is, it probably is what you'll be using), the card must be at least 2GB large. The size really depends on what you plan to do with it, but 8 GB and 16 GB suit most of the needs. If you don't want to buy it right away, you can use any SD card you have laying around, like those used in cameras.
  • Power supply: this is the most delicate component. If you look at the Quick Start Guide, the Pi requires a "[...] good quality, micro USB power supply that can provide at least 700mA at 5V.". Depending on where you live, this may be easier or harder to find (apparently, I live in the hardest place on Earth). Remember to get the voltage right, but what's important is that 700mA is the minimum required for the Raspberry. If you want to plug other devices that need power supply to it, you're probably going to need more power. I was really lucky that my Motorola cell phone charger meets almost exactly the requirements (check the picture on the right).
  • HDMI (or RCA) cable: in order to plug it to a monitor or TV, you'll need a HDMI cable. These can be found on any decent electronics store. Alternatively, you can use the RCA port to connect to older devices.
  • USB keyboard: any keyboard with a USB cable will do. A mouse is not strictly necessary, but if you like, you can use any one with a USB cable.
So now we have all the components, we need to prepare our SD card. Check the next post for instructions on that, but meanwhile you can go ahead and download the Raspbian image from the Raspberry Pi foundation, which is the OS we'll use.