Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Saturday, August 20, 2016

College research projects

A long time ago, when I was in college, I got involved in two research projects.

One of them involved integrating a home-(or, I should say, college-)grown spam filter based on artificial intelligence concepts with the traditional postfix mail delivery system.  That was a lot of fun and would later pave the way for my bachelor thesis, which you can find more about here.

The other one, which is the topic of this post, was a computer graphics research project.  The ultimate goal of this project was to develop a organ-surgery simulation system.  The team used a haptic feedback device (very much like this one:


to simulate liver surgery.  A computer model of the organ was displayed and could be interacted with in 3-D using the device.

My first task when I joined the project was to develop a small part to be later incorporated into the system: the suture string.  This would interact with the organ model and simulate stitching as in a real-life surgery.

To make a long story short, my involvement with the project was terminated earlier than it should (because of external reasons, nothing serious) and the code for the simulation hung around in my hard drive for years.  I always wanted to do something with it, because even at the rough stages it was in, it was already very cool (in my opinion =).

I recently revived it and finally got it to a stage where I feel comfortable sharing it with the world.  To be completely honest, I'm publishing it because I have wasted too much time playing with it.  It's really addictive.  Anyway, the entire code base is under a copy-left license (GPLv2), so you can take a look, experiment, do whatever you want with it:

https://github.com/bbguimaraes/college/tree/master/sms

Finally, here is a short demo:


Tuesday, January 20, 2015

2014 books

I read a lot of books. Last year I had some money laying around and decided to put it to good use. I ended up buying fourteen books from O'Reilly (simply the best publisher in the technology field) on May and finally finished reading them all (exception bellow) on the first days of this year. This a quick and informal review of each of them.

As you'll notice, the scheduling algorithm used was an optimistic, little-endian, depth-first, heuristic search: I always grabbed the book which seemed the smallest from the pile of unread books and read it cover-to-back.




tmux: Productive Mouse-Free Development


This is a really thin book. I struggle to even call it a book. It is meant as a quick introduction to using tmux. I had been using it for a while and the book seemed interesting, so I gave it a shot. It certainly didn't let me down.  It delivers on its promise to give you the essentials to integrate this wonderful tool on your current workflow and it's certainly worth its price (which is as small as the book itself).


The Cathedral & the Bazaar


I don't think this book needs any introduction or even a review at all. If you've ever been in contact with any kind of free software (and you have), read it.


The Developer's Code


This is one of those books about programming that have very little code in it.  I usually don't like them, but this one was certainly worth the read. An amusing look at software development as we all know it, from the perspective of a real developer. I found it interesting that I had already developed many of the techniques presented on the tips, but did so in a very empirical way. It would certainly be nice to have read this book when I was starting my career.


21st Century C


Even though I don't work with it in my current daily job (and haven't worked with it for a long time), c is a language I am very interested in. First of all, we can all argue whether there is a god, but one thing is certain: if there is one and he created the universe, he wrote it in c. Aside from that, it is the basis to pretty much everything related to computers. It always amuses me to see people dog on c without realizing their favorite language's environment is most certainly written in c.

Anyway, this book is a must-read for anyone working with c that hasn't followed recent developments on the language and the tools around it. From the first steps to automating portable compilation with autotools, this book teaches what you need to learn (and specially the parts you should forget) about this venerable language.


Test Driven Development for Embedded C


This is one of the "weirdness-coefficient" books. I work with TDD, but in a language that is very far from c (python) in an area that is very far from embedded systems (web). I always wondered how, and even if, it could be applied to these types of environments, so I decided to get this book. To my surprise, it was one of the best texts I have read about TDD and software testing in general. Not to say the "embedded" part was overlooked. If you work with TDD and specially on embedded systems, I'd say this is an obligatory read.


Practical Vim


I will take two excerpts from Tim Pope's foreword as my review:
Practical Vim tips teach lessons in thinking like a proficient Vim user. In a sense, they are more like parables than recipes.

It is for this reason that I am excited about the publication of Practical Vim. Because now when Vim novices ask me what's the next step, I know what to tell them. After all, Practical Vim even taught me a few things.

Apache Cookbook


Not much to say here. This is a pretty good reference to anyone working with apache. Well worth the read.


The Modern Web


We all have to work on things we hate sometimes, and to me that is when I have to mess with the front-end of web development. But not liking it doesn't mean I can choose not to work with it.

Ranting aside, this is a great book to get up to speed with current and future areas of modern (front-end) web development. It covers everything you need to know about css3, html5 and new javascript API's.


Understanding Computation


This book's synopsis made me add it instantly to the shopping cart and I don't regret it. A great overview of the theoretical aspects of computation. If you liked studying Turing machines in college, this book is for you. Also, a great display of one of the most elegant pieces of code I have ever seen (specially in a book).


Designing for Behavior Change


This was a big surprise. I wanted a book on software architecture and this book's title was very inviting. I saw the five-star reviews and ordered it without even reading the introduction. If I did, I'd seen it is nothing even close to what I was expecting. It even explicitly states that there will be little to no code on the book.

This is a book about software design and (human) behavior analysis. It presents a psychological study on the characteristics of software that can aid in (human) behavior change. I would never buy this book consciously, but I have to say it was a pretty interesting read, if almost completely unrelated to what I work on.


Metasploit: The Penetration Tester's Guide


Software security is one of my favorite areas. This book was recommended by so many people, I had to buy it. It is a great overview of this penetration testing tool. Recommended.


Hacking: The Art of Exploitation


As I said in the previous review, I really like software security, but there were still many "magical" parts I didn't understand well enough. This book tackled them all and much more. I would go as far as to recommend it to anyone that uses a computer, as it explains in detail how a system actually works beneath all the abstractions we use in software development.


Physics for Game Developers


I always loved math and physics since school. In computers, these are mostly used on simulations and I never had the chance to work with anything related, except on some classes in college and on a computer graphics research project.  Nevertheless, I play a lot with these things on my free time.

This book is a fun (if you, like me, think physics is fun) walkthrough of the application of physics in computer simulations (and games, obviously). I really enjoyed it and would recommend it to anyone interested in the area.


The Linux Programming Interface


When I said I read all the books, I obviously lied. As Boromir would gladly tell you, one does not simply read The Linux Programming Interface. First of all, I can't just put in in my backpack and take it out on the bus to read.  Someday I'll definitely read it, but for now it serves both as sporadic reference material and deeply intimidating table ornament.





So there you go. Now I'll be back to "normal" books for a while as my book pile has been growing steadily but surely these seven months. The first one on the list is Jane Austen's Pride and Prejudice. As for technical books, I still plan to finish Structure and Interpretation of Computer Programs, which I was very surprised to find in my college's library. I have been reading and practicing the exercises for almost a year (you can check it on github) and still haven't made it past section 2.3, but it is one of the most interesting programming books I have ever read.

After that, who knows. I may buy some more books. That depends a lot on my country's financial situation, which is nothing but a (downward) roller-coaster at the moment. To list some of the books I'm interested in: Pro Git's new edition, The Art of Memory Forensics (another O'Reilly book) and The Architecture of Open Source Applications.

Wednesday, December 10, 2014

pip --extra-index

This is a tale of debugging. That fine art of digging the darkest corners of a computer system to solve whatever problem is haunting it. This particular story is about python's package manager, pip.

Where I work, we have a server where we store all python packages used in development and production. A package "cache" or "proxy". The idea is similar to a http proxy: we don't have to hit PyPi for each package query and install.  That saves time, as local connections are much faster, both regarding latency and throughput, and bandwidth, as no packet has to leave our lan.

One day, all of a sudden, our testing and production servers started taking a long time to run a simple package update. And when I say "a long time", I mean taking more than ten minutes to run a simple `pip install --update` with roughly fifty packages on a requirements.txt file. That is a ridiculously long time. That would be crazy slow even if we were hitting the wan, but on a lan, that is just absurd. So, clearly, something fishy was going on.


Debugging begins

Step one when debugging an issue: figure out what changed. I thought long and hard (harder than longer, I must admit) about it, but couldn't think of anything I or anyone else had changed on these machines recently. So I advanced to step two: getting a small, reproducible test.

In this case, the test can be reduced to a simple command line execution:

    $ pip install -Ur requirements.txt

This is a standard pip requirements file, with the standard options to prefer our internal server over the official PyPi server:

    $ head -2 requirements.txt
    --index-url=http://pypi.example.com/simple/
    --extra-index-url=http://pypi.python.org/simple/


Here and in the next examples, I'll substitute the real domains for our servers for fake ones. Anyway, running that one simple command was all that was needed to test the strange behavior. On my personal development machine, that took quite a while:

    $ time pip install -Ur requirement.txt &> /dev/null

    real    0m41.280s
    user    0m3.557s
    sys     0m0.100s


Even more interestingly, this was way less time than we were seeing on the servers:

    $ time pip install -Ur requirement.txt &> /dev/null

    real    13m19.752s
    user    0m1.031s
    sys     0m0.184s



On the server

The three main servers affected by this issue (the ones I spend most time in) were our buildbot (i.e. continuous integration), staging and production servers. I decided the first test would be on the buildbot server, as it is the same server where the packages are hosted. That way, I can exclude many external factors that could be affecting the traffic.

So I fired my favorite tool: the amazing strace. If you don't know it, stop everything and go take a look at `strace(1)`. Since I know most people won't, here's a quick introduction: strace execve(2)'s your command, but sets up hooks to display every system call it executes, along with arguments, return values and everything. It is an amazing tool to have an overall idea of what a process is doing. If you are root (and have CAP_SYS_TRACE), you can even attach to a running process, which is an amazing way to debug a process that starts running wild.

Using it to run the command:

    $ strace -r -o strace.txt pip install -Ur requirement.txt

The arguments here are `-o strace.txt` to redirect output to a file and the super useful `-r` to output the relative time between each system call, which is perfect to identify the exact system calls slowing down the execution of the command.

After the execution was done, looking at the output, I found the culprit. Here is a sample of the log:

    $ grep -m 1 -B 5 close strace.txt
     0.000138 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
     0.000115 fcntl(4, F_GETFL)         = 0x2 (flags O_RDWR)
     0.000092 fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
     0.000089 connect(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("23.235.39.223")}, 16) = -1 EINPROGRESS (Operation now in progress)
     0.000128 poll([{fd=4, events=POLLOUT}], 1, 15000) = 0 (Timeout)
    15.014733 close(4)


From this point, my analysis had some flaws that delayed the final conclusion.  I will explain my line of thought as it happened, maybe you'll find the mistakes before I get to them. So, as can be seen in the output, a socket is opened to communicate with another server, which is normal behavior for pip, but then closing it takes around fifteen seconds. Hmm, that is really odd.

So I ran the command again and, while it was blocked waiting, I used another useful command to list the open file descriptors of the process:

    $ lsof -p $(pgrep pip) | tail -2
    pip     19928 django    3u  IPv4 8064331      0t0     TCP
    pypi.example.com:38804->pypi.example.com:http (CLOSE_WAIT)
    pip     19928 django    4u  IPv4 8064359      0t0     TCP
    pypi.example.com:30470->199.27.76.223:http (SYN_SENT)


Here we have two open sockets. One of them is in the CLOSE_WAIT state. Anyone who's ever done socket programming knows this dreaded state, where the local socket is closed but the remote end doesn't send the FIN packet to terminate the connection. A few minutes of tcpdump later, I was convinced that was the problem: something was preventing the connection from ending and each operation was waiting for the timeout to close the socket. That would explain why closing the socket took so long.


The mistakes

At this point, I realised my first mistake. If you take a look at the strace output again, the remote end of the socket is *not* our server. Take a look at the remote address (the `sin_addr` parameter to the `socket` call): 23.235.39.223 is not the ip address of our server, and taking a look at the rest of the output showed that the address changed over time.

There should be no other servers involved, since we explicitly told pip to fetch packages from our own server. So I thought: what other server could be involved here? So I took a guess:

    $ dig pypi.python.org | grep -A 2 'ANSWER SECTION'
    ;; ANSWER SECTION:
    pypi.python.org.        52156   IN      CNAME   python.map.fastly.net.
    python.map.fastly.net.  30      IN      A       23.235.46.223


Damn... Wait!

    $ dig pypi.python.org | grep -A 2 'ANSWER SECTION'
    ;; ANSWER SECTION:
    pypi.python.org.        3575    IN      CNAME   python.map.fastly.net.
    python.map.fastly.net.  7       IN      A       23.235.39.223


Bingo! So it was a connection to one of PyPi's servers. I went back to the strace output and realised my second mistake. If you read strace's man page section for the `-r` option carefully, the delta shown before each line is not the time each syscall took, but the time between that syscall and the last. So the operation that was getting stuck was not `close`, but the previous, `epoll`.

In hindsight, it is obvious. You can see the indication that the call timed out. You can even see the timeout is one of the parameters. And so the mystery was solved. By some unknown reason, pip was trying to make a connection to PyPi after checking our server. Since we don't allow that, the operation hang around until the timeout was reached. One final test confirmed our suspicion:

    $ time pip install -Ur requirement.txt &> /dev/null

    real    0m42.981s
    user    0m3.720s
    sys     0m0.070s
    $ sed -ie 's/^--extra-index/#&/' requirements.txt
    $ time pip install -Ur requirement.txt &> /dev/null

    real    0m1.049s
    user    0m0.853s
    sys     0m0.057s


Removing the extra index option eliminated the issue (and gave us a ~42x speed up, something you don't see everyday).


Conclusion

So, what do we take out of this (unexpectedly long) story? If you are using a local package server, don't use `--extra-index`. I have no idea why pip was trying to contact the extra index after finding the package on our server. The only reason I can think of is it is trying to find a newer version of the package, but even then, most of our requirements are fixed, i.e. they have '==${some_version}' appended.

Even on my development machine, where pip can reach the remote server, it is worth it to remove the option. The time it takes just to reach the server for each package, even just to receive a "package up-to-date" message, slows down the operation considerably:

    $ time pip install -Ur requirement.txt &> /dev/null

    real    0m46.816s
    user    0m3.853s
    sys     0m0.130s
    $ sed -ie 's/^--extra-index/#&/' requirements.txt
    $ time pip install -Ur requirement.txt &> /dev/null

    real    0m1.125s
    user    0m0.947s
    sys     0m0.053s



Coda

Thank you for making it this far. I hope this story was entertaining and hopefully it taught you a thing or two about investigation, debugging and problem solving. Take the time to learn the basic tools: ps, lsof, tcpdump, strace. I assure you they will be really useful when you encounter this type of situation.

Friday, May 2, 2014

Manufacturer's blues

The state of technology sometimes gives me the blues. In 2014, I found myself installing a package called `dosfstools` to format my pendrive. That's because the car's radio wouldn't accept any device with a filesystem that's not fat32:


Of course possessing a filesystem of another kind makes the device "not a device". After a few seconds on the command line the problem went away:
$ sudo pacman -S dosfstools
$ sudo mkfs.vfat -n BILLY /dev/sdb1
but the sadness remained. At l(e)ast I can listen to some badvoltage while driving.

Oh yes, did I mention it doesn't accept any format except mp3? And how does it signal it? By not showing the files at all, of course.

Sunday, July 14, 2013

A post-mortem

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-morten

A lot has happened since my first rπ post. I've used that little thing to death (quite literally), as a ssh box, firewall, general server, web server, web browser, movie box and a whole lot more. Then, I've had the SD card die on me. For me, it was a perfect little linux computer that could stay on all the time, consuming little-to-no power, making absolutely no noise, occupying minimal space. In a few words: really, really awesome. And all that for 35 dollars!

As promised, I'll post here the steps to get from a rπ on a closed box to a little box of awesomeness. I know there a ton of posts like these, but hey, storage is free (for now) on blogger. If you haven't read the first post, this is a good time to do it. I will first make a few post-morten (again, literally) observations about my first assumptions.

SD card: let's start with the most important thing. When I first went shopping for the various attachments, I got the most inexpensive card I found. The only requisite was that it had at least a 4gb storage. After it died, I decided to spend a few extra bucks and get a decent one. Man, I was surprised. There isn't any way of comparison between the π before and now. Instead of moaning the old days, I'll just say this: get a decent, class 10 SD card and you'll have nothing to worry about. And since these things get cheaper and cheaper, get a 16gb one.

Power supply: again, when I started using the π, I just used my cell phone charger, which, by extreme luck, was compatible and provided almost exactly the amount of power needed. Then I started using a wireless adapter sometimes. Then I started using hdmi output sometimes. Then I tried to use both at the same time. And it died. Not that it does any harm to the board, but any moderate load will simply power-off the device. So, if you want to use more than the π itself, get an appropriate power supply. Those on the farnell website worked just fine for me. And just for completeness sake, mine is 1000mA.

HDMI: I did my first tests on my living room TV. But after a while (as addiction got worse), I felt the need to use it on my bedroom, where I have only an old computer monitor with VGA input. So I got a VGA-to-HDMI adapter and things worked fine (well, almost, more on that later).

OS: the first distribution I installed was raspbian, mainly because I was used to debian and, by the time I started playing, there weren't many alternatives. Now we have pidora and even riscos. But anyway, raspbian is a really good distribution and I never had a problem with it. But I am restless. Since I was going to reinstall the OS from /dev/zero, I decided to jump from the cliff and install arch. For no special reason, just to try something else. So, the instructions from now on will be arch centered, although many will be usable on other distributions.

Man, I write a lot. Let's stop right here and I'll get to the first installation steps later.

Sunday, February 24, 2013

Handy bash substitution

Working on the command line is awesome. But there are times where all the typing gets unwieldy. No one likes to type a long file name a lot of times. Sure, using the up arrow and tab keys can make things faster sometimes, but I'll show you some tricks to make your life easier.

Let's start with a really common operation. You download a tarball, extract, cd and have fun. Usually, you'd do this:
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf linux-3.7.9.tar.bz2
$ cd linux-3.7.9
$ bob's your uncle
>
Using the Tab key, you can save a lot of typing:
$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2
$ tar -xjf l<TAB>
$ cd l<TAB>
$ bob's your uncle
>
However, this is highly dependant on the contents of your directory, names in your path and auto-complete configurations. I'm not being picky here. While testing these commands, I had to press Tab twice. First, I had a directory called "libs", so it stopped at "li". Second, I had an episode of the Linux Action Show in a file called linuxactionshowep248.mp4, so it stopped at "linux". And, after you decompress the file, it stops at "linux-3.7.9". Also, if your auto-completion isn't smart enough to filter names that aren't files or directories when you type Tab after cd and tar, you could have many more conflicts.

I don't mean to bash (ha!) on auto-completion, just point you to more efficient alternatives: bash (get it?) has a nice feature called substitutions. You may have come across it before if you ever tried to add an unescaped exclamation mark (henceforth mentioned using the niftier name "bang"):
$ echo "Bang!"
bash: !": event not found
Or got surprised by it:
$ echo "balrog:!you shall not pass:/nonexistent:/bin/false" >> /etc/passwd
echo "balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false" >> /tmp/test

$ sleep 3d; grep balrog /etc/passwd
balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false
That is because the bang is used to reference previous commands. The simplest is substitution of the last command:
$ echo billy
billy
$ !!
echo billy
billy
One thing I should mention: to avoid chaos and destruction (or at least inform you that chaos and destruction have happened) the command with all substitutions made is printed before its output. That is the fourth line in the example, showing that !! was substituted by echo billy.

This substitution is simple and needs no explanation, but it raises a question: what is it useful for? Have you ever typed a (possibly long) command, just to get the following message after running it?
$ cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
cat: /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_: Permission denied
You can solve this by typing <UP><HOME>sudo<SPACE> (if you type <UP><LEFT>{201}sudo<SPACE>, I will personally punch you in the face). But you can save a lot of typing just using
$ sudo !!
sudo cat /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_
[sudo] password for billy:
Here, "!!" got substituted by the last command, as you can see on the echo. Some other uses I found (which you may not understand if you're not or is on the path to enlightenment becoming a bash enthusiast) were:
$ cat my_files
file1
file2
file3
$ cat $(!!)
cat $(cat my_files)
contents_of_file1
contents_of_file2
contents_of_file3
$ perl -e 'bang your head on the keyboard'
# god only knows
$ echo "!!" > file_to_save_command # This could be done by clever use of Ctrl-x Ctrl-e, but sometimes you're in a hurry. That's alright, as long as you do it carefully.
When ! is followed by a number n, it executes the nth command in the history (which you can check with the history command). More useful, though, is that, if n is negative, it will execute the last nth command:
$ echo some
some
$ echo thing
thing
$ !-2
echo some
some
$ !-3 other thing
echo some other thing
some other thing
Without thinking much, you can discover that !! is just an easier way to spell !-1.

Now, these are useful techniques, but we are just getting started! Usually, you don't need the whole command, but just one or a few arguments. Common scenario:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd ef
or how about:
$ mv some_directory a_diferent_name_for_some_directory
$ cd a_diferent_name_for_some_directory
It is very common to address the same file on multiple, successive commands. You can, again, save some typing:
$ ls ef
son daughter dog
$ # oh my, ef is a directory
$ cd !$

cd ef
$ mv some_directory a_diferent_name_for_some_directory
$ cd !$

cd some_directory a_diferent_name_for_some_directory
As you can see, !$ is substituted by the last argument of the last command. If you know your regular expressions, you can make a link with the "end of line" symbol. But there's more. This is just a nice shortcut to the more general form !n:m. This means: from the nth, get the mth argument.
$ tar -f my_tar -cz file1 file2 file3
$ du -sh !:2
4.0K    my_tar
$ du -sh !-2:4-6
4.0K    file1
4.0K    file2
4.0K    file3
What goes before the colon can be any of the previous substitutions: !!, !n or !-n. Note that you can specify a range using :n-m, which will get substituted by the parameters n to m. If you need them in a different order, or some non-contiguous arguments, just use more than one substitution:
$ diff ours theirs
$ # gah
$ diff !:2 !:1
diff theirs ours
Alright, I guess this is enough information for a single post. Have fun playing with The Bash and see you soon for more advanced uses of history substitution.

P.S.: I couldn't leave without some closing notes.

  • I learned everything shown here from scattered information on the internet. I decided to compile them because I never found a document which explained them in detail AND without burying them on other bash-related commands.
  • That said, a good reference (not so good for self-learning, but, if you read the text, you should be okay) is bash's man page. Search for "Event Designators" or "\!\!" (escaping is mandatory):
$ man bash
/Event Designators<CR>
  • While all this may seem hard to use practically,  I recommend you just keep it in a dark corner of your mind. I promise you'll start to find places where you can apply it. They can be used to write commands quickly, especially several one-shot commands in a row. Just don't try forcing the use when it's not necessary. After all, they were created to save time.
  • If you ever got stabbed by not knowing how the bang works (like the examples I showed in the beginning), you can avoid getting stabbed again by escaping the bangs, either with a backslash or single quotes:
$ echo Bang\!
Bang!
$ echo 'balrog:!you shall not pass:/noexistent:/bin/false' >> /etc/passwd
  • More of a side note, but to get that humongous file name, I discovered a new (string of) command(s) to get the longest line from a text.
$ find /sys/ | awk '{print length, $0;}' | sort -nr | head -n 1
  • You can probably do the same thing (although it might be a little slower) using a while loop and another bash trick: while $variable (or ${variable}) will give you the contents of a variable, ${#variable} will give you the length of that content. 
$ find /sys/ | while read i; do echo "${#i} $i" | sort -nr | head -n 1
  • Apparently, Jethro Tull's Too Old to Rock 'n' Roll, Too Old to Die generates pretty long file names (the title song is the second only to a really long cache file name on my file system).

Wednesday, February 6, 2013

An image is worth a thousand megabytes

Now we have all the components we need to run our Raspberry Pi, we can start preparing them to actually work together. The first thing we need to do is get the boot record, partitions and operating system on the SD card, so our little friend can boot and run a true system. Thankfully, some nice folks have already done the hard work, so we can jump many of the complex steps.

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem


I'm going to use the Raspbian distribution, which is based on the famous Debian (GNU/)Linux distribution, and includes many of the standard software packages already prepared to run on the Pi. If you haven't already, grab Raspbian's image, which you can find on Raspberry Pi Foundation's downloads web page.
$ wget http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip
Note that this is the current version as I write this post. Visit the page to get the most recent version. An optional but recommended step after you download the image is to check its SHA-1 checksum. That will ensure nothing has been corrupted during the download, which is always a good thing to know, specially when dealing with sensitive data such as disk images. To do that, you can use the sha1sum program:
$ sha1sum 2012-12-16-wheezy-raspbian.zip
514974a5fcbbbea02151d79a715741c2159d4b0a 2012-12-16-wheezy-raspbian.zip
Compare the output with the hash on the SHA-1 section of the download page. Or, if you have the SHA-1 stored in a file:
$ sha1sum -c 2012-12-16-wheezy-raspbian.zip.sha1 2012-12-16-wheezy-raspbian.zip: OK
If you don't get the appropriate result, just download the file again and check it, until you get it right. Now we are going to copy the image to the SD card. Before you insert the SD card on your computer (you may need an external SD card reader, depending on the ports available on your machine), execute the command
$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sda6  /dev/sda8
/dev/sda1  /dev/sda3  /dev/sda5  /dev/sda7
That will list all the disks available. If you prefer a more friendly output, you can use
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda6        46G   30G   14G  69% /
udev            2.9G  4.0K  2.9G   1% /dev
tmpfs           1.2G  1.1M  1.2G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            2.9G   76K  2.9G   1% /run/shm
/dev/sda4       107G   50G   53G  49% /mnt/data
This will list the devices used by the mounted filesystems. Now you can insert the SD card and run the command again. You will notice that new entries will appear on the output. It will usually be /dev/sdb* entries, if you have only one disk. That is the device file for the card. If the card is automatically mounted, you can see the size on df's output, so you can be sure that it is the right device.

A really important note here: you need to get the correct device name. We are going to use a program that does raw block copying, which can destroy your data if you target the wrong device. The device mounted on root ('/') is probably not the device you want, for example.

Another important thing: if the card already has partitions, some extra device files may appear, such as /dev/sdb1, /dev/sdb2 and so on. We need the device, not any partition, so be sure to use the device file with no number suffix. When you have the device name, make sure the device is unmounted so we can write data to it:
$ umount /dev/sdb*
Then use good old dd to copy the image:
$ dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M
A couple things deserve mention. It is always good to check (and re-check) the parameters, to make sure the image is the if (input file) value and the device is the of (output file) parameter. The bs parameter stands for block size, and is the size of each chunk of data copied at a time. Larger values make the process faster. Usually, 4mb is a reasonable Following UNIX's philosophy, dd doesn't output anything while it is working. Since it can take quite some time to finish, you may think it's froze. But don't worry, it hasn't. If you want to be absolutely sure, send the program a signal and it will output the progress:
$ ps aux | grep dd
  4821 pts/2    R+     0:33 dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M
$ kill -USR1 4821
When it is done, run
$ sync
$
to make sure everything is properly written and remove the card. Now insert the card on the Raspberry, plug the power supply and, if everything worked, you will get a nice show of lights as it boots. You can plug the HDMI to a monitor and see the boot messages. Here's a (somewhat light-impaired) gallery:




I know I promised we'd get it to work! Well, if you (unlike me) think this isn't actual work, check the next posts, where we will put the Pi to good use.

Wednesday, January 23, 2013

The raspberry

Part I - The raspberry
Part II - An image is worth a thousand megabytes
Part III - A post-mortem

My late Christmas gift (from me to myself) arrived on Monday, and it is a little Raspberry Pi. If you're not familiar with it (and you should be), it's a tiny board which contains everything a computer needs to run (or, as I explained to my perplexed mother, "it has everything a computer need to be a computer").

I won't go over the process of acquiring one, which is not as simple as it may seem, if you live in Brazil. I'll skip to the part just after you console yourself for paying the rough equivalent of 85 dollars, or 242% the original price (of course this includes shipping, but it still is absurd).

So, if you survive that step, you'll get this pretty box:

Raspberry Pi in-a-box

And inside it...

The Raspberry Pi unboxed


Prelude

If you're reading this before actually buying it, or before it arrives (and that is a smart thing to do), you'll probably want to get everything prepared so you can start working as soon as you get your hands on it. Here are the things you absolutely need to get started:

  • SD card: this is where all the files are stored. If you'll be using Raspbian (and if you don't know what that is, it probably is what you'll be using), the card must be at least 2GB large. The size really depends on what you plan to do with it, but 8 GB and 16 GB suit most of the needs. If you don't want to buy it right away, you can use any SD card you have laying around, like those used in cameras.
  • Power supply: this is the most delicate component. If you look at the Quick Start Guide, the Pi requires a "[...] good quality, micro USB power supply that can provide at least 700mA at 5V.". Depending on where you live, this may be easier or harder to find (apparently, I live in the hardest place on Earth). Remember to get the voltage right, but what's important is that 700mA is the minimum required for the Raspberry. If you want to plug other devices that need power supply to it, you're probably going to need more power. I was really lucky that my Motorola cell phone charger meets almost exactly the requirements (check the picture on the right).
  • HDMI (or RCA) cable: in order to plug it to a monitor or TV, you'll need a HDMI cable. These can be found on any decent electronics store. Alternatively, you can use the RCA port to connect to older devices.
  • USB keyboard: any keyboard with a USB cable will do. A mouse is not strictly necessary, but if you like, you can use any one with a USB cable.
So now we have all the components, we need to prepare our SD card. Check the next post for instructions on that, but meanwhile you can go ahead and download the Raspbian image from the Raspberry Pi foundation, which is the OS we'll use.

Friday, December 21, 2012

The super user is more powerful than god

$ useradd god
$ mkdir a
$ chmod 000 a
$ su god -c ls a
ls: cannot open directory a: Permission denied
$ sudo ls a
$

And it can even tell god what to do:

$ sudo su god -c "sed -i 's/\(-DWINDOWS\|-DMAC\)//' Makefile; make -B world"

Happy winter solstice.