tag:blogger.com,1999:blog-44924424206417226742024-03-14T07:40:55.769-03:00iffalseif(false) { assert("this should not happen"); }Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-4492442420641722674.post-60830542083340782712019-07-21T18:39:00.000-03:002019-07-21T18:39:53.370-03:00mv ~Both from personal experience and from having read all sorts of blogs, I
learned that the secret to not having to give excuses for updating a blog in a
very infrequent and irregular fashion is to completely ignore the fact that it
has been more than a year since the last post (I would say this is an
autological self-reference, but that would be paradoxically contradictory).
<br />
<br />
The big news this time is that from now on, the new home for this blog is:
<br />
<blockquote>
<a href="https://bbguimaraes.com/blog">
https://bbguimaraes.com/blog
</a>
</blockquote>
Posts will be published there exclusively, although the old website will remain
where it is (until Google decides to kill Blogspot; given its record, that
should be any day now). If you follow it using RSS, please update your client
to use the <a href="https://bbguimaraes.com/blog/rss.xml">new URL</a>.<br />
<br />
There you will find all previous posts, converted from an XML export after a
day of furious text editing (made exponentially easier by <a href="https://www.vim.org/"> the best text editor in the world</a>). I resisted
as much as I could the urge to edit the posts and instead made only minimal
spelling corrections.<br />
<br />
You will also find <a href="https://bbguimaraes.com/">there</a> all of my other
insane endeavours. For example: last year, I added <a href="https://bbguimaraes.com/random/the_jack">another blog</a> I used to have
where my teenage self reviewed — and shared in a not-quite-legal manner — music
albums. I wasted the chance to make my yearly blog post, so consider this also
an announcement for it.<br />
<br />
And did you know all source code is available under a free software license?
Yes! Run and
<a href="https://gitlab.bbguimaraes.com/bbguimaraes/bbguimaraes.com">get it</a>
before more people find out about it.<br />
<br />
<br />
<h2>
A dark interlude</h2>
<br />
<h3>
1984 was not supposed to be a user manual</h3>
<br />
So why did I decide to make this change?<br />
<br />
It is no secret that I have, for a long time now, tried to stay away as much as
possible from the so-called <i>technological
<a href="https://en.wikipedia.org/wiki/Goliath">giants</a></i>. The reasons
were always difficult to explain to people outside of the IT industry, for whom
concepts such as free (i.e. <i>libre</i>) software, or even <i>software</i> in
any but the most superficial levels, are difficult to grasp.<br />
<br />
Thankfully, over the past years, Google, Facebook, and friends have made that
much easier. Ever since the Snowden leaks, we have been shown time and time
again how our dystopian fears were, if anything, an underestimation. And
things have taken a darker turn recently with these companies taking
increasingly political and aggressive actions in our time of political and
moral confusion.<br />
<br />
<br />
<h2>
Fighting back</h2>
<br />
To counteract that, I have always opted to use open and free technologies.
Some examples:
<br />
<ul>
<li>
<a href="https://www.kernel.org/">Linux</a> as the operating system in all my
computers.
</li>
<li>
<a href="https://lineageos.org/">LineageOS</a> on my mobile devices.
</li>
<li>
<a href="https://mozilla.org/">Firefox</a> as my web browser.
</li>
<li>
<a href="https://matrix.org/">Matrix</a> for instant messaging and video calls.
</li>
<li>
<a href="https://gitlab.com/">Gitlab</a> for source code hosting.
</li>
<li>
<a href="https://nextcloud.org/">Nextcloud</a> for calendar, contacts, file
sharing, etc.
</li>
</ul>
I choose to use free or open-source software first and resort to closed-source
only if it cannot in way be avoided (and, in my experience, it can in virtually
all cases).<br />
<br />
And this brings us back to my blog now being under my personal website and not
hosted in a service owned by Google anymore. It is now just a bunch of static
HTML files served by my webserver. Youtube embedded videos have also been
converted to simple links.<br />
<br />
<br />
<h2>
Future</h2>
<br />
Having finally moved away from Blogspot, there are now only two more items on
my list of SaaS(S)es to get rid of.<br />
<br />
The first is Gmail, which I have used almost since I started using e-mail.
Hosting one's own e-mail server is not easy, so I do not expect this to change
anytime soon. And, with the unfortunate demise of e-mail after social media
started dominating the internet, giving Google access to my e-mail bothers me
less and less.<br />
<br />
The final one is the more complicated and the more mysterious: Youtube. From
talks to musical or otherwise instructional videos to podcasts, I consume
videos at an almost fanatical and certainly unhealthy rate, and a big portion
of them are on Google's platform. I don't see it loosing its dominance in the
near future (although <a href="https://joinpeertube.org/">PeerTube</a> in on my
list of things to try), but hopefully its recent censorship scandals will
accelerate that.<br />
<br />
And finally: are you really a blogger if you don't say "I want to start posting
here more frequently"? It is in no way an excuse, but the fact that this blog
was hosted on Blogspot did have an effect on my willingness to write. That
meant I missed opportunities to talk about some of the interesting things I
have been working on, and I want to correct that.<br />
<br />
As always, thanks for reading.
Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-3586641967458124332017-03-12T18:12:00.000-03:002017-03-12T19:26:54.300-03:00~/musicFor too long my musical (if you can call them that) experiments have been scattered all over the internet. Not anymore! For those who, for whatever reason, find themselves on my website, there's now a <span style="font-family: "courier new" , "courier" , monospace;"><a href="https://bbguimaraes.com/music/" target="_blank">music</a></span> section.<br />
<br />
There, one can find a list of all my performances (yes, even the bad ones), with videos and scores. The scores are as true to the original as they can be, and come in pdf and "source code" (<a href="https://musescore.org/" target="_blank">musescore</a>: if you don't know about it, you should) form.<br />
<br />
I have some other plans for that section (and others), but this is what I was able to put together this weekend. There may be more updates in the future.<br />
<br />
Enjoy.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-44959921383740731942016-08-20T17:47:00.000-03:002016-08-20T17:48:57.762-03:00College research projectsA long time ago, when I was in college, I got involved in two research projects.<br />
<br />
One of them involved integrating a home-(or, I should say, college-)grown spam filter based on artificial intelligence concepts with the traditional <span style="font-family: "courier new" , "courier" , monospace;">postfix</span> mail delivery system. That was a lot of fun and would later pave the way for my bachelor thesis, which you can find more about <a href="https://github.com/bbguimaraes/tcc.git" target="_blank">here</a>.<br />
<br />
The other one, which is the topic of this post, was a computer graphics research project. The ultimate goal of this project was to develop a organ-surgery simulation system. The team used a haptic feedback device (very much like this one:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuQ-kejFB3cx-GIzvHAz74_pjY9anIZjtwHGAurnjuR4Ybj0hPLo3ciEXLXxqPomiDWDCbL-kua9auV9kq5JodCpmpdplMazy33X0ZSk-WheNGoobsDRNgGCnwWODVFrfTsAUNgGVTE5o/s1600/LargePHANTOMOmniImage.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuQ-kejFB3cx-GIzvHAz74_pjY9anIZjtwHGAurnjuR4Ybj0hPLo3ciEXLXxqPomiDWDCbL-kua9auV9kq5JodCpmpdplMazy33X0ZSk-WheNGoobsDRNgGCnwWODVFrfTsAUNgGVTE5o/s320/LargePHANTOMOmniImage.jpg" width="320" /></a></div>
<br />
to simulate liver surgery. A computer model of the organ was displayed and could be interacted with in 3-D using the device.<br />
<br />
My first task when I joined the project was to develop a small part to be later incorporated into the system: the suture string. This would interact with the organ model and simulate stitching as in a real-life surgery.<br />
<br />
To make a long story short, my involvement with the project was terminated earlier than it should (because of external reasons, nothing serious) and the code for the simulation hung around in my hard drive for years. I always wanted to do something with it, because even at the rough stages it was in, it was already very cool (in my opinion =).<br />
<br />
I recently revived it and finally got it to a stage where I feel comfortable sharing it with the world. To be completely honest, I'm publishing it because I have wasted too much time playing with it. It's really addictive. Anyway, the entire code base is under a copy-left license (GPLv2), so you can take a look, experiment, do whatever you want with it:<br />
<br />
<a href="https://github.com/bbguimaraes/college/tree/master/sms">https://github.com/bbguimaraes/college/tree/master/sms</a><br />
<br />
Finally, here is a short demo:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/WB3s7ypA7WQ/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/WB3s7ypA7WQ?feature=player_embedded" width="320"></iframe></div>
<br />Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-85011487711298606712015-01-20T14:42:00.001-02:002016-04-07T04:32:33.343-03:002014 booksI read a lot of books. Last year I had some money laying around and decided to put it to good use. I ended up buying fourteen books from O'Reilly (simply the best publisher in the technology field) on May and finally finished reading them all (exception bellow) on the first days of this year. This a quick and informal review of each of them.<br />
<br />
As you'll notice, the scheduling algorithm used was an optimistic, little-endian, depth-first, heuristic search: I always grabbed the book which seemed the smallest from the pile of unread books and read it cover-to-back.<br />
<span id="goog_1097017255"></span><span id="goog_1097017256"></span><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnoCHyE9D7aK-Rn0NscppyRl63fFEWwHs88VGVD05u_R0xLEwpytscN-VFf5F-3FRN5ucV6xUX2BmXZLGbp1kcPwH9jDQxFf7FVJpWGbQhQz5VIsh0dqb6to4GGEOzY47jbD589NNY-UQ/s1600/2014-05-27_22-47-14_798.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><br /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD7Uz-8xqVfsHTru4DwkYNkdWtiHtEEr_Tv3NsTysE7by_anCbXkFIrEPDMdXglsEDtz-Lgj7r8pOvhHjRSUzHWtcMS-7234ADxiRKP3pzH_Ot9fYvbMX_C0Domaw8ujeH724kI4hjDsA/s1600/2014-05-27_22-38-21_61.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD7Uz-8xqVfsHTru4DwkYNkdWtiHtEEr_Tv3NsTysE7by_anCbXkFIrEPDMdXglsEDtz-Lgj7r8pOvhHjRSUzHWtcMS-7234ADxiRKP3pzH_Ot9fYvbMX_C0Domaw8ujeH724kI4hjDsA/s1600/2014-05-27_22-38-21_61.jpg" width="179" /></a></div>
<span id="goog_1097017255"></span><span id="goog_1097017256"></span><br />
<br />
<h4>
tmux: Productive Mouse-Free Development</h4>
<br />
This is a really thin book. I struggle to even call it a book. It is meant as a quick introduction to using tmux. I had been using it for a while and the book seemed interesting, so I gave it a shot. It certainly didn't let me down. It delivers on its promise to give you the essentials to integrate this wonderful tool on your current workflow and it's certainly worth its price (which is as small as the book itself).<br />
<br />
<br />
<h4>
The Cathedral & the Bazaar</h4>
<br />
I don't think this book needs any introduction or even a review at all. If you've ever been in contact with any kind of free software (and you have), read it.<br />
<br />
<br />
<h4>
The Developer's Code</h4>
<br />
This is one of those books about programming that have very little code in it. I usually don't like them, but this one was certainly worth the read. An amusing look at software development as we all know it, from the perspective of a real developer. I found it interesting that I had already developed many of the techniques presented on the tips, but did so in a very empirical way. It would certainly be nice to have read this book when I was starting my career.<br />
<br />
<br />
<h4>
21st Century C</h4>
<br />
Even though I don't work with it in my current daily job (and haven't worked with it for a long time), c is a language I am very interested in. First of all, we can all argue whether there is a god, but one thing is certain: if there is one and he created the universe, he wrote it in c. Aside from that, it is the basis to pretty much everything related to computers. It always amuses me to see people dog on c without realizing their favorite language's environment is most certainly written in c.<br />
<br />
Anyway, this book is a must-read for anyone working with c that hasn't followed recent developments on the language and the tools around it. From the first steps to automating portable compilation with autotools, this book teaches what you need to learn (and specially the parts you should forget) about this venerable language.<br />
<br />
<br />
<h4>
Test Driven Development for Embedded C</h4>
<br />
This is one of the "weirdness-coefficient" books. I work with TDD, but in a language that is very far from c (python) in an area that is very far from embedded systems (web). I always wondered how, and even if, it could be applied to these types of environments, so I decided to get this book. To my surprise, it was one of the best texts I have read about TDD and software testing in general. Not to say the "embedded" part was overlooked. If you work with TDD and specially on embedded systems, I'd say this is an obligatory read.<br />
<br />
<br />
<h4>
Practical Vim</h4>
<br />
I will take two excerpts from Tim Pope's foreword as my review:<br />
<blockquote class="tr_bq">
<i>Practical Vim tips teach lessons in thinking like a proficient Vim user. In a sense, they are more like parables than recipes.<br /><br />It is for this reason that I am excited about the publication of Practical Vim. Because now when Vim novices ask me what's the next step, I know what to tell them. After all, Practical Vim even taught me a few things.</i></blockquote>
<br />
<h4>
Apache Cookbook</h4>
<br />
Not much to say here. This is a pretty good reference to anyone working with apache. Well worth the read.<br />
<br />
<br />
<h4>
The Modern Web</h4>
<br />
We all have to work on things we hate sometimes, and to me that is when I have to mess with the front-end of web development. But not liking it doesn't mean I can choose not to work with it.<br />
<br />
Ranting aside, this is a great book to get up to speed with current and future areas of modern (front-end) web development. It covers everything you need to know about css3, html5 and new javascript API's.<br />
<br />
<br />
<h4>
Understanding Computation</h4>
<br />
This book's synopsis made me add it instantly to the shopping cart and I don't regret it. A great overview of the theoretical aspects of computation. If you liked studying Turing machines in college, this book is for you. Also, a great display of one of the most elegant pieces of code I have ever seen (specially in a book).<br />
<br />
<br />
<h4>
Designing for Behavior Change</h4>
<br />
This was a big surprise. I wanted a book on software architecture and this book's title was very inviting. I saw the five-star reviews and ordered it without even reading the introduction. If I did, I'd seen it is nothing even close to what I was expecting. It even explicitly states that there will be little to no code on the book.<br />
<br />
This is a book about software design and (human) behavior analysis. It presents a psychological study on the characteristics of software that can aid in (human) behavior change. I would never buy this book consciously, but I have to say it was a pretty interesting read, if almost completely unrelated to what I work on.<br />
<br />
<br />
<h4>
Metasploit: The Penetration Tester's Guide</h4>
<br />
Software security is one of my favorite areas. This book was recommended by so many people, I had to buy it. It is a great overview of this penetration testing tool. Recommended.<br />
<br />
<br />
<h4>
Hacking: The Art of Exploitation</h4>
<br />
As I said in the previous review, I really like software security, but there were still many "magical" parts I didn't understand well enough. This book tackled them all and much more. I would go as far as to recommend it to anyone that uses a computer, as it explains in detail how a system actually works beneath all the abstractions we use in software development.<br />
<br />
<br />
<h4>
Physics for Game Developers</h4>
<br />
I always loved math and physics since school. In computers, these are mostly used on simulations and I never had the chance to work with anything related, except on some classes in college and on a computer graphics research project. Nevertheless, I play a lot with these things on my free time.<br />
<br />
This book is a fun (if you, like me, think physics is fun) walkthrough of the application of physics in computer simulations (and games, obviously). I really enjoyed it and would recommend it to anyone interested in the area.<br />
<br />
<br />
<h4>
The Linux Programming Interface</h4>
<br />
When I said I read all the books, I obviously lied. As Boromir would gladly tell you, one does not simply read The Linux Programming Interface. First of all, I can't just put in in my backpack and take it out on the bus to read. Someday I'll definitely read it, but for now it serves both as sporadic reference material and deeply intimidating table ornament.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnoCHyE9D7aK-Rn0NscppyRl63fFEWwHs88VGVD05u_R0xLEwpytscN-VFf5F-3FRN5ucV6xUX2BmXZLGbp1kcPwH9jDQxFf7FVJpWGbQhQz5VIsh0dqb6to4GGEOzY47jbD589NNY-UQ/s1600/2014-05-27_22-47-14_798.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="179" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnoCHyE9D7aK-Rn0NscppyRl63fFEWwHs88VGVD05u_R0xLEwpytscN-VFf5F-3FRN5ucV6xUX2BmXZLGbp1kcPwH9jDQxFf7FVJpWGbQhQz5VIsh0dqb6to4GGEOzY47jbD589NNY-UQ/s1600/2014-05-27_22-47-14_798.jpg" width="320" /></a></div>
<br />
<br />
So there you go. Now I'll be back to "normal" books for a while as my book pile has been growing steadily but surely these seven months. The first one on the list is Jane Austen's Pride and Prejudice. As for technical books, I still plan to finish Structure and Interpretation of Computer Programs, which I was very surprised to find in my college's library. I have been reading and practicing the exercises for almost a year (you can check it on github) and still haven't made it past section 2.3, but it is one of the most interesting programming books I have ever read.<br />
<br />
After that, who knows. I may buy some more books. That depends a lot on my country's financial situation, which is nothing but a (downward) roller-coaster at the moment. To list some of the books I'm interested in: Pro Git's new edition, The Art of Memory Forensics (another O'Reilly book) and The Architecture of Open Source Applications.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-82157513755832731332014-12-11T20:51:00.000-02:002014-12-12T08:28:58.185-02:00Calculating π the unix wayI nice trick I learned a while ago that is worth sharing: calculating pi the unix way (you know, on the command line, with pipes, as god intended).<br />
<br />
I would like to give credit to the original source of this command, but I just couldn't find it. It was some of those "shell one-liners" you see on hacker news five times a day, except I didn't know half of them. The most interesting was a semi-cryptic command line with a pretentious comment besides it:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> # calculates pi the unix way</span><br />
<br />
I remember as if it was today how puzzled I was by that line. As I said, I didn't know much of the incantations on that list, but this was by far the most magical. The line goes like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '4/%g' 1 2 9999 | paste -sd-+ | bc -l</span><br />
<br />
If you like a challenge (as I do), try to figure it out by yourself. A shell and the man pages are your best friends.<br />
<br />
<br />
<b>seq</b><br />
<br />
If shell (or python) wasn't your first programming language, you were probably surprised by the way loops are done. It usually goes like this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ for x in 1 2 3 4 5; do echo "$x"; done</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 1</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 2</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 3</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 4</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 5</span><br />
<br />
If you have a little experience with shell, you probably learned there is a more idiomatic way of doing this using the seq command and some shell voodoo:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ for x in $(seq 1 5); do echo "$x"; done</span><br />
<br />
And if you were truly initiated on the dark arts of bash programming, you probably know this is functionally equivalent to this:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ for x in {1..5}; do echo "$x"; done</span><br />
<br />
I won't explain how shell command substitution works, suffice to say seq is a nice utility to generate sequences (get it?) of numbers. From the first lines of the man page:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ man seq | grep -A 3 SYNOPSIS</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> SYNOPSIS</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> seq [OPTION]... LAST</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> seq [OPTION]... FIRST LAST</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> seq [OPTION]... FIRST INCREMENT LAST</span><br />
<br />
So the main part of the first command on the pipe is no magic: we are generating numbers from 1 to 9999 with a step of 2:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ echo $(seq 1 2 9999 | head -5)</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 1 3 5 7 9</span><br />
<br />
There is a useful option to this command to control how the value is output:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '%02g' 1 3 10<br /> 01<br /> 04<br /> 07<br /> 10</span><br />
<br />
Programmers familiar with c will recognize the printf format string. Moving down the pipe...<b> </b><br />
<br />
<br />
<b>paste </b><br />
<br />
There are some commands that do something so simple they seem almost useless:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ whatis -l paste<br /> paste (1) - merge lines of files<br /> paste (1p) - merge corresponding or subsequent lines of files</span><br />
<br />
Nothing really interesting here, right?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ paste <(seq 1 3) <(seq 4 6)<br /> 1 4<br /> 2 5<br /> 3 6</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq 1 6 | paste - -<br /> 1 2<br /> 3 4<br /> 5 6</span><br />
<br />
Well, that is interesting. What if we play with the other options?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ paste -sd, <(seq 1 3) <(seq 4 6)<br /> 1,2,3<br /> 4,5,6<br /> $ seq 1 6 | paste -sd,<br /> 1,2,3,4,5,6</span><br />
<br />
This simple command is starting to show complex behavior. Maybe there is something interesting in those old unix books after all... Wait:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq 1 6 | paste -sd+<br /> 1+2+3+4+5+6</span><br />
<br />
Nice, a mathematical expression. If only we had some way of interpreting it...<br />
<br />
<br />
<b>bc</b><br />
<br />
There are people who say: the python/ruby interpreter is my calculator. To that I say: screw that!<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ bc -ql<br /> 1 + 2 + 3<br /> 6<br /> 10 / 12<br /> .83333333333333333333<br /> scale = 80<br /> 10 / 12<br /> .8333333333333333333333333333333333333333333333333333333333333333333\<br />3333333333333</span><br />
<br />
Do you see that `\` character? It's almost as if it was meant to be used on a shell...<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq 1 6 | paste -sd+ | bc -l</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 21</span><br />
<br />
<br />
<b>Interlude: Gregory-Leibniz</b><br />
<br />
There are many ways of calculating π. You can find many of them on its wikipedia page. One of them is named after two mathematicians, James Gregory and Gottfried Leibniz, goes like this (again from wikipedia):<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/math/9/e/8/9e804b8a1a11e442be93fed1d52205a9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://upload.wikimedia.org/math/9/e/8/9e804b8a1a11e442be93fed1d52205a9.png" /></a></div>
<br />
<br />
This is an infinite series with a simple pattern, which I'm sure you can identify (you weren't sleeping on those calculus classes, were you?). Just in case you can't (and because it is a pretty equation), here it is:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/math/c/a/a/caafcd11bc553464dd4873a76ae4d9a7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://upload.wikimedia.org/math/c/a/a/caafcd11bc553464dd4873a76ae4d9a7.png" /></a></div>
<br />
<br />
<b>Back in unix-land</b><br />
<br />
So here is the challenge: how can we generate and evaluate the terms of this series? Generating each term, without the sign, can be done easily with seq and a format string:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '1/%g' 1 2 9</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> 1/1<br /> 1/3<br /> 1/5<br /> 1/7<br /> 1/9</span><br />
<br />
Remember our useful-where-you-never-imagined friend paste?<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '1/%g' 1 2 9 | paste -sd-+<br /> 1/1-1/3+1/5-1/7+1/9</span><br />
<br />
This may take some time to understand, it's ok. Read those man pages! But once you understand, the only thing left is to evaluate the expression:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '1/%g' 1 2 9 | paste -sd-+ | bc -l<br /> .83492063492063492064</span><br />
<br />
Hmm, not much π-like, is it? Right, this is π/4. Ok, we can rearrange the terms a bit to fit our tools (that is the essential hacker skill). Lets move the denominator on the right side to the numerator on the left.<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '4/%g' 1 2 9 | paste -sd-+ | bc -l<br /> 3.33968253968253968254</span><br />
<br />
That's more like it! As any infinite series approximation, we can increase the number of terms to increase accuracy:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '4/%g' 1 2 9999 | paste -sd-+ | bc -l<br /> 3.14139265359179323814</span><br />
<br />
Now just for the heck of it:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ seq -f '4/%g' 1 2 999999 | paste -sd-+ | bc -l<br /> 3.14159065358979323855</span><br />
<br />
And there you have it. Enjoy your unix π.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-4658802536513117042014-12-10T10:46:00.000-02:002014-12-10T10:49:13.642-02:00 pip --extra-indexThis is a tale of debugging. That fine art of digging the darkest corners of a computer system to solve whatever problem is haunting it. This particular story is about python's package manager, pip.<br />
<br />
Where I work, we have a server where we store all python packages used in development and production. A package "cache" or "proxy". The idea is similar to a http proxy: we don't have to hit PyPi for each package query and install. That saves time, as local connections are much faster, both regarding latency and throughput, and bandwidth, as no packet has to leave our lan.<br />
<br />
One day, all of a sudden, our testing and production servers started taking a long time to run a simple package update. And when I say "a long time", I mean taking more than ten minutes to run a simple `pip install --update` with roughly fifty packages on a requirements.txt file. That is a ridiculously long time. That would be crazy slow even if we were hitting the wan, but on a lan, that is just absurd. So, clearly, something fishy was going on.<br />
<br />
<br />
<b>Debugging begins</b><br />
<br />
Step one when debugging an issue: figure out what changed. I thought long and hard (harder than longer, I must admit) about it, but couldn't think of anything I or anyone else had changed on these machines recently. So I advanced to step two: getting a small, reproducible test.<br />
<br />
In this case, the test can be reduced to a simple command line execution:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ pip install -Ur requirements.txt</span><br />
<br />
This is a standard pip requirements file, with the standard options to prefer our internal server over the official PyPi server:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ head -2 requirements.txt<br /> --index-url=http://pypi.example.com/simple/<br /> --extra-index-url=http://pypi.python.org/simple/</span><br />
<br />
Here and in the next examples, I'll substitute the real domains for our servers for fake ones. Anyway, running that one simple command was all that was needed to test the strange behavior. On my personal development machine, that took quite a while:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 0m41.280s<br /> user 0m3.557s<br /> sys 0m0.100s</span><br />
<br />
Even more interestingly, this was way less time than we were seeing on the servers:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br /> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 13m19.752s<br /> user 0m1.031s<br /> sys 0m0.184s</span><br />
<br />
<br />
<b>On the server</b><br />
<br />
The three main servers affected by this issue (the ones I spend most time in) were our buildbot (i.e. continuous integration), staging and production servers. I decided the first test would be on the buildbot server, as it is the same server where the packages are hosted. That way, I can exclude many external factors that could be affecting the traffic.<br />
<br />
So I fired my favorite tool: the amazing strace. If you don't know it, stop everything and go take a look at `strace(1)`. Since I know most people won't, here's a quick introduction: strace execve(2)'s your command, but sets up hooks to display every system call it executes, along with arguments, return values and everything. It is an amazing tool to have an overall idea of what a process is doing. If you are root (and have CAP_SYS_TRACE), you can even attach to a running process, which is an amazing way to debug a process that starts running wild.<br />
<br />
Using it to run the command:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ strace -r -o strace.txt pip install -Ur requirement.txt</span><br />
<br />
The arguments here are `-o strace.txt` to redirect output to a file and the super useful `-r` to output the relative time between each system call, which is perfect to identify the exact system calls slowing down the execution of the command.<br />
<br />
After the execution was done, looking at the output, I found the culprit. Here is a sample of the log:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ grep -m 1 -B 5 close strace.txt<br /> 0.000138 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4<br /> 0.000115 fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)<br /> 0.000092 fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0<br /> 0.000089 connect(4, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("23.235.39.223")}, 16) = -1 EINPROGRESS (Operation now in progress)<br /> 0.000128 poll([{fd=4, events=POLLOUT}], 1, 15000) = 0 (Timeout)<br /> 15.014733 close(4)</span><br />
<br />
From this point, my analysis had some flaws that delayed the final conclusion. I will explain my line of thought as it happened, maybe you'll find the mistakes before I get to them. So, as can be seen in the output, a socket is opened to communicate with another server, which is normal behavior for pip, but then closing it takes around fifteen seconds. Hmm, that is really odd.<br />
<br />
So I ran the command again and, while it was blocked waiting, I used another useful command to list the open file descriptors of the process:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ lsof -p $(pgrep pip) | tail -2<br /> pip 19928 django 3u IPv4 8064331 0t0 TCP<br /> pypi.example.com:38804->pypi.example.com:http (CLOSE_WAIT)<br /> pip 19928 django 4u IPv4 8064359 0t0 TCP<br /> pypi.example.com:30470->199.27.76.223:http (SYN_SENT)</span><br />
<br />
Here we have two open sockets. One of them is in the CLOSE_WAIT state. Anyone who's ever done socket programming knows this dreaded state, where the local socket is closed but the remote end doesn't send the FIN packet to terminate the connection. A few minutes of tcpdump later, I was convinced that was the problem: something was preventing the connection from ending and each operation was waiting for the timeout to close the socket. That would explain why closing the socket took so long.<br />
<br />
<br />
<b>The mistakes</b><br />
<br />
At this point, I realised my first mistake. If you take a look at the strace output again, the remote end of the socket is *not* our server. Take a look at the remote address (the `sin_addr` parameter to the `socket` call): 23.235.39.223 is not the ip address of our server, and taking a look at the rest of the output showed that the address changed over time.<br />
<br />
There should be no other servers involved, since we explicitly told pip to fetch packages from our own server. So I thought: what other server could be involved here? So I took a guess:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ dig pypi.python.org | grep -A 2 'ANSWER SECTION'<br /> ;; ANSWER SECTION:<br /> pypi.python.org. 52156 IN CNAME python.map.fastly.net.<br /> python.map.fastly.net. 30 IN A 23.235.46.223</span><br />
<br />
Damn... Wait!<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ dig pypi.python.org | grep -A 2 'ANSWER SECTION'<br /> ;; ANSWER SECTION:<br /> pypi.python.org. 3575 IN CNAME python.map.fastly.net.<br /> python.map.fastly.net. 7 IN A 23.235.39.223</span><br />
<br />
Bingo! So it was a connection to one of PyPi's servers. I went back to the strace output and realised my second mistake. If you read strace's man page section for the `-r` option carefully, the delta shown before each line is not the time each syscall took, but the time between that syscall and the last. So the operation that was getting stuck was not `close`, but the previous, `epoll`.<br />
<br />
In hindsight, it is obvious. You can see the indication that the call timed out. You can even see the timeout is one of the parameters. And so the mystery was solved. By some unknown reason, pip was trying to make a connection to PyPi after checking our server. Since we don't allow that, the operation hang around until the timeout was reached. One final test confirmed our suspicion:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 0m42.981s<br /> user 0m3.720s<br /> sys 0m0.070s<br /> $ sed -ie 's/^--extra-index/#&/' requirements.txt<br /> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 0m1.049s<br /> user 0m0.853s<br /> sys 0m0.057s</span><br />
<br />
Removing the extra index option eliminated the issue (and gave us a ~42x speed up, something you don't see everyday).<br />
<br />
<br />
<b>Conclusion</b><br />
<br />
So, what do we take out of this (unexpectedly long) story? If you are using a local package server, don't use `--extra-index`. I have no idea why pip was trying to contact the extra index after finding the package on our server. The only reason I can think of is it is trying to find a newer version of the package, but even then, most of our requirements are fixed, i.e. they have '==${some_version}' appended.<br />
<br />
Even on my development machine, where pip can reach the remote server, it is worth it to remove the option. The time it takes just to reach the server for each package, even just to receive a "package up-to-date" message, slows down the operation considerably:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;"> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 0m46.816s<br /> user 0m3.853s<br /> sys 0m0.130s<br /> $ sed -ie 's/^--extra-index/#&/' requirements.txt<br /> $ time pip install -Ur requirement.txt &> /dev/null<br /><br /> real 0m1.125s<br /> user 0m0.947s<br /> sys 0m0.053s</span><br />
<br />
<br />
<b>Coda</b><br />
<br />
Thank you for making it this far. I hope this story was entertaining and hopefully it taught you a thing or two about investigation, debugging and problem solving. Take the time to learn the basic tools: ps, lsof, tcpdump, strace. I assure you they will be really useful when you encounter this type of situation.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-9165375815157430922014-05-02T13:13:00.000-03:002014-05-02T13:19:07.584-03:00Manufacturer's bluesThe state of technology sometimes gives me the blues. In 2014, I found myself installing a package called `dosfstools` to format my pendrive. That's because the car's radio wouldn't accept any device with a filesystem that's not fat32:
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiyvdoAxmbRzgLTps2oTUzqB-8VvQZ9TPEYmdn1sfFArJy4SZOIJWQCMOFXAt1Vac1B8uNJVjIsCAsP1pomr7YZKs2FcQbmP74lwKNBcMMF7pMz3wHGbKbpS-w1hFd-22aKfIC3PgaOCg/s1600/2014-05-02_15-37-20_841.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiyvdoAxmbRzgLTps2oTUzqB-8VvQZ9TPEYmdn1sfFArJy4SZOIJWQCMOFXAt1Vac1B8uNJVjIsCAsP1pomr7YZKs2FcQbmP74lwKNBcMMF7pMz3wHGbKbpS-w1hFd-22aKfIC3PgaOCg/s320/2014-05-02_15-37-20_841.jpg" /></a></div>
<br />
Of course possessing a filesystem of another kind makes the device "not a device". After a few seconds on the command line the problem went away:
<br />
<pre>$ sudo pacman -S dosfstools
$ sudo mkfs.vfat -n BILLY /dev/sdb1</pre>
but the sadness remained. At l(e)ast I can listen to some <a href="http://www.badvoltage.org/">badvoltage</a> while driving.
<br />
<br />
Oh yes, did I mention it doesn't accept any format <em>except</em> mp3? And how does it signal it? By not showing the files at all, of course.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-5628402732675695332014-03-03T21:26:00.000-03:002014-05-02T13:15:47.748-03:00Uploading files the unix way<i>I just did something which I thought was worth sharing. It shows a couple of fundamental techniques in unix systems used in a context where most people would not think of.</i>
<br />
<br />
The context: my mom sent me a recipe of potato souffle (I hope google translate is doing the right thing here, <a href="https://www.google.com/search?q=potato+souffle"> it seems like it is</a>). But as most of e-mails coming from non-tech people, there was a lot of noise along the actual recipe text. Since I wanted to view it in my cell phone while I cooked, I decided to clean it up a bit.
<br />
<br />
So did I: copied the text to vim (which got rid of most of the noise immediately), edited a bit and it was ready:
<br />
<br />
<pre> Ingredientes
450 gramas: Batata cozida
200 gramas: Queijo ricota caseiro – Guloso e Saudável
4 colheres de sopa: Queijo gorgonzola
3 colheres de sopa: Farinha de trigo branca
300 mililitros: Leite desnatado
2 unidades: Gema de ovo
3 unidades: Clara de ovo
1 colher de sopa: Fermento em pó
1/2 unidade: Cebola ralada
4 colheres de sopa: Salsa fresca cortada
Modo de preparo
Amasse os queijos, reserve
Amasse as batatas, reserve
Preaqueça o forno a 200ºC
Bata as claras em neve com uma pitada de sal, reserve
Numa vasilha misture as gemas, a farinha, o leite, a cebola, a batata, os queijos, a salsinha
Junte delicadamente o fermento e as claras em neve
Coloque o suflê de batata em ramequins e leve ao forno por 30 minutos ou até assar
Sirva de imediato o suflê de batata com queijos, acompanhado de salada</pre>
<br />
Now all I needed was a way to get it on my cellphone. I could use dropbox or any of the related services, but that didn't seem exciting enough[<a href="#note1">1</a>]. So I thought: well, I'll just put it in my webserver and access it in my mobile browser.
<br />
<br />
Fair enough. But this is a file opened in vim[<a href="#note2">2</a>]. After a little thinking (which lasted for approximately 423ms), this is what I did. From inside vim, I executed the following command:
<br />
<blockquote class="tr_bq">
<pre>:!ssh raspberrypi 'nc -l 31415 > /srv/http/sufle.txt' &</pre>
</blockquote>
Which launched an ssh client connecting to my webserver[<a href="#note3">3</a>] running the netcat command. If you are not familiar with netcat, think of it as the traditional cat command which reads and writes to sockets instead of files. Here, I'm telling it to listen (<span style="font-family: "Courier New",Courier,monospace;">-l</span>) on port 31415 and output anything read to the file <span style="font-family: "Courier New",Courier,monospace;">/srv/http/sufle.txt</span>, where <span style="font-family: "Courier New",Courier,monospace;">/srv/http</span> is the document root of the webserver.
<br />
<br />
Since I added the ampersand to the end of the command, the command is launched in the background and just stays there waiting for input, releasing the shell immediately so I can continue using vim. Now that we're back, another vim command:
<br />
<blockquote class="tr_bq">
<pre>:w !cat > /dev/tcp/raspberrypi/31415</pre>
</blockquote>
Here we are telling vim to write the contents of the buffer (<span style="font-family: "Courier New",Courier,monospace;">:w</span>). Normally, we would give it the name of a file to write, but the <span style="font-family: "Courier New",Courier,monospace;">:write</span> command accepts a special syntax, where instead of a file name, we put a <span style="font-family: "Courier New",Courier,monospace;">!</span> and the rest of the line is interpreted as a command<span style="font-family: "Courier New",Courier,monospace;"></span>[<a href="#note4">4</a>]. This command is executed in a standard unix shell, but with stdin redirected to read the contents of the buffer. As an example, try opening a new empty buffer in vim (<span style="font-family: "Courier New",Courier,monospace;">c-w n</span>), writing something and executing the command[<a href="#note5">5</a>]:
<br />
<blockquote class="tr_bq">
<pre>:w !cowsay</pre>
</blockquote>
Continuing the analysis of the command, we are using the <span style="font-family: "Courier New",Courier,monospace;">:w !</span> command to write to the program run by the shell, which is
<br />
<blockquote class="tr_bq">
<pre>$ cat > /dev/tcp/raspberrypi/31415</pre>
</blockquote>
Well, you don't need to be a unix wizard to know what this is doing. cat is a command that reads the contents from a list of files (or stdin if it's invoked without arguments, as it is here) and writes them in the same order to its stdout. Since its stdin will read the contents of the vim buffer, we already know the effect of this command: the contents of the buffer will get written to a file.
<br />
<br />
But what is that funny file starting with <span style="font-family: "Courier New",Courier,monospace;">/dev/tcp/</span>? That is feature of bash (I don't know about other shells). You can read about using the command[<a href="#note7">7</a>]:
<br />
<blockquote class="tr_bq">
<pre>$ man -P 'less -p /dev/tcp' bash</pre>
</blockquote>
What we are doing here is telling bash to write the data coming from cat's output to port 31415 on the host raspberrypi. Remember our little ssh friend we left running in the background a while ago? He's not done much since[<a href="#note8">8</a>], but now there is data coming from the socket it's listening on, so the os wakes him up to do its job.
<br />
<br />
And so it does, reading the data from the socket and writing it to stdout. Remember what that is? That's the file we passed to the shell, <span style="font-family: "Courier New",Courier,monospace;">/srv/http/sufle.txt</span>. And so our long journey comes to an end. When nc finishes reading and writing the data, it dies[<a href="#note9">9</a>] and so does the ssh client process, which was also sitting there, waiting for it's child process do die[<a href="#note10">10</a>]. The shell that spawned the ssh client has long died: it had nothing more to do because we started the process asynchronously (using the <span style="font-family: "Courier New",Courier,monospace;">&</span> on the command line), so that's the end of that (process) family tree[<a href="#note11">11</a>].
<br />
<br />
The result? The contents of the buffer have now been transfered and written to the file <span style="font-family: "Courier New",Courier,monospace;">/src/http/sufle.txt</span> on the remote host raspberrypi.
<br />
<br />
<h3>
Conclusion</h3>
<br />
You might be thinking: "What the hell? You are telling me all this is simpler than dropbox?". And I would be lying if I said "yes". But the point here is not that this is simple. In fact, it's the base of one of the most complex systems in computer history: the operating system. And even though I have detailed some parts of the processes, there are a lot, and I really mean *a lot* more things going on behind the scenes here[<a href="#note12">12</a>].
<br />
<br />
But if you step back, you will realize that all that was needed were two commands:
<br />
<blockquote class="tr_bq">
<pre>ssh raspberrypi 'nc -l 31415 > /srv/http/sufle.txt'
cat > /dev/tcp/raspberrypi/31415</pre>
</blockquote>
What I'm trying to present here is the incredible complexity and richness that can be achieved by using shell commands and unix concepts like input and output redirections and socket communication. Another goal was to show how a powerful text editor incorporates this concept in its design, taking advantage of of the powerful features of the operating system to extend its capabilities[<a href="#note13">13</a>].
<br />
<br />
And I bet I can type these two commands faster than your dropbox client can transfer the file to your cellphone[<a href="#note14">14</a>]. Now, I you'll excuse me, there is a potato souffle that needs to be cooked.<br />
<br />
<h3>
Bonus</h3>
<br />
While doing some scientific tests on the cowsay program (trying to make the cow say the text of this post), I discovered perl has a special mention of my blog on its source code[<a href="#note6">6</a>]:
<br />
<blockquote class="tr_bq">
<pre>:w !cowsay
This shouldn't happen at /usr/share/perl5/core_perl/Text/Wrap.pm line 84, <stdin> line 63.
shell returned 255</pre>
</blockquote>
<h3>
Notes</h3>
<br />
<a href="https://www.blogger.com/null" name="note1"></a>1. What sane geek uses dropbox, anyway?<br />
<a href="https://www.blogger.com/null" name="note2"></a>2. Actually just a buffer, since I didn't even write it to a file.<br />
<a href="https://www.blogger.com/null" name="note3"></a>3. A rπ.<br />
<a href="https://www.blogger.com/null" name="note4"></a>4. The space after <span style="font-family: "Courier New",Courier,monospace;">:w</span> is important here. If we wrote <span style="font-family: "Courier New",Courier,monospace;">:w!cat ...</span>, it would be interpret as the <span style="font-family: "Courier New",Courier,monospace;">:write!</span> command, which is not what we want here. See <span style="font-family: "Courier New",Courier,monospace;">:help :write!</span> on vim.<br />
<a href="https://www.blogger.com/null" name="note5"></a>5. What do you mean "I don't have cowsay installed"?<br />
<a href="https://www.blogger.com/null" name="note6"></a>6. You can check it <a href="http://sources.debian.net/src/perl/5.18.2-2/cpan/Text-Tabs/lib/Text/Wrap.pm?hl=1#L84" target="_blank">here</a> (thanks sources.debian.net for their awesome service).<br />
<a href="https://www.blogger.com/null" name="note7"></a>7. Don't worry, that will only open a man page passing a special flag to the pager telling it to go directly to the right section.<br />
<a href="https://www.blogger.com/null" name="note8"></a>8. He hasn't been doing anything, actually, thanks to the magic of process scheduling and blocking io syscalls, so no resources are wasted (except for a few bytes on the process and file tables).<br />
<a href="https://www.blogger.com/null" name="note9"></a>9. Ok, it terminates its execution, "dies" sounds so dramatic.<br />
<a href="https://www.blogger.com/null" name="note10"></a>10. Just when you thought it could not get more dramatic.<br />
<a href="https://www.blogger.com/null" name="note11"></a>11. Ok, that was on purpose =)<br />
<a href="https://www.blogger.com/null" name="note12"></a>12. Including, for example, the whole file buffer cache algorithm that makes this whole process absurdly efficient, even though we seem to be copying data around dozens of times. That is the subject of whole book chapters (like <a href="http://www.amazon.com/The-Design-UNIX-Operating-System/dp/0132017997" target="_blank">this one</a>).<br />
<a href="https://www.blogger.com/null" name="note13"></a>13. My trademark phrase is: "do *that* in gedit".<br />
<a href="https://www.blogger.com/null" name="note14"></a>14. After it has sent your data, maybe without using encryption, across the internet and sent a copy of the recipe to the NSA for analysis.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-53009641893193993812013-07-14T19:14:00.000-03:002013-07-14T19:18:01.680-03:00A post-mortem<a href="http://iffalse.blogspot.com/2013/01/the-raspberry.html" target="_blank">Part I - The raspberry</a><br />
<a href="http://iffalse.blogspot.com/2013/02/an-image-is-worth-thousand-megabytes.html" target="_blank">Part II - An image is worth a thousand megabytes</a><br />
<b>Part III - A post-morten</b><br />
<br />
<i><a href="http://iffalse.blogspot.com/2013/05/that-moment-you-realize-something.html" target="_blank">A lot</a> has happened since <a href="http://iffalse.blogspot.com/2013/01/the-raspberry.html" target="_blank">my first rπ post</a>. I've used that little thing to death (quite literally), as a ssh box, firewall, general server, web server, web browser, movie box and a whole lot more. Then, I've had the SD card die on me. For me, it was a perfect little linux computer that could stay on all the time, consuming little-to-no power, making absolutely no noise, occupying minimal space. In a few words: really, really awesome. And all that for 35 dollars!</i><br />
<br />
As promised, I'll post here the steps to get from a rπ on a closed box to a little box of awesomeness. I know there a ton of posts like these, but hey, storage is free (for now) on blogger. If you haven't read the first post, this is a good time to do it. I will first make a few post-morten (again, literally) observations about my first assumptions.<br />
<br />
<b>SD card</b>: let's start with the most important thing. When I first went shopping for the various attachments, I got the most inexpensive card I found. The only requisite was that it had at least a 4gb storage. After it died, I decided to spend a few extra bucks and get a decent one. Man, I was surprised. There isn't any way of comparison between the π before and now. Instead of moaning the old days, I'll just say this: get a decent, class 10 SD card and you'll have nothing to worry about. And since these things get cheaper and cheaper, get a 16gb one.<br />
<br />
<b>Power supply</b>: again, when I started using the π, I just used my cell phone charger, which, by extreme luck, was compatible and provided almost exactly the amount of power needed. Then I started using a wireless adapter sometimes. Then I started using hdmi output sometimes. Then I tried to use both at the same time. And it died. Not that it does any harm to the board, but any moderate load will simply power-off the device. So, if you want to use more than the π itself, get an appropriate power supply. <a href="http://uk.farnell.com/raspberry-pi-accessories#usbpowersupply" target="_blank">Those on the farnell website</a> worked just fine for me. And just for completeness sake, mine is 1000mA.<br />
<br />
<b>HDMI</b>: I did my first tests on my living room TV. But after a while (as addiction got worse), I felt the need to use it on my bedroom, where I have only an old computer monitor with VGA input. So I got a VGA-to-HDMI adapter and things worked fine (well, almost, more on that later).<br />
<br />
<b>OS</b>: the first distribution I installed was raspbian, mainly because I was used to debian and, by the time I started playing, there weren't many alternatives. Now we have pidora and even riscos. But anyway, raspbian is a really good distribution and I never had a problem with it. But I am restless. Since I was going to reinstall the OS from /dev/zero, I decided to jump from the cliff and install arch. For no special reason, just to try something else. So, the instructions from now on will be arch centered, although many will be usable on other distributions.<br />
<br />
Man, I write a lot. Let's stop right here and I'll get to the first installation steps later.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-57148418162818376942013-05-30T19:18:00.000-03:002014-05-02T13:15:47.752-03:00That moment you realize something really wrong is going on<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVavWHwyJgnYvfpgW5JefWsIFaCZeXnt4xwCYUKyz17UG-i0vF4uG4TGKm9eEbTGatTJHxruXeeTytbmNgWjTjq9-KYaWpT1KKq3yyoDVlI1tYeiObd55g0skh0dRRP3ugzcgGHCI-OY4/s1600/ls.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVavWHwyJgnYvfpgW5JefWsIFaCZeXnt4xwCYUKyz17UG-i0vF4uG4TGKm9eEbTGatTJHxruXeeTytbmNgWjTjq9-KYaWpT1KKq3yyoDVlI1tYeiObd55g0skh0dRRP3ugzcgGHCI-OY4/s320/ls.png" width="320" /></a></div>
<br />Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com1tag:blogger.com,1999:blog-4492442420641722674.post-17336134557470901842013-05-11T16:17:00.000-03:002013-05-11T16:17:28.137-03:00python mocking<i>If you are into unit testing, you probably have been introduced to mocking. And if that is the case, you probably already have been bitten by it. Mocking requires some understanding of code execution, importing and name resolution that most people lack when first encountering such situations. In python, mocking is a relatively simple process, if you analyze carefully what needs to be done.</i><br /><br />Mocking simply means replacing an object with another. This is usually done to avoid instantiating costly systems or to change the behaviour of a system. To begin with a simple example:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>def</b> some_function_in_your_code(a):</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>if</b> a.do_something():</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>return</b> <i>3.14159</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>else</b>:</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>return</b> <i>6.26318</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"></span><br /><span style="font-family: "Courier New",Courier,monospace;"><b>def</b> some_other_function_in_your_code():</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <i># ...</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"> a = create_complicated_object(*thousand_parameters)</span><br /><span style="font-family: "Courier New",Courier,monospace;"> result = some_other_function_in_your_code(a)</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <i># ...</i></span></blockquote>
Here we have a function <span style="font-family: "Courier New",Courier,monospace;">some_other_function_in_your_code</span> that uses an object. Somewhere else in your code, <span style="font-family: "Courier New",Courier,monospace;">some_other_function_in_your_code</span> creates that object, which is a complicated process that involves hundreds of operations. If you just want to test <span style="font-family: "Courier New",Courier,monospace;">some_other_function_in_your_code</span>, you shouldn't need to go through this whole process[1].<br /><br />To avoid that, we can use mocking. Notice that all we need to test on <span style="font-family: "Courier New",Courier,monospace;">some_other_function_in_your_code</span> is that the argument passed has a member called "do_something" that can be called with no arguments and returns something that can be converted to <span style="font-family: "Courier New",Courier,monospace;">bool</span>. There are many ways to do that, but to keep things short, I'll skip right to the library I use most of the times, <span style="font-family: "Courier New",Courier,monospace;">mock</span>.<br /><br />The main component of the <span style="font-family: "Courier New",Courier,monospace;">mock</span> library is the <span style="font-family: "Courier New",Courier,monospace;">Mock</span> class, which is basically an empty object with a few useful characteristics (be sure to check the documentation, because they are really useful). Two of them are important for this discussion. First, every time you access an attribute of a <span style="font-family: "Courier New",Courier,monospace;">Mock</span> object, another <span style="font-family: "Courier New",Courier,monospace;">Mock</span> object is created and assigned to the attribute being accessed.<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m = mock.Mock()</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>print</b>(m)</span><br /><i><span style="font-family: "Courier New",Courier,monospace;"><Mock id='140095453080976'></span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>print</b>(<b>id</b>(m.some_attribute))</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">140095453131664</span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>print</b>(m.some_attribute)</span><br /><i><span style="font-family: "Courier New",Courier,monospace;"><Mock name='mock.some_attribute' id='140095453131664'></span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>print</b>(<b>id</b>(m.some_attribute))</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">140095453131664</span></i></blockquote>
As you can see, when we tried to access some_attribute, a <span style="font-family: "Courier New",Courier,monospace;">Mock</span> object was created for us. We could do it by hand with a single line of code, but it makes it makes the code easier, shorter and cleaner.<br /><br />The other feature of <span style="font-family: "Courier New",Courier,monospace;">Mock</span> objects is the <span style="font-family: "Courier New",Courier,monospace;">return_value</span> attribute. Whatever is contained in this attribute gets returned when the object is called as a function object [2].<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m = mock.Mock()</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m.return_value = <i>u'the return value'</i></span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">u'the return value'</span></i></blockquote>
Using this two techniques, we can now test our function:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;"></span><br /><span style="font-family: "Courier New",Courier,monospace;"><b>def</b> your_test():</span><br /><span style="font-family: "Courier New",Courier,monospace;"> a = mock.Mock()</span><br /><span style="font-family: "Courier New",Courier,monospace;"> a.do_something.return_value = <i>True</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>if</b> some_function_in_your_code(a) != <i>3.14159</i>:</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>raise</b> <b>Exception</b>(<i>u'return wasn't 3.14159.'</i>)</span><br /><span style="font-family: "Courier New",Courier,monospace;"> a.return_value = <i>False</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>if</b> some_function_in_your_code(a) != <i>6.28318</i>:</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>raise</b> <b>Exception</b>(<i>u'return wasn't 6.28318.'</i>)</span></blockquote>
Let's break that down. We first create a <span style="font-family: "Courier New",Courier,monospace;">Mock</span> object to represent the argument passed to the function. The next line takes care of the two things we need to test the function: the <span style="font-family: "Courier New",Courier,monospace;">do_something</span> attribute and its return value. Then, all we have to do is call the function, passing the mocked argument, and check the return value. After that, repeat the process, this time with a different return value.<br />
<br />
<h4>
That was the easy part</h4>
<br />
This first section was easy, nothing you couldn't find out with a quick search on the internet. But the real world is not that pretty (at least mine isn't). The trickiest situation is when you have to change the behaviour of something inside a function, but you don't <i>pass</i> that something as an argument. Suppose we have this:<br />
<blockquote class="tr_bq">
<i><span style="font-family: "Courier New",Courier,monospace;"># some_file.py</span></i><br /><span style="font-family: "Courier New",Courier,monospace;"><b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;"></span><br /><span style="font-family: "Courier New",Courier,monospace;"><b>def</b> f():</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>if</b> random.random() < <i>0.5</i>:</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>return</b> <i>3.14159</i></span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>else</b>:</span><br /><span style="font-family: "Courier New",Courier,monospace;"> <b>return</b> <i>6.28318</i></span></blockquote>
How can you test that function if the value tested is conjured from oblivion in the middle of the function? Fear not, you can actually do it. The trick here is to mock <i>the <span style="font-family: "Courier New",Courier,monospace;">random</span> function</i> from the <span style="font-family: "Courier New",Courier,monospace;">random</span> module before the first line of <span style="font-family: "Courier New",Courier,monospace;">f</span> is executed. Let's start with this simple example, just to get the basic idea:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">0.5212285734499994</span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">0.40492920488281725</span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random = mock.Mock(return_value=<i>0.5</i>)</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">0.5</span></i></blockquote>
This seems pretty simple. But here is where everybody gets lost:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random = mock.Mock(return_value=<i>0.5</i>)</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> some_file</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> some_file.f() == <i>6.28318</i> <b>or</b> <b>raise</b> ThisShouldNotBeHappeningException()</span></blockquote>
This may not explode the first time, but you have a 50% chance of getting an exception. To see why, you can put a <span style="font-family: "Courier New",Courier,monospace;">print random.random</span> inside <span style="font-family: "Courier New",Courier,monospace;">f</span> and see that the mock didn't work. To understand why, we have to dig a little deeper.<br />
<br />
<h4>
Import</h4>
<br />
What happens when you run <span style="font-family: "Courier New",Courier,monospace;">import random</span>? You can watch <a href="http://pyvideo.org/video/1707/how-import-works" target="_blank">this video</a> to know exactly what. Or you can just continue reading to get a summary. Or both. Anyway:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>locals</b>()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', '__doc__': None, '__package__': None}</span></i><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>locals</b>()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">{'__builtins__': <module '__builtin__' (built-in)>, '__name__': '__main__', 'random': <module 'random' from '/usr/lib/python2.7/random.pyc'>, '__doc__': None, '__package__': None}</span></i></blockquote>
In python, <span style="font-family: "Courier New",Courier,monospace;">locals</span> is a built-in function that returns the local variables accessible on the current state of execution (don't believe me, run <span style="font-family: "Courier New",Courier,monospace;">print(__builtins__.locals)</span>). When you execute <span style="font-family: "Courier New",Courier,monospace;">import random</span>, the interpreter does its magic to find the module and load it, but more important, it creates an entry called "random" on the current namespace referring to the module loaded. The critical part here is "current namespace". Try this:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>def</b> f(): <b>print</b>(<b>locals</b>()</span>)<br /><span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> f()</span><br /><i><span style="font-family: "Courier New",Courier,monospace;">{}</span></i></blockquote>
Here, importing random didn't affect the namespace on f. The same thing applies to namespaces of other modules. Our example fails because the namespace on the <span style="font-family: "Courier New",Courier,monospace;">some_file</span> module is different than the namespace where we run our tests. To change the namespace of <span style="font-family: "Courier New",Courier,monospace;">some_file</span>, we have to do it explicitly:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> some_file</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> some_file.random.random = <b>lambda</b>: <i>0.5</i></span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> some_file.f() == <i>6.28318</i> <b>or</b> <b>raise</b> ThisShouldNotBeHappeningException()</span></blockquote>
You can run that many times if you don't trust me, but that will always succeed. And it does because we now are changing the correct namespace. You can check it by putting a <span style="font-family: "Courier New",Courier,monospace;">print(random.random)</span> on <span style="font-family: "Courier New",Courier,monospace;">f</span> again.<br />
<br />
<h4>
Being nice</h4>
<br />
Now you know how to mock, but there is something I must say before you leave. Always, always, ALWAYS restore any mock you do. Seriously. Even if you're sure no one will use the mocked attribute. You don't want to loose an entire day of work just to find out that the problem was an undone mock.<br /><br />And doing it is so simple: store the original value on a variable and restore it after the operation. I like to do it as soon as the operation is complete, before anything else is executed, but you don't need to, if you're not paranoid. Just to clear any doubt, here is exactly how to do it:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> original_random = random.random</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random = <b>lambda</b>: <i>0.5</i></span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> <i># do something</i></span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> random.random = original_random </span></blockquote>
Now you have no excuse. Better yet, you can use another feature of the <span style="font-family: "Courier New",Courier,monospace;">mock</span> library called <span style="font-family: "Courier New",Courier,monospace;">patch</span>. But that would be an extension to an already long post. Maybe I'll cover it in the future. Anyway, happy mocking!<br />
<br />
<h4>
Notes</h4>
<br />
1: You shouldn't have complicated processes that involve hundreds of operations anyway, but that is another problem.<br /><br />2: Curious to know what happens when you access <span style="font-family: "Courier New",Courier,monospace;">return_value</span> without setting it first? No? Well, I'll show you anyway:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">>>> <b>import</b> mock</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m = mock.Mock()</span><br /><span style="font-family: "Courier New",Courier,monospace;">>>> m.return_value</span><br /><i><span style="font-family: "Courier New",Courier,monospace;"><Mock name='mock()' id='140095453168080'></span></i></blockquote>
Since we didn't set it, we get the default behaviour of <span style="font-family: "Courier New",Courier,monospace;">__setattr__</span>, which is to create another <span style="font-family: "Courier New",Courier,monospace;">Mock</span>.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-42625825765230555082013-02-24T01:06:00.000-03:002014-05-02T13:15:47.741-03:00Handy bash substitution<i>Working on the command line is awesome. But there are times where
all the typing gets unwieldy. No one likes to type a long file name a
lot of times. Sure, using the up arrow and tab keys can make things
faster sometimes, but I'll show you some tricks to make your life
easier.</i><br />
<br />
Let's start with a really common operation. You download a tarball, extract, cd and have fun. Usually, you'd do this:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ tar -xjf linux-3.7.9.tar.bz2</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ cd linux-3.7.9</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ bob's your uncle</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">></span></b></blockquote>
Using the Tab key, you can save a lot of typing:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.9.tar.bz2</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ tar -xjf l<TAB></span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ cd l<TAB></span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ bob's your uncle</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">></span></b></blockquote>
However,
this is highly dependant on the contents of your directory, names in
your path and auto-complete configurations. I'm not being picky here.
While testing these commands, I had to press Tab twice. First, I had a
directory called "libs", so it stopped at "li". Second, I had an episode
of the <a href="http://www.jupiterbroadcasting.com/show/linuxactionshow/" target="_blank">Linux Action Show</a>
in a file called linuxactionshowep248.mp4, so it stopped at "linux".
And, after you decompress the file, it stops at "linux-3.7.9". Also, if
your auto-completion isn't smart enough to filter names that aren't
files or directories when you type Tab after cd and tar, you could have
many more conflicts.<br />
<br />
I don't mean to bash (ha!) on
auto-completion, just point you to more efficient alternatives: bash
(get it?) has a nice feature called substitutions. You may have come
across it before if you ever tried to add an unescaped exclamation mark
(henceforth mentioned using the niftier name "bang"):<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ echo "Bang!"</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">bash: !": event not found</span></blockquote>
Or got surprised by it:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>$ echo "balrog:!you shall not pass:/nonexistent:/bin/false" >> /etc/passwd</b><br />echo "balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false" >> /tmp/test</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>$ sleep 3d; grep balrog /etc/passwd</b></span><br />
<span style="font-family: "Courier New",Courier,monospace;">balrog:youtube-dl http://www.youtube.com/watch?v=pLgJ7pk0X-s shall not pass:/nonexistent:/bin/false</span></blockquote>
That is because the bang is used to reference previous commands. The simplest is substitution of the last command:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ echo billy</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">billy</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ !!</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">echo billy </span><br />
<span style="font-family: "Courier New",Courier,monospace;">billy</span></blockquote>
One
thing I should mention: to avoid chaos and destruction (or at least
inform you that chaos and destruction have happened) the command with all
substitutions made is printed before its output. That is the fourth line
in the example, showing that <span style="font-family: "Courier New",Courier,monospace;">!!</span> was substituted by <span style="font-family: "Courier New",Courier,monospace;">echo billy</span>.<br />
<br />
This
substitution is simple and needs no explanation, but it raises a
question: what is it useful for? Have you ever typed a (possibly long)
command, just to get the following message after running it?<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ cat
/sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">cat: /sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_: Permission denied</span></blockquote>
You can solve this by typing <span style="font-family: "Courier New",Courier,monospace;"><UP><HOME>sudo<SPACE></span> (if you type <span style="font-family: "Courier New",Courier,monospace;"><UP><LEFT>{201}sudo<SPACE></span>, I will personally punch you in the face). But you can save a lot of typing just using<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ sudo !!</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">sudo cat
/sys/module/fglrx/sections/.gnu.linkonce.t._ZN20OS_COMMON_INTERFACES22cailMicroEngineControlEPv14_MICRO_ENGINE_21_MICRO_ENGINE_ACTION_P28_MICRO_ENGINE_CONTROL_INPUT_P29_MICRO_ENGINE_CONTROL_OUTPUT_</span><br />
<span style="font-family: "Courier New",Courier,monospace;">[sudo] password for billy:</span></blockquote>
Here, "!!" got substituted by the last command, as you can see on the echo. Some other uses I found (which you may not understand if you're not or is on the path to <strike>enlightenment</strike> becoming a bash enthusiast) were:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ cat my_files</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">file1</span><br />
<span style="font-family: "Courier New",Courier,monospace;">file2</span><br />
<span style="font-family: "Courier New",Courier,monospace;">file3</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ cat $(!!)</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">cat $(cat my_files)</span><br />
<span style="font-family: "Courier New",Courier,monospace;">contents_of_file1</span><br />
<span style="font-family: "Courier New",Courier,monospace;">contents_of_file2</span><br />
<span style="font-family: "Courier New",Courier,monospace;">contents_of_file3</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ perl -e 'bang your head on the keyboard'</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;"># god only knows</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>$ echo "!!" > file_to_save_command # This could be done by clever use of Ctrl-x Ctrl-e, but sometimes you're in a hurry. That's alright, as long as you <a href="http://tldp.org/LDP/abs/html/escapingsection.html" target="_blank">do it carefully</a></b>.</span></blockquote>
When <span style="font-family: "Courier New",Courier,monospace;">!</span> is followed by a number <i>n</i>, it executes the <i>n</i>th command in the history (which you can check with the <span style="font-family: "Courier New",Courier,monospace;">history</span> command). More useful, though, is that, if <i>n</i> is negative, it will execute the last <i>n</i>th command:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ echo some</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">some</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ echo thing</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">thing</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>$ !-2</b></span><br />
<span style="font-family: "Courier New",Courier,monospace;">echo some</span><br />
<span style="font-family: "Courier New",Courier,monospace;">some</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>$ !-3 other thing</b></span><br />
<span style="font-family: "Courier New",Courier,monospace;">echo some other thing</span><br />
<span style="font-family: "Courier New",Courier,monospace;">some other thing</span></blockquote>
Without thinking much, you can discover that <span style="font-family: "Courier New",Courier,monospace;">!!</span> is just an easier way to spell <span style="font-family: "Courier New",Courier,monospace;">!-1</span>.<br />
<br />
Now, these are useful techniques, but we are just getting started! Usually, you don't need the whole command, but just one or a few arguments. Common scenario:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ ls ef</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">son daughter dog</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ # oh my, ef is a directory</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ cd ef</span></b></blockquote>
or how about:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ mv some_directory a_diferent_name_for_some_directory</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ cd a_diferent_name_for_some_directory</span></b></blockquote>
It is very common to address the same file on multiple, successive commands. You can, again, save some typing:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ ls ef</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">son daughter dog</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ # oh my, ef is a directory</span><br /><span style="font-family: "Courier New",Courier,monospace;">$ cd !$</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">cd ef</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ mv some_directory a_diferent_name_for_some_directory</span><br /><span style="font-family: "Courier New",Courier,monospace;">$ cd !$</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">cd some_directory a_diferent_name_for_some_directory</span></blockquote>
As you can see, <span style="font-family: "Courier New",Courier,monospace;">!$</span> is substituted by the last argument of the last command. If you know your regular expressions, you can make a link with the "end of line" symbol. But there's more. This is just a nice shortcut to the more general form <span style="font-family: "Courier New",Courier,monospace;">!<i>n</i>:<i>m</i></span>. This means: from the <i>n</i>th, get the <i>m</i>th argument.<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ tar -f my_tar -cz file1 file2 file3</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ du -sh !:2</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">4.0K my_tar</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ du -sh !-2:4-6</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">4.0K file1<br />4.0K file2<br />4.0K file3</span></blockquote>
What goes before the colon can be any of the previous substitutions: <span style="font-family: "Courier New",Courier,monospace;">!!</span>, <span style="font-family: "Courier New",Courier,monospace;">!<i>n</i></span> or <span style="font-family: "Courier New",Courier,monospace;">!-<i>n</i></span>. Note that you can specify a range using <span style="font-family: "Courier New",Courier,monospace;">:<i>n</i>-<i>m</i></span>, which will get substituted by the parameters <i>n</i> to <i>m</i>. If you need them in a different order, or some non-contiguous arguments, just use more than one substitution:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ diff ours theirs</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ # gah</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ diff !:2 !:1</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">diff theirs ours</span></blockquote>
Alright, I guess this is enough information for a single post. Have fun playing with The Bash and see you soon for more advanced uses of history substitution.<br />
<br />
P.S.: I couldn't leave without some closing notes.<br />
<br />
<ul>
<li>I learned everything shown here from scattered information on the internet. I decided to compile them because I never found a document which explained them in detail AND without burying them on other bash-related commands.</li>
<li>That said, a good reference (not so good for self-learning, but, if you read the text, you should be okay) is bash's man page. Search for "Event Designators" or "\!\!" (escaping is mandatory):</li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>$ man bash</b><br /><b>/Event Designators<CR></b></span></blockquote>
<ul>
<li>While all this may seem hard to use practically, I recommend you just keep it in a dark corner of your mind. I promise you'll start to find places where you can apply it. They can be used to write commands quickly, especially several one-shot commands in a row. Just don't try forcing the use when it's not necessary. After all, they were created to save time.</li>
</ul>
<ul>
<li>If you ever got stabbed by not knowing how the bang works (like the examples I showed in the beginning), you can avoid getting stabbed again by escaping the bangs, either with a backslash or single quotes:</li>
</ul>
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ echo Bang\!</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">Bang! </span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>$ echo 'balrog:!you shall not pass:/noexistent:/bin/false' >> /etc/passwd</b></span></blockquote>
<ul>
<li>More of a side note, but to get that humongous file name, I discovered a new (string of) command(s) to get the longest line from a text.</li>
</ul>
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ find /sys/ | awk '{print length, $0;}' | sort -nr | head -n 1 </span></b></blockquote>
<ul>
<li>You can probably do the same thing (although it might be a little slower) using a while loop and another bash trick: while $variable (or ${variable}) will give you the contents of a variable, ${#variable} will give you the length of that content.<b><span style="font-family: "Courier New",Courier,monospace;"> </span></b></li>
</ul>
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ find /sys/ | while read i; do echo "${#i} $i" | sort -nr | head -n 1</span></b></blockquote>
<ul>
<li>Apparently, Jethro Tull's Too Old to Rock 'n' Roll, Too Old to Die generates pretty long file names (the title song is the second only to a really long cache file name on my file system).</li>
</ul>
Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-27675407770513925032013-02-06T23:11:00.001-02:002013-07-14T19:15:31.293-03:00An image is worth a thousand megabytes<i>Now we have all the components we need to run our Raspberry Pi, we can start preparing them to actually work together. The first thing we need to do is get the boot record, partitions and operating system on the SD card, so our little friend can boot and run a true system. Thankfully, some nice folks have already done the hard work, so we can jump many of the complex steps.</i><br />
<br />
<a href="http://iffalse.blogspot.com/2013/01/the-raspberry.html" target="_blank">Part I - The raspberry</a><br />
<b>Part II - An image is worth a thousand megabytes</b><br />
<a href="http://iffalse.blogspot.com/2013/07/a-post-mortem.html" target="_blank">Part III - A post-mortem</a><b><br /></b><br />
<br />
I'm going to use the Raspbian distribution, which is based on the famous Debian (GNU/)Linux distribution, and includes many of the standard software packages already prepared to run on the Pi. If you haven't already, grab Raspbian's image, which you can find on <a href="http://www.raspberrypi.org/downloads" target="_blank">Raspberry Pi Foundation's downloads web page</a>.<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ wget http://downloads.raspberrypi.org/images/raspbian/2012-12-16-wheezy-raspbian/2012-12-16-wheezy-raspbian.zip</span></b></blockquote>
Note that this is the current version as I write this post. Visit the page to get the most recent version. An optional but recommended step after you download the image is to check its <a href="http://en.wikipedia.org/wiki/Sha1" target="_blank">SHA-1 checksum</a>. That will ensure nothing has been corrupted during the download, which is always a good thing to know, specially when dealing with sensitive data such as disk images. To do that, you can use the sha1sum program:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ sha1sum 2012-12-16-wheezy-raspbian.zip</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">514974a5fcbbbea02151d79a715741c2159d4b0a 2012-12-16-wheezy-raspbian.zip</span></blockquote>
Compare the output with the hash on the SHA-1 section of the download page. Or, if you have the SHA-1 stored in a file:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>$ sha1sum -c 2012-12-16-wheezy-raspbian.zip.sha1</b> 2012-12-16-wheezy-raspbian.zip: OK</span></blockquote>
If you don't get the appropriate result, just download the file again and check it, until you get it right. Now we are going to copy the image to the SD card. Before you insert the SD card on your computer (you may need an external SD card reader, depending on the ports available on your machine), execute the command<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ ls /dev/sd*</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">/dev/sda /dev/sda2 /dev/sda4 /dev/sda6 /dev/sda8</span><br />
<span style="font-family: "Courier New",Courier,monospace;">/dev/sda1 /dev/sda3 /dev/sda5 /dev/sda7</span></blockquote>
That will list all the disks available. If you prefer a more friendly output, you can use<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ df -h</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;">Filesystem Size Used Avail Use% Mounted on<br />/dev/sda6 46G 30G 14G 69% /<br />udev 2.9G 4.0K 2.9G 1% /dev<br />tmpfs 1.2G 1.1M 1.2G 1% /run<br />none 5.0M 0 5.0M 0% /run/lock<br />none 2.9G 76K 2.9G 1% /run/shm<br />/dev/sda4 107G 50G 53G 49% /mnt/data</span></blockquote>
This will list the devices used by the mounted filesystems. Now you can insert the SD card and run the command again. You will notice that new entries will appear on the output. It will usually be <span style="font-family: "Courier New",Courier,monospace;">/dev/sdb*</span> entries, if you have only one disk. That is the device file for the card. If the card is automatically mounted, you can see the size on <span style="font-family: "Courier New",Courier,monospace;">df</span>'s output, so you can be sure that it is the right device.<br />
<br />
<b>A really important note here</b>: you need to get the correct device name. We are going to use a program that does raw block copying, which can destroy your data if you target the wrong device. The device mounted on root (<span style="font-family: "Courier New",Courier,monospace;">'/'</span>) is probably not the device you want, for example.<br />
<br />
Another important thing: if the card already has partitions, some extra device files may appear, such as <span style="font-family: "Courier New",Courier,monospace;">/dev/sdb1</span>, <span style="font-family: "Courier New",Courier,monospace;">/dev/sdb2</span> and so on. We need the device, not any partition, so be sure to use the device file with no number suffix. When you have the device name, make sure the device is unmounted so we can write data to it:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ umount /dev/sdb*</span></b></blockquote>
Then use good old <span style="font-family: "Courier New",Courier,monospace;">dd</span> to copy the image:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ dd if=./</span><span style="font-family: "Courier New",Courier,monospace;">2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M</span></b></blockquote>
A couple things deserve mention. It is always good to check (and re-check) the parameters, to make sure the image is the <span style="font-family: "Courier New",Courier,monospace;">if</span> (input file) value and the device is the <span style="font-family: "Courier New",Courier,monospace;">of</span> (output file) parameter. The <span style="font-family: "Courier New",Courier,monospace;">bs</span> parameter stands for block size, and is the size of each chunk of data copied at a time. Larger values make the process faster. Usually, 4mb is a reasonable Following UNIX's philosophy, <span style="font-family: "Courier New",Courier,monospace;">dd</span> doesn't output anything while it is working. Since it can take quite some time to finish, you may think it's froze. But don't worry, it hasn't. If you want to be absolutely sure, send the program a signal and it will output the progress:<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ ps aux | grep dd</span></b><br />
<span style="font-family: "Courier New",Courier,monospace;"> 4821 pts/2 R+ 0:33 dd if=./2012-12-16-wheezy-raspbian.zip of=/dev/sdb bs=4M</span><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ kill -USR1 4821</span><span style="font-family: "Courier New",Courier,monospace;"><br /></span></b></blockquote>
When it is done, run<br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">$ sync</span></b><br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ </span></b></blockquote>
to make sure everything is properly written and remove the card. Now insert the card on the Raspberry, plug the power supply and, if everything worked, you will get a nice show of lights as it boots. You can plug the HDMI to a monitor and see the boot messages. Here's a (somewhat light-impaired) gallery:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjof-1QD02UprOlcqsgyWiap8k3Te_uB5rbpieZvdz9FWmEDd-j-vjHq3bgee_nRhlBtt-j_SQgv6tQMv42vZ1rYzZrGLcNnK2ETgtkmDeQk8sHNztAdHtbYszCkQGNT5akIwfTIL-B6p8/s1600/2013-02-06_22-46-11_667.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjof-1QD02UprOlcqsgyWiap8k3Te_uB5rbpieZvdz9FWmEDd-j-vjHq3bgee_nRhlBtt-j_SQgv6tQMv42vZ1rYzZrGLcNnK2ETgtkmDeQk8sHNztAdHtbYszCkQGNT5akIwfTIL-B6p8/s200/2013-02-06_22-46-11_667.jpg" width="200" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyson9LyhtMLIpI7qy_D8_94T_4OlmB-2KS5biVTc0mPJygLWu8oOwTbLSWKXpJ8rrVQsgLTjVOJfrwUPP4rdyMDMKqdX5LdedCL6kx2oOwSfGZs3UtlYl6LO36kf4F95AGgDcMRginT8/s1600/2013-02-06_22-46-01_692.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyson9LyhtMLIpI7qy_D8_94T_4OlmB-2KS5biVTc0mPJygLWu8oOwTbLSWKXpJ8rrVQsgLTjVOJfrwUPP4rdyMDMKqdX5LdedCL6kx2oOwSfGZs3UtlYl6LO36kf4F95AGgDcMRginT8/s200/2013-02-06_22-46-01_692.jpg" width="200" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiClvEG9a3RYPkQb2QIKlgVxsHyPa3td3LJlIwQXRgTZFPrzQ677sTU3a4iMjxIh50XqZwnkXrWkaJrIKIbZ9WCSkLNMEJd3QPPlrGsh8nHAtSkD3gqvsw3V6p159JGVuH0aBg9z6_Vl4/s1600/2013-01-23_08-13-55_890.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="123" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiClvEG9a3RYPkQb2QIKlgVxsHyPa3td3LJlIwQXRgTZFPrzQ677sTU3a4iMjxIh50XqZwnkXrWkaJrIKIbZ9WCSkLNMEJd3QPPlrGsh8nHAtSkD3gqvsw3V6p159JGVuH0aBg9z6_Vl4/s200/2013-01-23_08-13-55_890.jpg" width="200" /></a></div>
<br />
<div style="text-align: center;">
</div>
I know I promised we'd get it to work! Well, if you (unlike me) think this isn't actual work, check the next posts, where we will put the Pi to good use.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-29124651526870746022013-01-23T21:29:00.000-02:002013-07-14T19:17:29.616-03:00The raspberry<b>Part I - The raspberry</b><br />
<a href="http://iffalse.blogspot.com/2013/02/an-image-is-worth-thousand-megabytes.html" target="_blank">Part II - An image is worth a thousand megabytes</a><br />
<a href="http://iffalse.blogspot.com/2013/07/a-post-mortem.html" target="_blank">Part III - A post-mortem</a><br />
<br />
My late Christmas gift (from me to myself) arrived on Monday, and it is a little Raspberry Pi. If you're not familiar with it (and you should be), it's a tiny board which contains everything a computer needs to run (or, as I explained to my perplexed mother, "<i>it has everything a computer need to be a computer"</i>).<br />
<br />
I won't go over the process of acquiring one, which is not as simple as it may seem, if you live in Brazil. I'll skip to the part just after you console yourself for paying the rough equivalent of 85 dollars, or 242% the original price (of course this includes shipping, but it still is absurd).<br />
<br />
So, if you survive that step, you'll get this pretty box:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiM_zscBulwT7pwGYi7F0Ilf5UN3HKlcGWxrF4cN1nhiT_df6_JcqUy7HdHjOIU_TGNxCClJ5XJ4txQWg-bvS-HyDYt66EaGx4Gv3nYfXRpJQpWKCS7PfoFM3N0H8uoMhv94baXoewizig/s1600/2013-01-23_19-46-53_581.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Raspberry Pi in-a-box" border="0" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiM_zscBulwT7pwGYi7F0Ilf5UN3HKlcGWxrF4cN1nhiT_df6_JcqUy7HdHjOIU_TGNxCClJ5XJ4txQWg-bvS-HyDYt66EaGx4Gv3nYfXRpJQpWKCS7PfoFM3N0H8uoMhv94baXoewizig/s400/2013-01-23_19-46-53_581.jpg" title="Raspberry Pi in-a-box" width="400" /></a></div>
<br />
And inside it...<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7xinJnJQ0We0Rm2StHpx9oy6yOWTF85GNUzGVVQauRQJ9oz9IosjCYpn-9Q4du9Uzx_3FdYq8yXMcxEq6ldZTlYlSP2Jqom7F0QkVC3DC2ofrtQdEAjlS5auP-cpOoU2E0935ZgFDSyc/s1600/2013-01-23_19-49-44_482.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="The Raspberry Pi unboxed" border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg7xinJnJQ0We0Rm2StHpx9oy6yOWTF85GNUzGVVQauRQJ9oz9IosjCYpn-9Q4du9Uzx_3FdYq8yXMcxEq6ldZTlYlSP2Jqom7F0QkVC3DC2ofrtQdEAjlS5auP-cpOoU2E0935ZgFDSyc/s400/2013-01-23_19-49-44_482.jpg" title="The Raspberry Pi unboxed" width="400" /></a></div>
<br />
<br />
<span style="font-size: large;">Prelude</span><br />
<br />
If you're reading this before actually buying it, or before it arrives (and that is a smart thing to do), you'll probably want to get everything prepared so you can start working as soon as you get your hands on it. Here are the things you absolutely need to get started:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaNnSf9qD5ISBjci-FcYYLZ56LUqfBYjT_vjkXhZPTvbdX447e8GabwI87V5CpN_5tTaMM9pQerw0-PcTfvvFMtZP74Hcmjf9y8qhV1uwUVQWy8nt_64W7nHc5niE8Tg_FBu9BQ72QHPQ/s1600/2013-01-23_19-51-43_745.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaNnSf9qD5ISBjci-FcYYLZ56LUqfBYjT_vjkXhZPTvbdX447e8GabwI87V5CpN_5tTaMM9pQerw0-PcTfvvFMtZP74Hcmjf9y8qhV1uwUVQWy8nt_64W7nHc5niE8Tg_FBu9BQ72QHPQ/s200/2013-01-23_19-51-43_745.jpg" width="200" /></a></div>
<ul>
<li><b>SD card</b>: this is where all the files are stored. If you'll be using <a href="http://www.raspbian.org/">Raspbian</a> (and if you don't know what that is, it probably is what you'll be using), the card must be at least 2GB large. The size really depends on what you plan to do with it, but 8 GB and 16 GB suit most of the needs. If you don't want to buy it right away, you can use any SD card you have laying around, like those used in cameras.</li>
</ul>
<div style="text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_afF1tfh6QQB0prXV-R57cSHE4_V0dZ9QLilX9Id4sICSTOfvWHRIbmNyQ9pFdSjj0paXiLmlvzfxcYGDvj7K_jmqnZwueJMYLpmoshajQjRIZoAZqXQ7VUW_BreH4joLpULJZw0LPDg/s1600/2013-01-23_19-53-52_815.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_afF1tfh6QQB0prXV-R57cSHE4_V0dZ9QLilX9Id4sICSTOfvWHRIbmNyQ9pFdSjj0paXiLmlvzfxcYGDvj7K_jmqnZwueJMYLpmoshajQjRIZoAZqXQ7VUW_BreH4joLpULJZw0LPDg/s200/2013-01-23_19-53-52_815.jpg" width="200" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieD1TZpX4BkywQNpvRj5vVIgFSj698AfuZ018wuiZwypFwXkhyphenhyphendCqkHjTLBLzs5390InJu5fJXZQQL81DkcACvD0XHrRR1vVFhWzz8eBvJ7FYBXFAVF-GnFmdwDX2aCe3tHEqP3kq1yFQ/s1600/2013-01-23_20-01-31_202_2.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieD1TZpX4BkywQNpvRj5vVIgFSj698AfuZ018wuiZwypFwXkhyphenhyphendCqkHjTLBLzs5390InJu5fJXZQQL81DkcACvD0XHrRR1vVFhWzz8eBvJ7FYBXFAVF-GnFmdwDX2aCe3tHEqP3kq1yFQ/s200/2013-01-23_20-01-31_202_2.jpg" width="200" /></a> <span id="goog_952676621"></span><span id="goog_952676622"></span></div>
<ul>
<li><b>Power supply</b>: this is the most delicate component. If you look at the <a href="http://www.raspberrypi.org/wp-content/uploads/2012/12/quick-start-guide-v1.1.pdf" target="_blank">Quick Start Guide</a>, the Pi requires a "[...] good quality, micro USB power supply that can provide at least 700mA at 5V.". Depending on where you live, this may be easier or harder to find (apparently, I live in the hardest place on Earth). Remember to get the voltage right, but what's important is that 700mA is the minimum required for the Raspberry. If you want to plug other devices that need power supply to it, you're probably going to need more power. I was really lucky that my Motorola cell phone charger meets almost exactly the requirements (check the picture on the right). </li>
</ul>
<div style="text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkfdcB0xlJaUSz3jrD9VLWju_1pUzgV4aXU343OvE-z5UZmoJ-FJgkNUq8WQGnUG1Harefu9xh_8nMPm-L5eRFxaRl4rAfVeI165k5Uk9uuXwJmP281R4_6JlnX8oXeFPm9caJ6VE5Yvc/s1600/2013-01-23_20-07-46_173.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkfdcB0xlJaUSz3jrD9VLWju_1pUzgV4aXU343OvE-z5UZmoJ-FJgkNUq8WQGnUG1Harefu9xh_8nMPm-L5eRFxaRl4rAfVeI165k5Uk9uuXwJmP281R4_6JlnX8oXeFPm9caJ6VE5Yvc/s200/2013-01-23_20-07-46_173.jpg" width="200" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42n6Un1m1nTw1MnXDya3398c9XW2a_LGh5pAKFGE7sAnGJtyn1UvDKGTXQrMKL5Bwmk4oK-zZO3FXMJlVpzzm7hNc__JRrgm3d58yEnyyjuB2U9cFPVcyQBwso0MP2OEQqMpCm1l4TPY/s1600/2013-01-23_20-14-27_348.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh42n6Un1m1nTw1MnXDya3398c9XW2a_LGh5pAKFGE7sAnGJtyn1UvDKGTXQrMKL5Bwmk4oK-zZO3FXMJlVpzzm7hNc__JRrgm3d58yEnyyjuB2U9cFPVcyQBwso0MP2OEQqMpCm1l4TPY/s200/2013-01-23_20-14-27_348.jpg" width="200" /></a></div>
<ul>
<li><b>HDMI (or RCA) cable</b>: in order to plug it to a monitor or TV, you'll need a HDMI cable. These can be found on any decent electronics store. Alternatively, you can use the RCA port to connect to older devices.</li>
</ul>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj90CxFXcNpSGQxMdUdB1qbPzOd6eVJxBqiaYo0ZtODBwxaH3TM-df9wIQ-q9CcCSN89KZy1LQwZJLB09QZYW-MzuQs031TPVPJGgabV7R5-5nGXGvP0N7q07OT98oeXhM5HvK6TGzhgbY/s1600/2013-01-23_20-16-57_833.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="112" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj90CxFXcNpSGQxMdUdB1qbPzOd6eVJxBqiaYo0ZtODBwxaH3TM-df9wIQ-q9CcCSN89KZy1LQwZJLB09QZYW-MzuQs031TPVPJGgabV7R5-5nGXGvP0N7q07OT98oeXhM5HvK6TGzhgbY/s200/2013-01-23_20-16-57_833.jpg" width="200" /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOSXA9e1eCrHTouTFv_eSSRpFfcxnMsK1Xi_ClFlQqemJzr23uBpyYJv2dKzVdGL80_3E-siFBY3fjyEU8nlycN6w0ioF-Qj0PbxfR3LEUpN_nsWgJmriToOtdLU3f3s3B2mm4ixAO-_0/s1600/2013-01-23_20-18-33_434.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="111" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOSXA9e1eCrHTouTFv_eSSRpFfcxnMsK1Xi_ClFlQqemJzr23uBpyYJv2dKzVdGL80_3E-siFBY3fjyEU8nlycN6w0ioF-Qj0PbxfR3LEUpN_nsWgJmriToOtdLU3f3s3B2mm4ixAO-_0/s200/2013-01-23_20-18-33_434.jpg" width="200" /></a></div>
<div style="text-align: center;">
</div>
<ul>
<li><b>USB keyboard</b>: any keyboard with a USB cable will do. A mouse is not strictly necessary, but if you like, you can use any one with a USB cable.</li>
</ul>
So now we have all the components, we need to prepare our SD card. Check the next post for instructions on that, but meanwhile you can go ahead and <a href="http://www.raspberrypi.org/downloads/" target="_blank">download the Raspbian image from the Raspberry Pi foundation</a>, which is the OS we'll use.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-49788896566777075852012-12-21T08:48:00.000-02:002014-05-02T13:15:47.745-03:00The super user is more powerful than god<span style="font-family: "Courier New",Courier,monospace;">$ useradd god</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ mkdir a</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ chmod 000 a</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ su god -c ls a</span><br />
<span style="font-family: "Courier New",Courier,monospace;">ls: cannot open directory a: Permission denied</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo ls a</span><br />
<span style="font-family: "Courier New",Courier,monospace;">$</span><br />
<br />
And it can even tell god what to do:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">$ sudo su god -c "sed -i 's/\(-DWINDOWS\|-DMAC\)//' Makefile; make -B world"</span><br />
<br />
Happy winter solstice.<span style="font-family: "Courier New",Courier,monospace;"> </span>Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-33657369195632007452012-11-04T20:24:00.001-02:002016-12-10T07:17:22.829-02:00Hiding text in LaTeXWhen writing my dissertation, I like to leave some text that I won't use at<br />
that particular moment but know I will need in the (near) future. To remember<br />
to correct it later, I surrounded it with the (pseudo-randomly chosen) \ldots<br />
command, like this:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">This is the actual text.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">\dots</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">This is some text I'll need later.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">\ldots</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">This is more of the actual text.</span></blockquote>
That solved my problem, I could just grep the files and find the parts that<br />
needed refactoring. The problem is they ended up on the PDF, which is not the<br />
end of the world, but it disrupted the reading and didn't look so good. I<br />
finally decided to google the problem, and found a nice solution, using the if<br />
command:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">This is the actual text</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">\iffalse</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">This is some text I'll need latex.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">\fi</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">This is more of the actual text.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"></span></blockquote>
Now I can still grep for \iffalse, and it doesn't pollute the PDF. And it even<br />
advertises my blog. Yay!Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com1tag:blogger.com,1999:blog-4492442420641722674.post-4486241691514562192012-10-28T01:41:00.000-02:002012-10-28T01:41:57.106-02:00Problem when adding SSH key on github<i></i><br />
<i>Recently, I had I problem with github. I couldn't add SSH keys anymore, the web page would just hang and present me a "Connection reset" error after a few (annoying) minutes. I struggled for a week, but have finally found a solution, which is not the best, but at least I can access github from my machine once again.</i><br />
<br />
It all started two weeks ago, when I ran an everyday <span style="font-family: "Courier New",Courier,monospace;">git push</span>, and got a little surprise. It just sat there forever, and I had to SIGTERM it every time. What I did after was try to reset my key on github. I went to the SSH page and deleted my key, created a new one and tried to add it.<br />
<br />
Yes I know it was stupid. The right way to do it would be:<br />
<ul>
<li>back up my key</li>
<li>create a new one</li>
<li>add it</li>
<li>make absolutely sure it works </li>
<li>remove the old one in github and on the machine</li>
</ul>
But I didn't, for one simple reason: adding keys has been such a natural operation, I didn't even think about it. Replacing a key (which I did a lot of times when I was learning how public-key encryption works) only takes a few seconds, and I never had to worry about it. Until now.<br />
<br />
<br />
<h2>
Haystacking</h2>
I am no ssh expert. So I turned to github's support. And I have to say, the response was really good, fast and to the point. Sadly, it didn't solve the problem. Here is the response:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">Hi Bruno,</span></blockquote>
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">
What size SSH key are you using? We have a known bug where the page will give a 500 error if the SSH key is too large.</span>
<span style="font-family: "Courier New",Courier,monospace;"><br />
I would recommend using the default size key for now, and hopefully we will have this bug fixed soon.</span>
<span style="font-family: "Courier New",Courier,monospace;"><br />
Sorry for the trouble.</span>
<span style="font-family: "Courier New",Courier,monospace;"><br />
Cheers,<br />
Steve</span></blockquote>
<br />
It turns out the problem wasn't the key size, but you can see Steve gave a really polite response. Anyway, I started trying everything to make it work, but in a week I just couldn't do it. Then, yesterday, I decided to give the developer API a try. I didn't have much hope that it would work, because I had tried a lot of different things and because I thought the problem was exactly there. But I was wrong again.<br />
<br />
With just a few commands, I was able to add a ssh key to my account and finally access my repositories. And, in the process, I discovered a handy command line utility, so I guess it wasn't all that bad.<br />
<br />
<br />
<h2>
curl (a.k.a. cURL)</h2>
curl is a program you can use to fetch web pages just as your browser would do. It is present on most distributions, so you can start right away and run<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ <b>curl iffalse.blogspot.com</b></span></blockquote>
And you'll get the HTML output of the page. You can use that to access the github API. To list all the keys of an account, you'd use<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ <b>curl -u bbguimaraes https://api.github.com/user/keys</b></span></blockquote>
Use the <span style="font-family: "Courier New",Courier,monospace;">-u</span> option to set the username. You will be prompted for the password. The switch accepts the form <span style="font-family: "Courier New",Courier,monospace;">-u user:password</span>, but that just isn't very nice. You will get back a json output, listing all the keys registered for your account. There are several functions on the API, you can take a look at them on the <a href="http://developer.github.com/">docs page</a>.<br />
<br />
But the one that did it for me was the "<a href="http://developer.github.com/v3/users/keys/#create-a-public-key">create a puclib key</a>". A simple post<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;">$ <b>curl -u bbguimaraes -d '{"title": "[...]", "key": "ssh-rsa AAA[...]"}' https://api.github.com/user/keys</b></span></blockquote>
and, after a week of awful manual syncing of code, I was finally able to use my beloved github repositories as I always had. I hope this text can save you some time if you ever happen to have the same problem.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-12019770604162110082012-10-22T23:25:00.001-02:002012-10-28T01:42:12.383-02:00perl is beautiful - part 3<i>In the last post on the subject, I'll show a couple more perl tricks I use everyday. I'll touch on some not-so-basic regex concepts here, so feel free to take a look <a href="http://perldoc.perl.org/perlre.html">here</a> if you don't know what some of them mean.</i><br />
<br />
One thing you can do with the regex operators is using them in list context. Every operation in perl is done in a <a href="http://docstore.mik.ua/orelly/perl4/prog/ch02_07.htm">context</a>. When you assign to a variable, the expression is evaluated in scalar context. When you assign to an array, you get list context:<span style="font-family: "Courier New",Courier,monospace;"><b> </b></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> @my_array = (2, 3, 4, 5);</span> </blockquote>
The '@' tells perl that your variable will be an array, and the rhs is thus executed in list context. Try this:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> @my_array = (2, 3, 4, 5);</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> $my_var = @my_array;</span></blockquote>
Or, my personal favorite:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> @my_array = 2..5;</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> $my_var = @my_array;</span></blockquote>
Both do the same thing. When you use an array on the rhs of a scalar assignment, you get the length of that array, which is what's happening here. The regex operators also work this way:<span style="font-family: "Courier New",Courier,monospace;"><b> </b></span><br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> $scalar = <b>m/</b><i>my \$\w+</i><b>/</b>;</span> </blockquote>
will put a boolean value on $scalar if a match was or wasn't found (on $_, remember). This will (roughly) find all "use" directives on perl code. Since we are in scalar context, the result will be a boolean, which will be true if any match occurs. But try this:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> @array = <b>m/</b></span><span style="font-family: "Courier New",Courier,monospace;"><i>my \$(\w+)</i><b>/g</b>;</span></blockquote>
This will create an array with all the matches found. There are some new things here which need a better explanation. First, the regex modifier g (from global) will make the regex be applied repeatedly, instead of stopping on the first result.<br />
<br />
Since we are on a list context, the result is no longer a boolean, but a list of all the matches, here meaning the values captured on $1, $2, and so on. The output for a sample file:<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>use</b> warnings;<br /><b>use</b> strict;<br /><br /><b>local</b> $/;<br />$_ = <>;<br />$, = <i>"\n"</i>;<br />$\ = <i>"\n"</i>;<br /><br /><b>my</b> $scalar = <b>m/</b><i>use (\w+);</i><b>/</b>;<br /><b>my</b> @array = <b>m/</b><i>use (\w+);</i><b>/g</b>;<br /><br /><b>print</b> $scalar;<br /><b>print</b> @array;<br /><br /><i># Output</i><br />1<br />warnings<br />strict</span></blockquote>
Here we can see, when we incept the code through itself, that the variable gets a value of true, while the array is populated with all the results found. Just to show you another nice feature: if we add some parenthesis around "use":<br />
<blockquote class="tr_bq">
<span style="font-family: "Courier New",Courier,monospace;"><b>my</b> @array = <b>m/</b><i>(use) (\w+);</i><b>/g</b>;</span></blockquote>
we get four elements on the array. That's because, for each "pass", all values from ($1, $2, $3, ...) are stored.<br />
<br />
Well, I could go on and on writing about all the cool things you can do with perl, but I'm going to stop here. If this short introduction did leave you interested to learn more, there are several sources on-line. Two I use constantly are <a href="http://perldoc.perl.org/">perldoc</a> and <a href="http://docstore.mik.ua/orelly/perl4/prog/index.htm">the web version of Programming Perl</a>. If you want to learn more about regular expressions, I highly recommend O'Reilly's <a href="http://shop.oreilly.com/product/9780596528126.do">Mastering Regular Expressions</a> (but please, don't buy the Brazillian Portuguese translation, just don't).<br />
<br />
And as a last note, if you are a vim user, here's something really useful. You can select some text (using v, V or ctrl-v) and press ':'. You will automatically get a filter for the selected text. From there, you can type 'w !' (note the space), which will write the selected text to the standard input of the command. Combining all this with what we learned, we can, for example, use the regex to find python functions, on a single vim command:<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<blockquote class="tr_bq">
<b><span style="font-family: "Courier New",Courier,monospace;">:'<,'>w !perl -ne 'print if m/^\s*def \w+/'</span></b></blockquote>
And get a listing of the functions inside the selected text. Or you can do some bizarre things, like sending the output to a file, or changing the name of the functions, changing indentation... Your sanity is the limit.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-78697679714879773702012-10-22T10:01:00.001-02:002012-10-28T01:42:30.538-02:00Low-level language"Tudo é céu, tudo é mar; torvo negrume sobre as cabeças borrascoso pesa, e horrenda espessa treva enoita as ondas." MARÃO, Públio Virgílio. <i>Eneida</i>.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-11790137142401699512012-10-18T07:53:00.001-03:002019-07-20T18:37:19.105-03:00perl is beautiful - part 2<div class="tr_bq">
<i>We've seen how perl can be used, but you may be wondering, </i>so what<i>? Fasten your seatbelt, for fun begins now.</i></div>
<br />
<h2>
Interpreter switches</h2>
The single most useful tool I'm going to show you. The interpreter accepts <a href="http://perldoc.perl.org/perlrun.html">many parameters</a>, which alter the way it runs, but most importantly, they can alter the way the code is executed. First, though, I'll show you the -e switch, which will be used on all the other examples.<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -e 'print "hello\n";'</b></span></blockquote>
The switch tells the interpreter that, instead reading from <i>stdin</i> or from a file, the program will be contained inside the next argument. Note that, by perl's definition, you don't need to use a semicolon after the last (here also the only) command. But I've included it to avoid confusion on the quotes. With this switch, it will be easier to show you the others.<br />
<br />
Let's start with -n, which will <i>surround</i> your code with the following construct:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><b>while</b>(<>) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <i># your code</i></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span></blockquote>
Try running our previous example but adding this switch. It will start listening on stdin, and after every line it will print the string 'hello' (I call it the nice program). One thing to note is that the -e switch and the program string must be contiguous. You can run the program like the first three lines below, but the fourth will give you an error:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -ne 'print "hello\n"'</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -n -e 'print "hello\n"'</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -e 'print "hello\n"' -n</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -e -n 'print "hello\n"' # This is wrong!</b></span></blockquote>
But what really happened when we ran that program? If you remember <a href="http://iffalse.blogspot.com/2012/10/perl-is-beatiful.html">the last post</a>, <> is the input operator. So, "<b>while</b>(<>)" means "read lines from the input until it ends". And if you have really good memory (or are already tainted by the perl way of thinking), you'll remember that the input read will go to the variable $_. We can test it with our reimplementation of cat:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -ne 'print'</b></span></blockquote>
See what I did there? I tested your memory again.$_ is used as the default argument to many functions in perl, including print. This allows you to say just "<b>print</b>" here, instead of "<b>print</b> $_". And this code is such a common occurrence that perl already has another switch to do just that: -p.<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -pe ''</b></span></blockquote>
Running this, you can see that it is exactly the same as the previous. Just like -n, it surrounds your code with more code, but it adds a little something to what -n adds:<br />
<blockquote>
<span style="font-family: "courier new" , "courier" , monospace;"><b>while</b>(<>) {<br /> <i># your code</i><br /> <b>print </b><br />}</span></blockquote>
<br />
<h2>
Regular expressions revisited </h2>
Now, this is already fun. We can make a program that does anything to each line of input (like using ssh -Y, a list of swear words and notify-send on your coworker's computer). But the real power of what I'm showing you comes from combining these techniques with perl's <a href="http://en.wikipedia.org/wiki/MongoDB">humongous</a> power. Check this:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -ne 'print if m/=D/'</b></span></blockquote>
This will print every happy line on a file. Remember that the default parameter for the <b>m//</b> operator is also $_ (isn't it beautiful?). If you followed me all the way here, you can now start applying it, just by remembering this simple command line. Quick example I use every day: find all python functions on a file.<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -ne 'print if m/^\s*def /</b></span>'</blockquote>
From here, you can go anywhere. The nice thing is now you have the power of a real programming language, and can do all the bizarre things perl can do (and it can do some really bizarre things).One thing that took me a little time to get used to was the "boilerplate" part of the command line. But after I used it a few times, it's now natural, I don't even have to think about it. And it took only a couple of seconds to google "perl command line" anyway (at some point, I only had to type "perl" on firefox's address bar and it would show me the page right away).<br />
<br />
Coming next: on the last post on this (unexpectedly long) topic, I'll show you some (kind of ) advanced perl and regexp techniques.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-52745344480763896152012-10-01T23:18:00.001-03:002019-07-20T18:37:07.643-03:00perl is beautiful<i>Writing the first post on a blog about programming is fairly easy. Now, the second one needed a little thought to come up with. I decided to show a trick I use almost daily, be it at work or on personal projects. And that is <a href="http://en.wikipedia.org/wiki/Perl">perl</a>. Yes, the ancient language from 1987.</i><br />
<br />
If you're familiar with grep (and if you're not, stop reading this right now and get familiarized), you already know the enormous power of a general purpose stream filter. What I want to show you in this post is how you can augment that power using simple (aren't them all?) features from perl, both the language and the homonymous interpreter.<br />
<br />
<h2>
The interpreter</h2>
I'll start with the interpreter just to get you started on how to run simple programs. No perl experience is required, since the constructs used here are really simple. So, let's start:<br />
<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -v</b></span></blockquote>
<blockquote>
<span style="font-family: "courier new" , "courier" , monospace;">This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi<br />(with 53 registered patches, see perl -V for more detail)<br /><br />Copyright 1987-2011, Larry Wall<br /><br />Perl may be copied only under the terms of either the Artistic License or the<br />GNU General Public License, which may be found in the Perl 5 source kit.<br /><br />Complete documentation for Perl, including FAQ lists, should be found on<br />this system using "man perl" or "perldoc perl". If you have access to the<br />Internet, point your browser at http://www.perl.org/, the Perl Home Page.</span></blockquote>
If everything is setup right, you get a nice message along with the version number. If you get something like<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">perl: command not found</span></blockquote>
you probably need to install the perl interpreter or set your PATH right. Installing it may be as simple as<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>sudo apt-get install perl</b></span></blockquote>
depending on your system. But your platform's documentation is probably a much better resource on how to install programs. Once we have the interpreter, we can start fiddling with it:<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl</b></span><b><span style="font-family: "courier new" , "courier" , monospace;"> </span></b><br />
<b><span style="font-family: "courier new" , "courier" , monospace;">print "Hello world!\n";</span></b></blockquote>
<br />
Press Ctrl-D and you should get the (in)famous message on your screen. For longer scripts, you can save it to a file and pass the file name as an argument to the interpreter. But my personal favorite is the -e switch. It means that the string following the switch is actually the program to be run.<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$ <b>perl -e 'print "Hello world!\n"; </b></span></blockquote>
You won't have this problem here, but take care when mixing the quotes used inside the program from the quotes used to pass the command to the shell. You may need to escape some of them.<br />
<br />
<h2>
perl</h2>
Let's start building some more interesting examples, exploring one of the strongest side of perl: its support for <a href="http://en.wikipedia.org/wiki/Regular_expression">regular expressions</a>. We create a file to contain our script:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<b><span style="font-family: "courier new" , "courier" , monospace;">#!/usr/bin/perl</span></b><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>use strict</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>use warnings</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><b>my</b> $str = <>;;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>print</b> $str <b>if</b> $str =~ <b>m/</b><i>g w</i><b>/</b>;</span></blockquote>
<br />
I'll explain what each line does. <a href="http://en.wikipedia.org/wiki/Shebang_%28Unix%29">You should know what line 1 does</a>. Line 3 and 4 turn on useful warnings (if you don't set warnings to the
highest level possible on your compiler/interpreter, shame on you). On line 6, we create a variable called <i>str</i> (the <i>$</i> character is used to indicate simple variables, while the keyword <b>my</b> defines it as a local variable), and initialize it using the <a href="http://www.stat.berkeley.edu/~spector/extension/perl/notes/node46.html">diamond operator</a>. Long story short, this operator reads from the standard input (as long as you don't pass any parameters when executing the program, more on this later).<br />
<br />
Line 7 deserves a special paragraph. Here I'm using the inverted <b>if</b> syntax (which is a great feature that most languages lack), but don't get fooled by it. It could be written as<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><b>if</b>($str =~ <b>m/</b>g w<b>/</b>) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>print</b> $str;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span></blockquote>
but I find the inverted syntax much clearer and easier to read. The =~ operator is used to apply regular expressions. It is a binary operator, where the left-hand side is a string or a variable containing a string, and the right-hand side is a regex operator (in this case, the <b>m//</b> operator). The syntax to match a string against a regular expression is<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><i>string</i> =~ <b>m/</b><i>regex</i><b>/</b>;</span></blockquote>
Using some of <a href="http://www.perlmonks.org/?node_id=431511">perl's magic</a>, we could rewrite our example as<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;">$_ = <>;<br /><b>print</b> <b>if</b> <b>m/</b><i>g w</i><b>/</b>;</span></blockquote>
Now things start to get interesting! I decided to go light and present the omnipotent $_ variable explicitly. $_ is a special (a.k.a. magic) variable in perl, which is used as the default value of many functions. Two examples here are print and the match operator.<br />
<br />
One last thing before I delegate the rest of this topic to the next post: the diamond operator can be used with a loop to read all the lines from the input:<br />
<blockquote class="tr_bq">
<span style="font-family: "courier new" , "courier" , monospace;"><b>while</b>(<>) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>print</b>;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">}</span></blockquote>
Here,<> causes one line from the input to be read and stored on $_. Then, inside the body of the loop, print is called without arguments, causing it to use our old friend $_. Input, when using the diamond operator, can be one of two things. If the program is called with no arguments, input is <i>stdin</i>. If file names are passed as arguments, the input is the content of these files, read sequentially from first to last. The loop keeps reading until the input ends, which means reading until a EOF character on <i>stdin</i> or reading all the files. Congratulations! You have just implemented <i>cat</i>.<br />
<br />
Coming next: <i>actually</i> increasing productivity.Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com0tag:blogger.com,1999:blog-4492442420641722674.post-25512335307180351932012-10-01T00:13:00.001-03:002012-10-01T23:19:23.074-03:0099 Bottles of Beer<span style="font-family: "Courier New",Courier,monospace;">$ vim hello.asm</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <i>.section</i> <i>.rodata</i><br /><b>string:</b><br /> <i>.ascii</i> <i>"Hello, World!\n\0"</i><br /><b>length:</b><br /> <i>.quad</i> . -string <i>#Dot = 'here'</i><br /><br /> <i>.section</i> <i>.text</i><br /> <i>.globl</i> _start <i>#Make entry point visible to linker</i><br /><b>_start:</b><br /> <b>movq</b> $4, %rax <i>#4=write</i><br /> <b>movq</b> $1, %rbx <i>#1=stdout</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <b>movq</b> $string, %rcx<br /> <b>movq</b> length, %rdx<br /> <b>int</b> $0x80 <i>#Call Operating System</i><br /> <b>movq</b> %rax, %rbx <i>#Make program return syscall exit status</i><br /> <b>movq</b> $1, %rax <i>#1=exit</i><br /> <b>int</b> $0x80 <i>#Call System Again</i></span><br />
<span style="font-family: "Courier New",Courier,monospace;">:!as % -o %.o</span><br />
<span style="font-family: "Courier New",Courier,monospace;">:!ld %.o </span><br />
<span style="font-family: "Courier New",Courier,monospace;">:!./a.out</span>Bruno Barcarol Guimarãeshttp://www.blogger.com/profile/16230944209541773761noreply@blogger.com1