This blog is mainly a reminder for myself for the various possibilities to check my shell scripts for portability, but maybe it’s helpful for some other people, too.

First, why bother? Well, while bash is the default /bin/sh shell on many rpm-based Linux distributions (so it’s also the default shell on the systems I’m developing with and thus referring to here), it’s often not the case on other Linux distributions like Debian or Alpine, and it’s certainly not the case on non-Linux systems like the various *BSD flavors or illumos based installations.

Test your scripts with other shells

The most obvious suggestion is, of course, to run your script with a different shell than bash to see whether it works as expected.

Using dash

The probably most important thing to check is whether your script works with dash. dash is the default /bin/sh shell on most Debian-based distributions, so if you want to make sure that your script also works on such systems, this is the bare minimum that you should check. The basic idea of dash is to run scripts as far as possible, without adding bloat to the shell. Therefore the shell is restricted to a minimum with regards to the syntax that it understands, and with regards to the user interface, e.g. the interactive shell prompt is way less comfortable compared with shells like bash.

Since dash is also available in Fedora and in RHEL via EPEL, its installation is as easy as typing something like:

 sudo dnf install dash

Thus checking your scripts with dash is almost no additional effort and thus a very good place to start.

Using posh

posh stands for “Policy-compliant Ordinary SHell” – it’s another shell that has been developed within the Debian project to check shell scripts for POSIX compliance. Unlike dash, the syntax that this shell understands is really restricted to the bare minimum set that the POSIX standard suggests for shells, so if your script works with posh, you can be pretty sure that it is portable to most POSIX-compliant shells indeed.

Unfortunately, I haven’t seen a pre-compiled binary of posh for Fedora or RHEL yet, and I haven’t spotted a dedicated website for this shell either, so the installation is a little bit more complicated compared to dash. The best thing you can do on a non-Debian based system is to download the tar.xz source package from https://packages.debian.org/sid/posh and compile it on your own:

 wget http://deb.debian.org/debian/pool/main/p/posh/posh_0.14.1.tar.xz
 tar -xaf posh_0.14.1.tar.xz
 cd posh-0.14.1/
 autoreconf -i
 ./configure
 make
 ./posh ~/script-to-test.sh

Using virtual machines

Of course you can also check your scripts on other systems using virtual machines, e.g. on guest installations with FreeBSD, NetBSD, OpenBSD or one of the illumos distributions. But since this is quite some additional effort (e.g. you have to boot a guest and make your script available to it), I normally skip this step – testing with dash and posh catches most of the issues already anyway.

Test your scripts with shell checkers

There are indeed programs that help you to check the syntax of your shell scripts. The two I’ve been using so far are checkbashism and ShellCheck:

Using checkbashism

checkbashism is a Perl script, maintained again by the Debian people to check for portability issues that occur when a shell script has been only written with bash in mind. It is part of the devscripts package in Debian. Fortunately, the script is also available in Fedora by installing the so-called devscripts-checkbashisms package (which can also be used on RHEL by way). checkbashism focuses on the syntax constructs that are typically only available in bash, so this is a good and easy way to check your scripts on distributions where /bin/sh is bash by default.

Using ShellCheck

ShellCheck is another static shell script analyzer tool which is available for most distributions or can ben installed via the instructions provided on its GitHub page. The nice thing about ShellCheck is that they even provide the possibility to check your script via upload on their www.shellcheck.net website – so for small, public scripts, you don’t have to install anything at all to try it out, just copy and paste your script into the text box on the website.

Comments

When we talk about getting better at programming, we often talk about testing, writing reusable code, design patterns, and readability.

All of those things are important. But in this blog post, I want to talk about a different way to get better at programming: learning how the systems you’re using work! This is the main way I approach getting better at programming.

examples of “how things work”

To explain what I mean by “how things work”, here are some different types of programming and examples of what you could learn about how they work.

Frontend JS:

• how the event loop works
• HTTP methods like GET and POST
• what the DOM is and what you can do with it
• the same-origin policy and CORS

CSS:

• how inline elements are rendered differently from block elements
• what the “default flow” is
• how flexbox works
• how CSS decides which selector to apply to which element (the “cascading” part of the cascading style sheets)

Systems programming:

• the difference between the stack and the heap
• how virtual memory works
• how numbers are represented in binary
• what a symbol table is
• how code from external libraries gets loaded (e.g. dynamic/static linking)
• Atomic instructions and how they’re different from mutexes

you can use something without understanding how it works (and that can be ok!)

We work with a LOT of different systems, and it’s unreasonable to expect that every single person understands everything about all of them. For example, many people write programs that send email, and most of those people probably don’t understand everything about how email works. Email is really complicated! That’s why we have abstractions.

But if you’re working with something (like CSS, or HTTP, or goroutines, or email) more seriously and you don’t really understand how it works, sometimes you’ll start to run into problems.

your bugs will tell you when you need to improve your mental model

When I’m programming and I’m missing a key concept about how something works, it doesn’t always show up in an obvious way. What will happen is:

• I’ll have bugs in my programs because of an incorrect mental model
• I’ll struggle to fix those bugs quickly and I won’t be able to find the right questions to ask to diagnose them
• I feel really frustrated

I think it’s actually an important skill just to be able to recognize that this is happening: I’ve slowly learned to recognize the feeling of “wait, I’m really confused, I think there’s something I don’t understand about how this system works, what is it?”

Being a senior developer is less about knowing absolutely everything and more about quickly being able to recognize when you don’t know something and learn it. Speaking of being a senior developer…

even senior developers need to learn how their systems work

So far I’ve never stopped learning how things work, because there are so many different types of systems we work with!

For example, I know a lot of the fundamentals of how C programs work and web programming (like the examples at the top of this post), but when it comes to graphics programming/OpenGL/GPUs, I know very few of the fundamental ideas. And sometimes I’ll discover a new fact that I’m missing about a system I thought I knew, like last year I discovered that I was missing a LOT of information about how CSS works.

It can feel bad to realise that you really don’t understand how a system you’ve been using works when you have 10 years of experience (“ugh, shouldn’t I know this already? I’ve been using this for so long!“), but it’s normal! There’s a lot to know about computers and we are constantly inventing new things to know, so nobody can keep up with every single thing.

how I go from “I’m confused” to “ok, I get it!”

When I notice I’m confused, I like to approach it like this:

1. Notice I’m confused about a topic (“hey, when I write await in my Javascript program, what is actually happening?“)
2. Break down my confusion into specific factual questions, like “when there’s an await and it’s waiting, how does it decide which part of my code runs next? Where is that information stored?”
3. Find out the answers to those questions (by writing a program, reading something on the internet, or asking someone)
4. Test my understanding by writing a program (“hey, that’s why I was having that async bug! And I can fix it like this!“)

The last “test my understanding” step is really important. The whole point of understanding how computers work is to actually write code to make them do things!

I find that if I can use my newfound understanding to do something concrete like implement a new feature or fix a bug or even just write a test program that demonstrates how the thing works, it feels a LOT more real than if I just read about it. And then it’s much more likely that I’ll be able to use it in practice later.

just learning a few facts can help a lot

Learning how things work doesn’t need to be a big huge thing. For example, I used to not really know how floating point numbers worked, and I felt nervous that something weird would happen that I didn’t understand.

And then one day in 2013 I went to a talk by Stefan Karpinski explaining how floating point numbers worked (containing roughly the information in this comic, but with more weird details). And now I feel totally confident using floating point numbers! I know what their basic limitations are, and when not to use them (to represent integers larger than 2^53). And I know what I don’t know – I know it’s hard to write numerically stable linear algebra algorithms and I have no idea how to do that.

connect new facts to information you already know

When learning a new fact, it’s easy to be able to recite a sentence like “ok, there are 8 bits in a byte”. That’s true, but so what? What’s harder (and much more useful!) is to be able to connect that information to what you already know about programming.

For example, let’s take this “8 bits in a byte thing”. In your program you probably have strings, like “Hello”. You can already start asking lots of questions about this, like:

• How many bytes in memory are used to represent the string “Hello”? (it’s 5!)
• What bits exactly does the letter “H” correspond to? (the encoding for “Hello” is going to be using ASCII, so you can look it up in an ASCII table!)
• If you have a running program that’s printing out the string “Hello”, can you go look at its memory and find out where those bytes are in its memory? How do you do that?

The important thing here is to ask the questions and explore the connections that you’re curious about – maybe you’re not so interested in how the strings are represented in memory, but you really want to know how many bytes a heart emoji is in Unicode! Or maybe you want to learn about how floating point numbers work!

I find that when I connect new facts to things I’m already familiar with (like emoji or floating point numbers or strings), then the information sticks a lot better.

Next up, I want to talk about 2 ways to get information: asking a person yes/no questions, and asking the computer.

how to get information: ask yes/no questions

When I’m talking to someone who knows more about the concept than me, I find it helps to start by asking really simple questions, where the answer is just “yes” or “no”. I’ve written about yes/no questions before in how to ask good questions, but I love it a lot so let’s talk about it again!

I do this because it forces me to articulate exactly what my current mental model is, and because I think yes/no questions are often easier for the person I’m asking to answer.

For example, here are some different types of questions:

• Check if your current understanding is correct
  
  • Example: “Is a pixel shader the same thing as a fragment shader?”
• How concepts you’ve heard of are related to each other
  • Example: “Does shadertoy use OpenGL?”
  • Example: “Do graphics cards know about triangles?”
• High-level questions about what the main purpose of something is
  • Example: “Does mysql orchestrator proxy database queries?”
  • Example: “Does OpenGL give you more control or less control over the graphics card than Vulkan?”

yes/no questions put you in control

When I ask very open-ended questions like “how does X work?”, I find that it often goes wrong in one of 2 ways:

1. The person starts telling me a bunch of things that I already knew
2. The person starts telling me a bunch of things that I don’t know, but which aren’t really what I was interested in understanding

Both of these are frustrating, but of course neither of these things are their fault! They can’t know exactly what informatoin I wanted about X, because I didn’t tell them. But it still always feels bad to have to interrupt someone with “oh no, sorry, that’s not what I wanted to know at all!”

I love yes/no questions because, even though they’re harder to formulate, I’m WAY more likely to get the exact answers I want and less likely to waste the time of the person I’m asking by having them explain a bunch of things that I’m not interested in.

asking yes/no questions isn’t always easy

When I’m asking someone questions to try to learn about something new, sometimes this happens:

me: so, just to check my understanding, it works like this, right?
them: actually, no, it’s <completely different thing>
me (internally): (brief moment of panic)
me: ok, let me think for a minute about my next question

It never quite feels good to learn that my mental model was totally wrong, even though it’s incredibly helpful information. Asking this kind of really specific question (even though it’s more effective!) puts you in a more vulnerable position than asking a broader question, because sometimes you have to reveal specific things that you were totally wrong about!

When this happens, I like to just say that I’m going to take a minute to incorporate the new fact into my mental model and think about my next question.

Okay, that’s the end of this digression into my love for yes/no questions :)

how to get information: ask the computer

Sometimes when I’m trying to answer a question I have, there won’t be anybody to ask, and I’ll Google it or search the documentation and won’t find anything.

But the delightful thing about computers is that you can often get answers to questions about computers by… asking your computer!

Here are a few examples (from past blog posts) of questions I’ve had and computer experiments I ran to answer them for myself:

• Are atomics faster or slower than mutexes? (blog post: trying out mutexes and atomics)
• If I add a user to a group, will existing processes running as that user have the new group? (blog post: How do groups work on Linux?)
• On Linux, if you have a server listening on 0.0.0.0 but you don’t have any network interfaces, can you connect to that server? (blog post: what’s a network interface?)
• How is the data in a SQLite database actually organized on disk? (blog post: How does SQLite work? Part 1: pages!)

asking the computer is a skill

It definitely takes time to learn how to turn “I’m confused about X” into specific questions, and then to turn that question into an experiment you can run on your computer to definitively answer it.

But it’s a really powerful tool to have! If you’re not limited to just the things that you can Google / what’s in the documentation / what the people around you know, then you can do a LOT more.

be aware of what you still don’t understand

Like I said earlier, the point here isn’t to understand every single thing. But especially as you get more senior, it’s important to be aware of what you don’t know! For example, here are five things I don’t know (out of a VERY large list):

• How database transactions / isolation levels work
• How vertex shaders work (in graphics)
• How font rendering works
• How BGP / peering work
• How multiple inheritance works in Python

And I don’t really need to know how those things work right now! But one day I’m pretty sure I’m going to need to know how database transactions work, and I know it’s something I can learn when that day comes :)

Someone who read this post asked me “how do you figure out what you don’t know?” and I didn’t have a good answer, so I’d love to hear your thoughts!

Thanks to Haider Al-Mosawi, Ivan Savov, Jake Donham, John Hergenroeder, Kamal Marhubi, Matthew Parker, Matthieu Cneude, Ori Bernstein, Peter Lyons, Sebastian Gutierrez, Shae Matijs Erisson, Vaibhav Sagar, and Zell Liew for reading a draft of this.

---

This blog is totally free, but if you like it, please consider supporting it.

I was originally going to write a normal prose article regarding the topic of Pragmatism in Programming, however I thought I’d experiment in style by writing in a proverbial style. The following concepts express the school of thought that I subscribe to which I call Pragmatism in Programming. Proverbs written by Ginger Bill The concept of programming Programming is a tool to solve problems that you have in the domain of computers