Learning Sed

sed is one of Unix's oldest and (to my mind) weirdest tools. Its name means "Stream EDitor," and it's meant to edit a stream of text. That is, automated editing. But sed dates back to the days when the text editor of choice was ed, which is generally acknowledged to be hard-to-use. It evolved into the far more user-friendly vi. If you think Vi's single character command set is a nightmare ... sed is probably not for you.

It's very powerful, but - because of those single-character commands - also staggeringly obtuse. It's often the best tool for System Administration or Ops/Operations (depending on your naming preference), but it's so non-obvious that most ops people never learn anything more than sed 's/this/that/' when it can do so much more.

This isn't a lesson on how to use sed, but rather an introduction to its uses, competitors, and sources of information.

I was looking for a way to edit the last X lines of a file recently and ran across this page: https://catonmat.net/sed-one-liners-explained-part-one (the answer to my question is actually in part 2). "Sed One-Liners Explained" is very good, and acts as a good introduction to sed as he works through progressively more complex examples.

The answer to my question looks like this: sed -e :a -e '$q;N;11,$D;ba filename' - that's how to print the last ten lines of a file. tail -n 10 filename is so much easier ... but if you want to edit the last ten lines (rather than just print them, which is all tail can do), using sed makes more sense.

Bulk Rewriting Files - Different Tools

A year or two ago I found that the way Pelican (my static HTML blog software) handled Python RST (ReStructured Text) had changed. They had previously honoured

:title: Title of Blog Entry

as a way of setting the title, but now required a newer form:

Title of Blog Entry
####################

As I had literally hundreds of files that Pelican would no longer parse, I needed a way to rewrite all these files quickly. As it was always the first line of the file, this was somewhat simplified. In the end, I chose NeoVim (did you know that Vim can be scripted? It can be):

nvim +"1s/:title: //" +'r! echo ' +"s/$/####################/" +'r! echo ' +wq "file.rst"

This can also fairly easily be done with sed:

sed -i -e '1s/:title: //' -e '1a###################\n' file.rst

or awk (although this was a bit trickier):

awk -i inplace 'NR==1{ gsub(/:title: /, "") ; print ; print "####################\n" } ; NR>1{ print }' file.rst

All of these rewrite the file in place. If you do that, spend a LOT of time checking your output. What if I ever put the :title: tag on line two? Every one of these methods would have failed.

Bibliography