A "console" in UNIX speak (UNIX..? think Linux - best to confuse one concept at a time!) is the text-based interface, as opposed to the graphical interface (the "desktop" in people speak). The console is sometimes referred to as a "tty" (short for teletype, which is graybeard speak - the kind of people who refer to a monitor as a "glass teletype") You can change between consoles and the graphical environment by holding the keys Ctrl + Alt + Fn (eg. Ctrl + Alt + F1 will change to tty1 - the first console). Exactly how many consoles are available, and where the graphical environment is located, depends greatly on which version of UNIX you are running, and how it is configured. More on that later, but feel free to experiment! A console (technically a virtual console) is not the same as a terminal emulator, such as XTerm, which is a program you can run within a graphical environment. Neither is it a shell by the way, a shell like bash for instance runs inside a terminal emulator/virtual console. Confusion reigns supreme however since terminal emulators and virtual consoles are often just called "terminals", "consoles" or "shells" interchangeably (btw, all of them are "cli's" - command line interfaces).
If you feel confused at this point, the following analogy might help: Think of a "shell" as a language, such as English. A "console" as an oldschool office from the 50's, with only basic essentials like a typewriter and a filing cabinet. And a "desktop" as a modern office of plexiglas and shiny chrome, complete with air conditioning, a coffee machine, fancy artwork on the walls, soft ambient background music and what not. It even has some pencils and papers laying around somewhere if people get in a productive mood... Although the office environments are certainly different, and reflect different time periods, in practice the occupants often work on the same things, and of course they all use the same language.
Yes. The text-based environment in UNIX is incredibly versatile and powerful, programmers and system administrators who log into a UNIX server remotely will typically work exclusively in the console, working on a remote desktop (similar to TeamViewer in Windows) is just too inefficient. But can a console really function as an everyday desktop? That is what this article will try to explore, and I think you will be pleasantly surprised at just how useful a text-based environment can be!
Useful you say? Isn't a text based environment like... boring?!? That depends. Learning to use the console certainly requires effort, just as it does to learn any language. But once mastered, you don't think much about it as either boring or fun, instead you focus on the task at hand. An article you read might peak your interest or not, but you wouldn't consider the English language itself as boring would you? Once a language is learned you don't think much about it.
PS: Don't let the term language here scare you. Learning another human language is a huge task, since it consists of tens of thousands of words and painfully convoluted grammar. In contrast if you have learned only a hundred "words" (ie. commands), you are a fluent UNIX speaker, even knowing just a dozen or so is usually enough to get by. Grammar, if at all applicable, is very simple.
Since our goal here is to illustrate day-to-day "desktop" usage, we will not go too much into system administration and programming (although we will have to dip our toes into such matters a little bit). The internet is brimming with articles on these matters, so just Google around if you need some help ;^) The article assumes you are already familiar with basic UNIX commands, and many solutions are provided as shell scripts. If you don't have the skill to read such programs, the following books may be useful additions to your bookshelf:
Despite not delving deeply into administration and programming however, we will cover much ground and therefore I try to give brief instructions. You are encouraged to read the manpages and relevant documentation yourself to get the finer details. I will start off by discussing how to configure and use the console, and introduce some basic topics. Although necessary, this first section is mindnumbingly boring, so you might want to skip ahead to the fun parts. Before we do any of that though, lets settle some ugly disputes first.
WARNING: The following section contains strong language and gratuitous violence to truth, taste and civility. Do not let children or lawyers, nor any people with weak constitution or humor read it. Feel free to send me a complaint if you feel unflamed after reading this section.
You would be surprised how often, given any topic on UNIX, the discussion is derailed at the onset by some newb asking: By the way, what distro should I use? I have heard of this thing called BSD, is that better or worse then Linux? While we are at it, what text editor would you recommend? What browser? What desktop? What programming language? And so forth... Thankfully, we are more or less avoiding that entire GUI angle in this article, so many of these questions can be safely ignored. But the first two will no doubt be asked, so I suppose we'll have to answer them.
Originally, there were two kinds of UNIX, the one created by hippies at the University of California at Berkeley, called BSD (Berkeley Software Distribution), and the one created by evil monsters at AT&T headquarters, called SysV (UNIX System 5). Our distant forefathers from the 80's fought valiantly in the UNIX wars, and many brave men and women lost their arguments in Usenet battles, as the two giants fought for supremacy. Today however, AT&T UNIX is dead. Rumor has it that some remnant babies of this corporate monster still exist in the wild, in the form of AIX, HP-UX, and Solaris, but like I said, it's just a rumor. For all intents and purposes BSD is the only candidate that remains standing, and thus wins by default (apparently some kid in Finland has created his own clone of SysV, but I can't see that going anywhere). Confusingly though, there exists several versions of BSD today. So which is the best? Well... that is a tricky question. But I suppose if we have to pick one, and assuming you aren't into merchandising or fanaticism, the popular choice would be NetBSD. It can run on all the popular platforms, such as i386, and with pkgsrc you can even compile nethack on NetBSD via the net, what more could you want?
But what if you wanted to take this new Finnish hack for a spin, there's about a thousand Linux distros out there, so which one is the right one for you? Well, using logic we can narrow it down to two candidates: If you are one of those people who hate a multitude of distros, then Slackware is clearly the right choice, I mean all the other ones are just upstarts and spinoffs, exactly the kind of thing that you hate, right? On the other hand, if you are one of those people who love a multitude of distros, then LFS is the one for you, that way you can make as many distros as you like. You see, it doesn't have to be so hard if we just put our minds to it.
PS: Realistically BSD is the only candidate here as a day to day driver, but it's fun non the less to fire up a VM and play with Linux every so often. I mean, young folks do have good ideas on occasion.
Another source of inflammatory infighting in UNIX, is the choice of text editor. There are too many popular fad editors out there to mention, and they all have nifty little features that lure unsuspecting newbies into their fold. Now, I'm not here to judge. Some people cannot read a book without pictures, and some developers cannot write code without syntax highlighting, I get it. But for the true UNIX hacker there is only one editor, only one that adheres to the UNIX principals, the official one; ed. Like all true UNIX programs ed can take its input arbitrarily from the keyboard, a file or a pipe, and behave consistently in all cases. It can work in a true textual environment, meaning on paper. Try using vi with a lineprinter as your interface! Real programmers do not need visual aids, like some invalid, they see their code in their minds! To learn how to use ed, read the manpage. "It really is that simple", to quote Steve Jobs. If you need more hand holding however, I suggest reading Appendix 1 in The UNIX Programming Environment (1984) by Kernighan and Pike, or read Ed Mastery (2018) by Michael W. Lucas. Once you've got it all down, install bsdgames and run quiz function ed-command for a nice lesson in humility.
Even though the ed manual is an easy read, there are illiterate users, some by necessity, some by choice. Fear not, nano is for you! Start nano. See the ^X Exit at the bottom? It means: Hit Ctrl-X to Exit, I'm sure you can work out the rest yourself.
PS: The BSD editor ee (easy editor) is better, since it's twice as easy to type.
Even though ed is the standard UNIX editor, vi is the popular editor for the masses. No doubt because of it's renowned user-friendliness. It has spawned many forks, the most popular one being vim (vi improved). As the name suggests, vi is a visual editor, so you can actually see what's going on. And the keybindings are mnemonic enough to closely resemble spoken English. The only thing you really need to learn in order to become a vi user, are modes. It works like this: When you start vi it opens the file in normal mode. In this mode you can move around, and do basic bulk editing on the text. Things like d3w (delete three words), yy (yank, just yank - the line) and p (paste) it somewhere. To input text, just a (append) it. To escape input mode you hit Escape of course. See, it's dead simple. Finally, vi lets you run commands that effect the whole file, things like saving and quitting or search and replace. To run one of these commands, type colon in normal mode, and issue your orders. For example, :wq (write'n quit), :q! (quit dammit!), or :%s/good/bad/g (100% of file, substitute good for bad, globally). Using vi is simply a matter of understanding what the keys mean. Small wonder it's so ubiquitous, you just can't beat the simplicity. Here is a cheat sheet to get you started.
PS: The BSD editor nvi (new vi) is better, since it's bug-for-bug compatible with vi(1).
Emacs is best summarized in the following statement: Emacs is an operating system that only lacks a good text editor. There are several forks of Emacs around, but the most popular one by far is GNU Emacs. Essentially Emacs makes this entire article redundant, anything you can to in the console, you can do in Emacs. For example, you can type M-x shell-command (that is Alt-x, then type shell-command) to run any command, like ifconfig or firefox or what have you. To run a full shell session type M-x shell (if you need to run fancy ncurses programs use M-x ansi-term).
Time does not permit me to discuss the eww web browser, gnus news and mail client (or one of the several superior candidates), erc irc client, dired file manager, calendar, ses spreadsheets, calculator, tetris, snake, dunnet text adventure, the psychoanalyzer doctor and other tools and toys that ship with the editor by default. Not to mention the thousands of 3rd party apps that can be installed with its package manager. Such as the emms multimedia system, slime for serious Lisp hacking, and org-mode for serious text hacking. Naturally any kind of file, including PDF's and images can just be opened directly.
All of the above major modes can be started with M-x modename. To open up a file you can type C-x C-f (that is Ctrl-x then Ctrl-f). These modes and files are called "buffers" in Emacs jargon, to open up a non-current buffer type C-x b. Tab-autocomplete works everywhere, so use that for what it's worth. Delete a buffer with C-x k. To quit Emacs, type C-x C-c, and to cancel an action type C-g. Emacs is also a window manager. To split the window vertically in two type C-x 2, to split it horizontally type C-x 3 (these can be arbitrarily split again and again). To close a window type C-x 0, to make a window fullscreen type C-x 1, to switch to another window type C-x o (for "other").
You can even use Emacs as a text editor. It's true! You can start the tutorial with C-h t, to learn how, but I wouldn't recommend it. The one thing Emacs does not come with by default is a decent text editor. Luckily though, you can emulate vi in Emacs. You could use viper mode in vanilla Emacs, but a better solution might be to install the 3rd party package evil. You can do so using the above mentioned package manager, or you can use one of the many user-friendly Emacs distros out there, such as Prelude, Graphene, Emacs starter kit, Spacemacs or DoomEmacs. Wait... package manager... Lisp machine... distros?!? You though I was kidding when I called Emacs an operating system didn't you? (Honestly though, Emacs is an ungainly pile of pants, but it does have one saving grace (Lisp (the ultimate language (for programming)))))))))). Btw, here is a cheat sheet to get you started.
PS: The BSD editor mg (micro gnu emacs) is better, since it's written in a higher level language.
As mentioned you normally have multiple virtual consoles available. So if you're waiting for a job to finish in tty1, you can hit Ctrl + Alt + F2 to jump over to tty2. When you want to go back to the first console again, just hit Ctrl + Alt + F1. Most systems will also allow you to scroll back the text with Shift + PgUp and Shift + PgDown, this enables you to go back and read previous output. In the following sections I will talk about how to configure fonts, colors, keyboard layouts and the consoles in general. The method of doing this varies greatly from system to system, so you don't need to read all of the subsections, just the ones that are relevant for you.
This hardware vendor deserves special mention. Now I know that many a Windows gamer out there loves his Nvidia card like his favorite pet monkey. And sure enough the Nvidia engineers can produce stunning specs, and provide hours of gaming fun like none other, but they cannot write quality software if their life depended on it! Their driver is a bad joke, and an affront to the finer feelings of system administrators everywhere. It is clear that the Nvidia developers have the Windows mindset, where it is perfectly fine, nay expected, to write binary blobs that bloat the system and boldly crash it in ways no one have crashed it before!
If you are using their proprietary driver on any version of UNIX (especially anything besides Linux), then you will likely not be able to switch back and forth between the console and your graphical environment. In fact you often cannot run multiple graphical environments simultaneously either, something that is unheard of in the Windows world, but is a given on UNIX systems. Don't be surprised either if your system suddenly freezes or crashes for no apparent reason whatsoever, it's to be expected. The only way to work around these issues is to use either the reverse engineered open source Nvidia driver, or configure X to use the generic VESA driver. In both cases you will usually have very shabby resolution, both on your desktop and on your console. Using the console with an Nvidia card may cause serious headache, but take a look at the Putting a Console on the Desktop section below for a simple workaround.
As a side note: The OpenBSD developers have embargoed Nvidia on their system. If more operating systems would follow their example, instead of showing their finger in frustration (looking at you Linus Torvalds), Nvidia might just take a hint and clean up their act.
To be fair, other vendors like Intel, have also shown a shocking lack of concern for the quality of their own product. Of course Intel's specialty is multi-processing and virtualization, not graphics, so it's in those areas that they bury their bodies. As far as graphics are concerned, Intel should not give you any problems on the console.
To be extra fair, some have objected to my objections about Nvidia. I find such criticism of my criticism surprising. Not because I might be wrong, but because people have cared enough about my blog to criticize it. I am humbled. Let me stress that I am talking about *console* issues with proprietary Nvidia drivers. Desktop graphics usually work fine. And I am sure many Linux distros handle even these proprietary drivers gracefully on the console. The fact that I have issues does not mean that you have issues. Naturally, anything you read on my blog, or the web, can be obsolete, opinionated and/or dim. Beware.
If you have a graphical desktop already running, getting to it from the console is simply a question of finding out where it is located. If you are unsure just hit Ctrl + Alt + F1, then Ctrl + Alt + F2, and so on all the way up to F12 until you hit it (it is often at F5 or F7, but it varies). Note that while some systems allow you to use the shorthand Alt + Fn, when switching consoles, you need to use the full keybinding sequence Ctrl + Alt + Fn, when switching back and forth between the graphical environment and the text consoles.
You can also start a graphical environment from a console. If no graphical environment is running, just type startx. This will launch the systems default graphical environment, which is often twm, a horribly antiquated window manager (many who see this for the first time do not realize they are running a desktop, they just assume that the computer got broken somehow). You can configure startx to run something else by adding instructions in ~/.xinitrc (that is .xinitrc in your home directory). Here is a short example that sets the keyboard layout to Italian and launches Xfce:
setxkbmap it
exec dbus-launch --exit-with-session startxfce4
PS: Getting the launch instructions right for a modern desktop, such as KDE or GNOME, can be tricky (the above Xfce instructions can also vary from system to system). If you're having problems starting them I suggest you try with a simple window manager first, such as exec openbox, to check that the actual graphical environment (normally the X Window System, sometimes referred to as X11, X.Org, or just "X") is working. If it is, you have probably misconfigured the KDE/GNOME/Xfce launch instructions, and you'll know what to Google for.
If a graphical environment is already running you need to launch your new desktop in a different virtual screen. You can do so like this: startx -- :2, the number here is arbitrary, what matters is that the screen ID number must be unique.
Multi-Monitor setups is not possible on any UNIX console. That is to say, you can have as many monitors as you like, but they will all show the same screen. Multitasking however is possible, for one, you can configure a number of virtual consoles and switch between them (see also the tmux section below). Secondly, you can have a multi-monitor setup in X, and run a "console-like" desktop, such as dwm.
Linux will usually configure a number of consoles by default, 6 consoles and the graphical environment on the seventh seems to be a popular configuration. You can configure an arbitrary number of consoles on Linux, see the next sections for further details. You can also use the short hand Alt + Fn and Alt + Arrow-Key to change consoles.
You can set keyboard layout with loadkeys, such as loadkeys dvorak.map. The keyboard maps are often located in /usr/share/keymaps or /usr/share/kbd/keymaps (PS: This approach should also work on systemd distros, but did not work for me on Debian. You could try dpkg-reconfigure keyboard-configuration instead (naturally that will only work on Debian-like distros)). You can change fonts with setfont, such as setfont -v Uni3-Terminus12x6, the console fonts are usually located in /usr/share/consolefonts or /usr/share/kbd/consolefonts.
The Linux console supports multiple colors and you can use echo to send control sequences to manipulate the console color settings. Adding this to the end of ~/.profile will make your console use the Solarized color theme:
# solarize the tty
if [ "$TERM" = "linux" ]; then
echo -en "\e]P0073642" #black
echo -en "\e]P1dc322f" #darkgray
echo -en "\e]P2859900" #darkred
echo -en "\e]P3b58900" #red
echo -en "\e]P4268bd2" #darkgreen
echo -en "\e]P5d33682" #green
echo -en "\e]P62aa198" #brown
echo -en "\e]P7eee8d5" #yellow
echo -en "\e]P8002b36" #darkblue
echo -en "\e]P9cb4b16" #blue
echo -en "\e]PA586e75" #darkmagenta
echo -en "\e]PB657b83" #magenta
echo -en "\e]PC839496" #darkcyan
echo -en "\e]PD6c71c4" #cyan
echo -en "\e]PE93a1a1" #lightgray
echo -en "\e]PFfdf6e3" #white
clear #for background artifacting
fi
Not only can the Linux console display any number of colors, but it is also capable of displaying real graphics. It can do so using the framebuffer device. In order to use it you must first make sure your user has permission to access /dev/fb0 and the files under /dev/input. This usually means you need to add your user to the groups video and input, you can do so by running this command: usermod -a -G video,input <myuser>. You also need to install the fbdev driver, if it isn't already included. The method of doing so varies from distro to distro. For instance on Slackware you can install it like so: installpkg /mnt/cdrom/extra/xf86-video/*.txz, and on Debian: apt install xserver-xorg-video-fbdev fbset gpm. Look up your distros documentation.
Two examples of programs you can use on the Linux framebuffer is fbterm and fbi. The fbterm terminal can use any fonts available under the graphical environment, which easily enables you to have support for exotic languages like Arabic or Chinese. In combination with the image viewer fbi, you can also set a terminal wallpaper (Ps: You cannot launch another framebuffer program from within fbterm, you need to do that from a regular text console).
#!/bin/sh
# fbtermbg - start fbterm with wallpaper
# usage: fbtermbg wallpaper
# depend: fbi
(sleep 1; cat /dev/fb0 > /tmp/wallpaper.fbimg) &
fbi -t 2 -1 --noverbose -a $1
export FBTERM_BACKGROUND_IMAGE=1
cat /tmp/wallpaper.fbimg > /dev/fb0
fbterm
Some framebuffer programs will hijack your screen, so that you cannot switch to other consoles or the graphical environment. Don't panic! Everything should return to normal once you have quit the program (Ps: If your console is all garbled after you have quit a framebuffer program, the reset command should fix it).
Legacy Linux systems, and a few modern exceptions, such as Slackware, Gentoo and CRUX, do not use systemd.
You can change virtual console settings in /etc/inittab. To add an eighth tty in Slackware, add this line to /etc/inittab:
c8:12345:respawn:/sbin/agetty 38400 tty8 linux
This takes effect after a reboot. In non-systemd Linux you can also switch to tty13-24 with AltGr + Fn.
Slackware boots in a text based environment by default, to change this to a graphical login, change the line: id:3:initdefault to id:4:initdefault in /etc/inittab. On non-systemd Linux runlevel 3 is text mode login, whereas runlevel 4 is graphical login. The process described here should be fairly similar to any Linux distribution that doesn't use systemd.
Most modern Linux systems, including Debian, Red Hat, SUSE, Arch, and quite a few others, use the systemd meta system daemon.
To change the number of consoles, to say 12, in Debian, uncomment NAutoVTs in /etc/systemd/logind.conf and set it to NAutoVTs=12. This will take effect after a reboot. On Linux systems with systemd, you can switch between the tty13-24 with Shift + Alt + Fn.
To configure Debian to boot into text mode instead of a graphical login, do this: systemctl set-default multi-user.target, to switch back to graphical login: systemctl set-default graphical.target. The process described in this section should be fairly similar to any Linux distribution that uses systemd.
Linux users often take a lot for granted, they tend to get horrified when discovering that a different UNIX system doesn't have some functionality they have gotten used to, or even if they just do things differently (they may even shout "this isn't UNIX!!!" without realizing the irony - it's usually Linux that's doing things in a non-UNIX'y fashion). So if you're planning to use a BSD system or some other UNIX variety, and have a Linux background, keep an open mind and be prepared to adjust your expectations!
When it comes to the console, non-Linux systems will not support graphics on the framebuffer, nor support exotic UTF-8 characters (ei. English only), and color and font support is often very limited. In fact many ncurses applications may not work well on non-Linux consoles. Some systems also restrict how many consoles you are allowed to have. If you think this all sounds horribly limiting, you probably haven't spent your youth writing PDP-11 assembly in V6 using ed. (all of these restrictions can be easily overcome though by simply running terminal emulators in a graphical desktop). Lastly, keybindings might be a little different, although Ctrl + Alt + Fn will always work.
Despite these restrictions, you can still do neat things on any UNIX console with a little know-how. The above mentioned trick to make a Solarized console theme in Linux will not work on most UNIX consoles since they don't support 256 colors, but virtually all of them have 8 color ANSI support, so you can do some basic color tweaking. Here is a simple and portable script that can set a few basic tty color schemes (unlike the Linux trick though, the themes are not preserved if you run a terminal application with colors, such as vim or less, so a better solution is to configure your system to use the colors you want - more on that later):
#!/bin/sh
# ttycolor - choose tty colors
# usage: ttycolor theme
# bugs: not persistent across colorful tty apps
#
# explanation:
# printf '\e[1m' - turn on boldface (or "light" color)
# printf '\e[4m' - turn on underscore - not used here
# printf '\e[5m' - turn on blink - not used here
# printf '\e[7m' - reverse colors - not used here
# printf '\e[0m' - turn off attributed, eg. boldface
# printf '\e[4nm' - specify background color ("boldface" has no effect)
# printf '\e[3nm' - specify foreground color, where n is:
# 0 = black 2 = green 4 = blue 6 = cyan
# 1 = red 3 = yellow 5 = magenta 7 = white
usage(){
echo 'Usage: ttycolor (geek|tron|minoca|sun|obsd|nuke|default)' >&2
exit 1
}
if [ ! $# = 1 ]; then
usage
fi
# set tty colors
case $1 in
geek) printf '\e[0m\e[32m\e[40m' ;; # green on black
tron) printf '\e[1m\e[36m\e[40m' ;; # cyan on black
minoca) printf '\e[1m\e[33m\e[42m' ;; # yellow on green
sun) printf '\e[0m\e[30m\e[47m' ;; # black on white
obsd) printf '\e[1m\e[37m\e[44m' ;; # white on blue
nuke) printf '\e[1m\e[33m\e[41m' ;; # yellow on red
default) printf '\e[0m\e[37m\e[40m' ;; # white on black
*) usage ;;
esac
clear
FreeBSD has 8 virtual consoles configured and the graphical interface on the ninth by default. Modern versions of FreeBSD use the vt(4) console driver. It allows you to configure up to 16 consoles, and switch to tty13-16 with Shift + Alt + F1 ... F4. You can also use the Alt + Fn shortcut, but not the arrow keys. To scroll backwards and read previous output hit Scroll Lock, and then use Page Up and Page Down, hit Scroll Lock again when you are done. (if your laptop doesn't have Scroll Lock you can usually simulate it in various ways, Fn + k works on my ThinkPad T430 for instance) To configure more than 8 consoles edit /etc/ttys, for instance this line: ttyv9 "/usr/libexec/getty Pc" xterm onifexists secure will enable a ninth console. To enable all 16 consoles use ttyva ... ttyvf, not ttyv10 ... ttyv15. You can tweak other console settings by adding options to /etc/rc.conf. For instance, these lines:
moused_enable="YES"
allscreens_flags="-f 8x8 /usr/share/vt/fonts/terminus-b32.fnt green"
keymap="us.dvorak.kbd"
kld_list="i915kms"
will enable the console mouse, use large green terminus fonts and the dvorak keyboard layout. Lastly it will load the Intel graphics firmware, you need to have the drm-kmod package installed for this to work (if you are using Nvidia or Radeon (AMD) you need to change this line accordingly). You can interactively select a console font with vidfont, or use the vidcontrol command to adjust all of the above settings.
Even with 16 ttys configured in /etc/ttys, FreeBSD limits you to a maximum of 12 consoles by default, to add up to 16, you need to tweak the kernel a bit:
# cd /usr/src/sys/$(uname -m)/conf
# cp GENERIC MYKERN
# sed -i '' 's/GENERIC/MYKERN/' MYKERN
# echo options VT_MAXWINDOWS=16 >> MYKERN
# cd /usr/src
# make buildkernel KERNCONF=MYKERN
# make installkernel KERNCONF=MYKERN
# shutdown -r now
DragonFly BSD uses the old sc(4) console driver from FreeBSD, but otherwise the setup is exactly the same. By the way, you don't need to tweak the kernel in order to enable all 16 consoles, like you do in FreeBSD (PS: the sc driver might have problems switching between the graphical environment and consoles).
PS: BSD systems write kernel messages to the first console. This can be quite annoying if you happen to be working on other things there, so work on any console except the first one to avoid much consternation.
OpenBSD uses the wscons(4) console driver. It requires you to switch between consoles using Ctrl + Alt + Fn, you cannot use the shorthand Alt + Fn. This is a design choice, made to avoid keybinding conflicts. There are no key bindings to switch beyond 12 consoles.
In OpenBSD you have four consoles enabled by default, with the fifth tty being reserved for the graphical environment. (if you plan on using a graphical login manager, make sure that ttyC4 is off!) Adding more consoles is done in /etc/ttys:
ttyC7 "/usr/libexec/getty std.9600" pccon0 on secure
Oh, by the way, once your editing /etc/ttys,
change the vt220 option for your active consoles to pccon0.
vt220 is a ultra-conservative safe choice,
but pccon0 has more modern capabilities,
such as unicode, colors and curses.
The 11th and 12th console must be ttyCa/ttyCb,
not ttyC10/ttyC11.
Rebooting will not activate your new consoles however, you need to "create" them first. Doing so manually is a bit tedious, instead we can recompile the kernel with some new settings:
# cd /usr/src/sys/arch/$(uname -m)/conf
# cp GENERIC.MP CUSTOM
(Assuming you are running a multi-processor kernel, if not copy GENERIC) Now open up CUSTOM in an editor and add the following lines:
option WS_KERNEL_FG=WSCOL_WHITE
option WS_KERNEL_BG=WSCOL_BLACK
option WS_DEFAULT_FG=WSCOL_GREEN
option WS_DEFAULT_BG=WSCOL_BLACK
option FONT_BOLD8x16
These settings will result in kernel messages being written in a white font, and normal text in green. We also changed the font to a smaller type, fitting more text onto our screen (you'll find a list of available fonts in /usr/src/sys/dev/wsfont/wsfont.c, if you have installed sources). Finally change the WSDISPLAY_DEFAULTSCREEN=6 to WSDISPLAY_DEFAULTSCREEN=12 in GENERIC (regardless of whether you are running a multi-processing kernel or not). Now that the customization's are done, we can recompile the kernel:
# config CUSTOM
# cd ../compile/CUSTOM
# make clean
# make
# make install
# shutdown -r now
PS: X is big, bloated and old, it is a mayor security risk! The OpenBSD developers are somewhat conscientious about security. They have therefore reconfigured X to run in user mode, denying it any direct access to the system. This is great for security, but a consequence of doing so is that you cannot run more than one graphical environment at a time. You can use Xephyr or Xnest to nest X instances inside each other though (eg. (Xephyr :2 -screen 1920x1080 &); sleep 1; DISPLAY=:2 openbox). Also the normal way of starting X in OpenBSD from the console is not with startx, but rather doas xenodm, which starts the xdm-like login manager. Finally, as mentioned OpenBSD does not support Nvidia cards. Graphics may still "work" on such cards, but you will only get generic VESA output with poor resolution.
PS: OpenBSD, like all BSD systems, write kernel messages to the first console. So it's a good idea to work on a different console to avoid interruptions.
NetBSD also uses the wscons(4) console driver, like OpenBSD. It requires you to switch between consoles using Ctrl + Alt + Fn, you cannot use the shorthand Alt + Fn.
Configuring the NetBSD console is fairly similar to OpenBSD. Add new consoles to /etc/ttys, for instance:
ttyE7 "/usr/libexec/getty Pc" wsvt25 on secure
NetBSD also has four consoles by default and uses the fifth as the graphical interface (so keep ttyE4, the fifth tty in /etc/ttys, off if you plan on using a graphical login manager!), We can change some of the kernel defaults and recompile like so:
# cd /usr/src/sys/arch/$(uname -m)/conf
# cp GENERIC CUSTOM
# vi CUSTOM
Lets tweak some color and font settings, as well as enabling 8 ttys: Change WS_KERNEL_FG=WSCOL_GREEN to WS_KERNEL_FG=WHITE, and add options WS_DEFAULT_FG=WSCOL_GREEN. Uncomment WSDISPLAY_DEFAULTSCREENS=4 and change it to WSDISPLAY_DEFAULTSCREENS=8, and add options FONT_DEJAVU_SANS_MONO12x22 (you'll find a list of available fonts in /usr/src/sys/dev/wsfont/wsfont.c). Then recompile the kernel with these changes:
# config CUSTOM
# cd ../compile/CUSTOM
# make depend
# make
# mv /netbsd /netbsd.old
# mv netbsd /
# shutdown -r now
By default the NetBSD kernel only allows you to have a maximum of 8 consoles. Enabling more than that would require some more kernel hacking (see Sidenote below), I recommend using tmux instead.
PS: NetBSD, like all BSD systems, write kernel messages to the first console. So it's a good idea to work on a different console to avoid interruptions.
Sidenote: If you absolutely have to have all 12 ttys, here is how:* Edit /usr/src/sys/dev/wscons/wsksymdef.h and add these two lines after #define KS_Cmd_Screen9:
#define KS_Cmd_Screen10 0xf40a
#define KS_Cmd_Screen11 0xf40b
Now edit wsdisplay.c in the same directory and change #define WSDISPLAY_MAXSCREEN 8 to 12. Edit wskbd.c and add these two lines after case KS_Cmd_Screen9::
case KS_Cmd_Screen10:
case KS_Cmd_Screen11:
Then we have to update the default keyboard maps for USB keyboards and PS/2. (alternatively, you can tweak /etc/wscons.conf, eg. wsconsctl -k -w map+='keycode 87 = Cmd_Screen10 f11 F11) Edit /usr/src/sys/dev/hid/hidkbdmap.c, so that it reads:
KC(68), KS_Cmd_Screen10, KS_f11,
KC(69), KS_Cmd_Screen11, KS_f12,
And /usr/src/sys/dev/pckbport/wskbmap_mfii.c, so that it reads:
KC(87), KS_Cmd_Screen10, KS_f11,
KC(88), KS_Cmd_Screen11, KS_f12,
Now, go to the CUSTOM kernel you created (see above), and change WSDISPLAY_DEFAULTSCREENS to 12, and add four new ttys in /etc/ttys. (the last two must be ttyE10, ttyE11, not ttyEa and ttyEb!) Finally, make a few device nodes and recompile the kernel and userland (sysadmin commands such as wsconsctl also need to be recompiled to work with the new code):
# cd /dev
# ./MAKEDEV ttyE8
# ./MAKEDEV ttyE9
# ./MAKEDEV ttyE10
# ./MAKEDEV ttyE11
# cd /usr/src/sys/arch/$(uname -m)/conf
# config CUSTOM
# cd ../compile/CUSTOM
# make clean
# make depend
# make
# mv /netbsd /netbsd.old
# mv netbsd /
# shutdown -r now # verify that the new kernel works
# cd /usr/src
# ./build.sh -O ../obj -T ../tools -U NetBSD-custom
# ./build.sh -O ../obj -T ../tools -U install=/
# shutdown -r now
One reason why stock NetBSD does not support the F11/F12 keys in tty switching, might be that using these keys will cause keyboard conflicts with some of its (many) supported architectures, such as dec or hpc. In theory, you can create more then 12 ttys, but tty switching is hard coded to Ctrl + Alt + ... in wscons, so it wouldn't be practical. (you can manually switch to tty 13 with wsconscfg -s 13)
Oddly enough virtual consoles have traditionally not been available in Solaris, and it does not come enabled by default in newer versions, such as OpenIndiana,* either. To add six consoles in OpenIndiana for instance, you need to run:
# svcadm enable vtdaemon
# for i in 2 3 4 5 6; do
> svcadm enable console-login:vt$i
> done
# svccfg -s vtdaemon setprop options/secure=false
# svccfg -s vtdaemon setprop options/hotkeys=true
# svcadm refresh vtdaemon
# svcadm restart vtdaemon
By default you are only allowed to activate a maximum of 6 consoles. Like Linux you can change between consoles using the Alt + Fn for ttys 1-12, and AltGr + Fn for ttys 13-24, or you can use the Alt + Arrow-Key shortcuts. To get back to the desktop hit Ctrl + Alt + F7. Not all graphical drivers will allow you to switch between console and desktop however. On one of my test machines I had to disable the graphical environment with svcadm disable lightdm and do a cold reboot. From the console I could then enable the desktop again with svcadm enable lightdm, but to get to the console again I would have to do another disable and reboot.
I was able to work around these problems by configuring X to use the generic VESA driver, but naturally this meant that I had a very poor screen resolution. Hopefully your graphic driver will be more cooperative. If not it might be better to ignore the tty consoles altogether and use terminal emulators in a graphical desktop instead. But if you really want to use the VESA graphics driver you can do so by adding this file in /etc/X11/xorg.conf.d/vesa.conf:
Section "Device"
Identifier "Card0"
Driver "vesa"
EndSection
The only way I have managed to add more then 6 consoles in OpenIndiana is the following hackish method: edit /lib/svc/manifest/system/console-login.xml as root, find the <instance name='vt6' enabled='false'> line and copy this until the trailing </instance>. Edit the copy so that it reads vt8 and /dev/vt/8, instead of vt6 and /dev/vt/6, in the three places that these values are mentioned. You can now repeat this process for vt's 9-15. Reboot and run the above svcadm commands for vt's 9-15 and you now have 14 consoles (vt7 is still reserved for the desktop). If you want more then 14 consoles you need to make more device files in /dev/vt first.
Setting the console keyboard layout, to say French, is done with kbd -s French. To choose a language interactively just type kbd -s. In theory, running this command as root should change the default console keyboard layout permanently, but due to a bug in the version of OpenIndiana I tested this on, it got reset to US qwerty on every reboot. I have yet to figure out how to adjust fonts and default colors in the Solaris console, or use the mouse and scrollback buffer (I suspect this doesn't work on all hardware).
It's somewhat of a digression, but another painful issue with Solaris systems is the lack of software. For virtually all Linux and BSD variants, most of the software mentioned in this article can just be installed with the package manager, but Solaris and Illumos repositories are much thinner. Besides manually compiling the things you need by hand (it's worse then it sounds), a nice workaround here is to install the NetBSD ports collection, called pkgsrc. It is designed to work across multiple operating systems, including Solaris. Of course, not everything will work out of the box, but it does help:
# get and bootstrap pkgsrc
$ cd /usr
$ wget ftp://ftp.NetBSD.org/pub/pkgsrc/pkgsrc-2021Q2/pkgsrc.tar.gz
$ tar xzf pkgsrc.tar.gz
$ cd pkgsrc/bootstrap
$ env CFLAGS=-O2 CC=/usr/bin/gcc ./bootstrap
$ export PATH=/usr/pkg/bin:$PATH
# install a game and have some fun
$ echo 'ACCEPTABLE_LICENSES+= nethack-license' >> /usr/pkg/etc/mk.conf'
$ cd ../games/nethack
$ bmake install clean # outside of NetBSD pkgsrc uses "bmake"
$ nethack
The procedure is much the same for whatever OS you want to use pkgsrc on. You can find some specific Solaris pointers here, although these instructions are for Solaris 9 and 10, and thus do not reflect Illumos systems perfectly.
The process, nay the art, of getting organized is a complex and abstract topic. We will not even try to present a self-help-book-like theory of how to do this! (there are plenty of those on Amazon if you are into that sort of thing) What we will do is show you the tools you need, and leave it as an exercise for you, dear reader, to figure out what to do with them. (see the pim section for more tips)
You are probably familiar with basic tools such as ls, cd, mkdir, rm, ln, pwd and so on. (if you'd like to try out something new though, you can check out eza or lsd (ls), zoxide or autojump (cd), bat (cat), ripgrep (grep) and fd (find) - you can find more suggestions here) The problem with organizing yourself on the command line though, is that your files are largely out of view. Out of sight, out of mind, as they say. You can quickly check what the filesystem looks like with tree -L 2 (or eza/lsd --tree). Another fantastic tool to quickly get an overview of your files, including their contents, is the file manager ranger (alternatively mc or nnn). Or you could check out the newcomer fzf (fuzzy finder - it can actually plow through any list such as the command history or git commits). On the other hand, if you just need to figure out what's taking so much space on your harddrive try ncdu/dust, or simply du -hs * | sort -h.
Sometimes though you don't need to get a general overview, but find a specific file that you have lost. The quickest way to do this is locate myfile. But the locate command has a weakness; it relies on a database that needs to be updated manually (read the man page for the specifics). If the database is out of date, it may not contain the file your looking for (tip: update the locate database periodically with a cron job). Worse, you may not remember the exact name, only certain details, such as general filesize and when you last worked on it. find is your friend! This command can do some impressive file searching, here are some examples:
Search for a directory named exactly MyDiR somewhere under /some/path:
$ find /some/path -type d -name MyDiR
Search for a PDF file somewhere under the current directory that is more then 10 Megabytes in size but less then 100:
$ find . -iname *.pdf -size +10M -size -100M
Case insensitive search for a file called myfile somewhere under the current directory, modified within the last 10 days and belonging to the www group:
$ find . -type f -iname myfile -group www -mtime -10
Last but not least, don't forget the invaluable grep command. With it you can recursively search for any file containing a certain text, eg:
$ grep -R "UNIX is awesome" /some/path
A terminal can run multiple programs at once. You have probably launched programs in the background before with command &. What you may not realize is that you can easily switch programs from the background to the foreground arbitrarily. To demonstrate: Type sleep 60 & and hit Enter, three times, now run jobs, and you will see these three background processes listed. Put job number 2 to the foreground with fg 2. Your console is now unresponsive; it is busy sleeping. Hit Ctrl + z and you will suspend the process to the background again. Run jobs and notice that job nr 2 doesn't have a & sign, it is currently paused. You can continue the process in the background with bg 2.
Of course, managing multiple jobs in this way is tedious. Some times however you might find that you need to freeze the program you are currently running, and do something real quick. Well that's easy, hit Ctrl + z, do your thing, and run fg when your done. Presto! Your program resumes like nothing happened.
For serious multitasking on the console you need terminal multiplexers. Think of a multiplexer as a window manager for the terminal. There are many alternatives, but we will focus on tmux (The classic multiplexer screen has similar capabilities, and there are other candidates, such as dvtm and zellij).
To quickly demonstrate some of it's abilities do the following: Fire up tmux and hit Ctrl + b and then c. At the bottom line you will now see 0:bash- 1:bash*. This means that there are two bash programs running, you are now seeing the program with an asterix, 1:bash*. You can change back to 0:bash by hitting Ctrl + b and then 0, and then back to 1:bash again with Ctrl + b and then 1. But tmux can do more... Hit Ctrl + b and then %, and you will see the screen split vertically. Hit Ctrl + b and then ", and it will split horizontally. You can navigate between these window frames by hitting Ctrl + b and then one of the Arrow keys. To resize the frames, do the same, but keep holding down Ctrl as you use the Arrow keys.
If you hit Ctrl + b and then d, tmux will quit. But you haven't just exited the program, tmux is detached. If you run ps -ely | grep tmux, you will see that it is still running. You can connect to this tmux session again by running the command tmux attach. This functionality is invaluable when working remotely on a server. If the internet connection is broken when you had tmux running, just log in again and run tmux attach, and you can continue where you left off. Naturally you can have many tmux sessions running in the background, use tmux ls to list them.
There is an absurd amount of things you can do with this program. Check the tmux manpage for the gory details. You can also check the available key-bindings with Ctrl + b and then ? (the notation here can be a bit confusing though: bind-key is usually set to Ctrl + b, so bind-key -T prefix d detach-client just means "hit Ctrl + b and then d to detach-client"). PS: depending on your system, tmux is often smart enough to understand mouse gestures, such as clicking on the window you want, or click and drag to resize it, after hitting the initial Ctrl + b key combo.
The console mouse in Linux is gpm. Many distros will have this daemon enabled by default, or at least will enable it if you install gpm. In Slackware you can start the console mouse with: chmod +x /etc/rc.d/rc.gpm and then /etc/rc.d/rc.gpm start. On systemd distros you can enable it with systemctl gpm.service start. When gpm is running you should be able to see a square jump across the screen when you move your mouse, this is the mouse "pointer."
Copy pasting with gpm is straight forward. Left click and drag to mark text, any marked text is automatically copied to the gpm clipboard. You can paste it by middle clicking (if you don't have a middle mouse button click both buttons simultaneously). Why the middle mouse button?!? Actually this is the standard UNIX behavior, go ahead and try it on your desktop! The inefficient, left click + copy + left click + paste method, stems from Windows.
Pasting text straight to the command prompt is not that practical however, and opening up a text editor will clear the clipboard! So how do we copy something and save it to a file? Simple: run cat << eof > myfile and hit Enter. Now copy paste what you want, when you're finished write eof and hit Enter. The text is now saved in myfile. If you find this solution confusing, I recommend reading up on shell redirection.
Ps: BSD systems use moused as their console mouse, but it works just the same as gpm (Solaris systems don't have a console mouse AFAIK).
tmux also works great for copy pasting text, especially since you can run windows side by side and copy paste between them. But learning the keybindings will take a bit of practice. First hit Ctrl + b and then [ (ei. open bracket) to enter copy mode, in this mode you can move the cursor around freely. Navigate to the start of the text you want to copy and hit Ctrl + Space, then navigate to the end of the text (the section you want should now be highlighted) and hit Alt + w. The text is now copied to the tmux clipboard. You can paste it any time with Ctrl + b and then ] (ei. close bracket). You can manipulate the clipboard in many useful ways, read the documentation for further details.
If the above keybindings seemed weird, you're probably not too familiar with emacs. By default tmux uses the emacs style keybindings in copy mode. You can use vi style instead if you want, just add set-option -g mode-keys vi to ~/.tmux.conf. You can now copy-paste by hitting Ctrl + b and then [ to enter copy mode. Start marked text with Space, end it with Enter. Finally paste it with Ctrl + b and then ].
Now, I know I promised that I wouldn't talk much about system administration. But if you are going to use the console as a desktop, you still need to know how do do some basic things, like shutting down the system and so on. In the following paragraphs we will therefore consider some basic sysadmin topics. There sections are quite verbose since we are dealing with multiple operating systems, but again, you only need to read whatever is relevant for your setup.
There are various ways to shut down a computer, many systems have halt and reboot for instance. These commands are often just aliases for the shutdown command, which is extremely ubiquitous (even Windows has it for crying out loud!). But it seems like every incarnation of this command behaves a little differently:
One of the popular command line thrills, is to watch colorful streaming text, of presumably important information. The granddaddy monitoring application, is of course top. Which is the closest thing you will come to a taskmanager for the terminal (there is also the slightly more hip variant, htop). This ubiquitous tool can also run in batch mode, which is useful if you need a free command on non-Linux systems: top | grep Mem. The BSD's ship with a very nice system monitoring app in addition, called systat. There are many other monitoring apps available, glances and btop are two examples, but feel free to search your repository for "top" or "mon" to discover more. Personally I don't find these programs all that useful, but they are handy when you want to make a cool screenshot of your terminal, or if you need to pretend that you are doing something important at work ;^)
To check overall disk usage run the df -h command. You can also check how much space a directory uses with du -hs mydir, or recursively check how much space each file in this directory uses with du -ha mydir.
You can quickly check what disks are mounted on your system with mount | column -t, or by checking the systems disk configuration file /etc/fstab. Note that disks are handled very differently on UNIX then on Windows. In Windows each disk is its own top level directory beginning with its drive letter, such as C:\. In UNIX a disk is represented as a file in /dev and is usually called something like sda, and in contrast to Windows it can be mounted anywhere on the filesystem. For example a common configuration is to put the root filesystem, /, on a fast but small SSD harddisk, and have the home partition, /home, on a slower but bigger harddisk. Or to have partitions that change files rapidly, like /tmp or /var, on faster harddisks. Another very useful configuration is to place /tmp directly in memory, we will talk more about that later. The point is that you can arrange the disks anywhere you want in the filetree, and with new filesystems like ZFS, UNIX systems allow for even greater flexibility!
We cannot go into the finer details of every UNIX filesystem, but as an example, lets repartition and reformat a USB memory stick with the ultra archaic and portable DOS filesystem, and mount it:
After attaching the USB stick to your computer, you can run the dmesg | tail command to see if your system has detected it. You should be able to see something similar to this line towards the end:
[ 4198.053323] sd 8:0:0:0: [sdd] Attached SCSI removable disk
Don't worry too much about the details here, what's important is that the kernel detected a removable disk and called it sdd. Armed with this knowledge we can now reformat our disk:
# fdisk /dev/sdd
> p # Print a list of partitions that are on the disk
> d # Delete the partition(s)
> n # Create a new partition (just go with the defaults)
> t # Change the partition type to b - W95 FAT32
> p # List partitions and verify that everything looks correct
> w # Write changes to disk
> q # Quit fdisk
There are more user friendly alternatives to fdisk such as cfdisk, but the classic fdisk command is easy enough once you have used it a couple of times. Even though we have now created a new partition table on our memory stick, we haven't actually created a filesystem on it yet. Assuming the memory stick has only one partition, we can create a fat filesystem like so: mkfs.vfat /dev/sdd1
Now that our memory stick has a freshly reformatted filesystem, how do we use it? Well we need to mount it somewhere on the filesystem. For instance if you run the command mount /dev/sdd1 /home/myuser/Documents, you can access the memory stick from this directory. Of course that's a rather daft place to put it since you then cannot access your documents. So lets place it somewhere else (don't worry, your documents will pop back once we unmount the memory stick):
# umount /home/myuser/Documents
# mkdir -p /mnt/usb
# mount /dev/sdd1 /mnt/usb
And safely remove it by:
# umount /mnt/usb && eject /dev/sdd1
You can check what the usb stick is called with the dmesg | tail command, or just by logging into tty1 and see what the kernel has written there. On FreeBSD/DragonFly the device will be called something like da0 and the first partition is da0s1. On OpenBSD and NetBSD the device will be called something like sd0, and you can check what the partitions are called with disklabel sd0.
The fdisk command is slightly different on all the BSD versions, so read the man page before using it. On FreeBSD/DragonFly and NetBSD you can run fdisk -u <diskname>, and an interactive process will guide you through the partitioning (make sure you choose file type 11 on your FAT partition). In OpenBSD run fdisk -e <diskname> to edit the partition, use file type 0b for your FAT partition.
When all that is done you can use newfs_msdos <diskpart> to create a new FAT 32 filesystem on it (on FreeBSD/DragonFly <diskpart> might be da0s1, on OpenBSD/NetBSD it might be sd0i - check with disklabel).
On FreeBSD/DragonFly mount and safely remove it like so:
# mkdir -p /mnt/usb
# mount -t msdosfs /dev/da0s1 /mnt/usb
# umount /mnt/usb
# sync
On OpenBSD/NetBSD mount and safely remove it like so:
# mkdir -p /mnt/usb
# mount /dev/sd0i /mnt/usb
# umount /mnt/usb
# eject /dev/sd0i
After you have inserted the USB stick, you can look for its device name with the rmformat command. It should produce an output similar to this:
Looking for devices...
1. Logical Node: /dev/rdsk/c0t0d0p0
...
Connected Device: USB DISK 2.0 1219
...
You can now edit the partition table of the disk with fdisk /dev/rdsk/c0t0d0p0. The Illumos/Solaris fdisk program is similar to cfdisk on Linux and is straight forward to use. Just make sure you create a D partition type and specify its size as 100 (assuming you want a single FAT 32 partition on the usb stick).
Finally create a FAT 32 filesystem on this partition with: mkfs -F pcfs -o fat=32 /dev/rdsk/c0t0d0p0:c. The memory stick should automatically be mounted under /media, but you can manually mount it to a different location if you want:
# umount /media/<usbname>
# mkdir -p /mnt/usb
# mount -F pcfs /dev/dsk/c0t0d0p0:c /mnt/usb
And safely remove it by:
# umount /mnt/usb && eject /dev/dsk/c0t0d0p0
PS: The raw disk devices in Illumos/Solaris is under /dev/rdsk. It's this device you use when formatting a disk. When mounting a disk however you use the block device under /dev/dsk.
The archaic FAT filesystem from MS-DOS is awkward to use under the best of circumstances. It cannot store any files larger than 4 Gb, and even worse it has no concept of UNIX file permissions, these vital security settings will be wiped when you copy them over to a FAT memory stick. You can work around the permission problem by bundling your files into a tar archive, but it's a bit awkward.
So why are people still using this rusty old filesystem, aren't there any better alternatives?!? Well, yes and no. A popular choice is the newer Windows filesystem NTFS since it can store files larger than 4 Gb. Of course it too has no concept of UNIX file permissions, so for UNIX users this isn't a good alternative (And even though Linux can use this filesystem, there is considerably less support in the UNIX world for NTFS then FAT). Why not just use a native UNIX filesystem on the memory stick?
You can, and in many ways, it is much simpler. Tools such as mount and newfs for instance will assume a native filesystem by default. And of course you can store large files on it, and keep your permission settings. But there is a catch! Native UNIX filesystems have virtually no support on other operating systems, including different versions of UNIX! If you happen to use only FreeBSD machines at home and at work, then by all means, put UFS on your memory sticks! But don't be surprised when Solaris refuses to mount them, even though it too uses "UFS". The most portable UNIX'y filesystem is probably old versions of the Linux EXT filesystem, such as EXT2 or EXT3. (note that EXT2 might also limit max file size to a few Gb, depending on how it is configured)
For better or worse, DOS was a very simple and hugely popular operating system, so support for its filesystem is ubiquitous. There may not be many good alternatives to FAT, but there are good alternatives to memory sticks, such as sftp, scp or sshfs, which are all part of the ssh suit of programs. Another excellent tool for syncing files across the network is of course rsync. You can also set up permanent network file shares with NFS or Samba. And of course ZFS filesystems allow you to easily share files across the network with other ZFS filesystems.
UNIX systems can use filesystems that reside entirely, or partially, in memory rather then on disk. There are many ways you can use this, but we will only briefly look at how to configure your /etc/fstab to mount /tmp in memory. Bizarrely some UNIX systems don't do this by default, but assuming you have this thing called "RAM", there isn't any reason not to. For one, your /tmp directory will be much faster, and it is convenient for security too. So without further adieu, simply take the line below suited to your system, adapt it to your local needs.
Of course, these are merely suggestions. You may want to use a different size then 1 Gb, and you may want to add or drop some of the mount options here. Btw, DragonFly, NetBSD, Solaris, and a few Linux distros, I am sure, will use a tmpfs /tmp partition by default. PS: edit /etc/vfstab, not /etc/fstab, on Solaris systems.
If you are having permission issues after creating this memory filesystem, try the following commands (if they don't help, make a valiant quest to a Guru on top of a network stack, and plead politely for his wisdom):
# umount /tmp
# chmod 1777 /tmp
# mount /tmp
On Linux you can use the acpi command to check your battery status and the powertop command to check which nasty little interactive programs are sucking your battery dry.
You can check how much battery life you have left with acpiconf -i 0 | grep 'Remaining capacity' (your battery may have a different numerical ID then 0).
OpenBSD uses sysctl to list battery and temperature statistics, usually something like this will work: sysctl hw.sensors.acpibat0. Figuring out exactly how much time you have left on your battery require a bit of arithmetic, for example:
#!/bin/sh
# battery - print how much battery life is left
# usage: battery
remaining=$( sysctl hw.sensors.acpibat0 |
sed 's/.*=//' | awk '/remaining/ { print $1 }')
chargerate=$(sysctl hw.sensors.acpibat0 |
sed 's/.*=//' | awk '/rate/ { print $1 }')
left=$(echo "scale=2; ($remaining / $chargerate) * 60" | bc)
echo $left minutes of battery life left
NetBSD uses envstat to list battery and temperature statistics, we can make a simple script to digest this information succinctly:
#!/bin/sh
# battery - print how much battery life is left
# usage: battery
envstat | awk '/charge:/ { gsub(/[()]/,""); print "Remaining:", $6 }'
You can retrieve battery information with the kstat command, but like the OpenBSD solution above, you need to manually work out what this information means in practice. Here is a simple script that prints remaining battery percentage:
#!/bin/sh
# battery - print how much battery life is left
# usage: battery
lastfull=$(kstat acpi_drv | awk '/last_cap/ { print $2 }')
remaining=$(kstat acpi_drv | awk '/rem_cap/ { print $2 }')
echo $(echo "($remaining * 100) / $lastfull" | bc)% remaining
There is no such thing as a perfectly secure system, rather, it's a question of how much security you need. Installing a shark infested moat around your house may provide good protection against common burglars, but not against paratroopers. Pros and cons must be carefully weighed. Private citizens usually don't need to worry about paratroopers, but government officials might. Is the expense and maintenance worth it? Is the added protection worth the inconvenience of having your clumsy kids accidentally tripping into the moat from time to time? (this absurd analogy is more relevant to system administration - indeed parenting - then you may think) In that vein, we will not talk about over engineered security mechanisms such as ACL, SELinux, PAM, Docker and so on, we will only talk about security basics that all system administrators need to know, and know well. Naturally the importance of system administration is directly proportionate to the number of users on your machine. If you only administrate yourself, the following topics are merely important.
The core security concept in UNIX is managing user and group permissions to files. By setting file permissions you can include or exclude users from collaborating on common projects, from tweaking system configurations, and from running certain programs. Consider the following example:
$ ls -ld $HOME
drwxr-xr-x 81 dan dan 2560 May 26 15:38 /home/dan
$ ls -l /bin/sh
-r-xr-xr-x 3 root bin 617544 Apr 19 18:16 /bin/sh
$ chmod +w /bin/sh
chmod: /bin/sh: Operation not permitted
$ cp /bin/sh $HOME/sh
$ chmod +w $HOME/sh
$ ls -l $HOME/sh
-rwxr-xr-x 1 dan dan 617544 May 26 15:39 /home/dan/sh
The information we are interested in here is the first string of characters after the ls -l commands. Ignoring the very first character, they show us whether or not the file has read, write or execute permissions for the owner, the group and everyone else. /home/dan has owner dan and group dan, with permission rwxr-xr-x. That is, the owner dan has permission to read, write and execute, rwx. The group dan has permission to read and execute, r-x. And so does everyone else, the last r-x. So no one except the file owner in this case can edit this directory. But everyone can look at the files, and indeed make a copy of the files to their directory and freely edit the copies. This last point is illustrated above, where we first tried to give write permissions to /bin/sh (a rather daft thing to do!). This was denied because only the file owner, root, is allowed to do so. Hence we made a private copy of it, and then made the necessary adjustments (naturally the owner of a copy is whoever made the copy). Note the security implications here: Anyone who can read a file, can make a copy of it, and subsequently modify and execute the copy!
UNIX has always been an open share and share alike environment, business practices of AT&T not withstanding, and the default file permissions reflect this ideology somewhat. If we don't want to allow everyone access to snoop and copy our private files, we can simply run this command chmod 750 $HOME. With this set, our files are safe from prying eyes (well, not really - more on that later). If we want to grant other users the exclusive rights to read and copy our files, to work on some common project for instance, we can add them to the dan group. We could also have written the above command like so: chmod o-wx $HOME. This later form is perhaps easier to understand, it reads: for others (ei. everyone else) remove write and execute permissions. If we wrote ug+r, it would mean, for user (ei. the owner) and group give read permission. Often you will see the octal chmod form however, which can be tricky to understand, but you only need to know a handful of common values to get by:
Now running the command chmod 750 $HOME will indeed prevent (almost) anyone from tampering with our home directory, but what about all of the files within this directory? Actually, a 750 permission in $HOME is quite sufficient. But there are situations where it might be desirable to recursively set file permissions throughout a file tree. So lets assume that we want to set a 750 permission for all files and directories under $HOME, we could do so with the following command: chmod -R 750 $HOME. But this is a very sloppy thing to do! The problem is that we don't really want execute permissions for ordinary files, except of course our custom shell scripts and assorted programs in ~/bin, we want a 640 permission on plain files, and only a 750 permission for directories (directories NEED execute permission!). Setting unnecessary execute permissions on plain files will not mean the end of humanity, but it is an uncouth move, and it will earn you newb points! The correct solution would be something more akin to this:
$ find $HOME \( -path "*/bin/*" -prune \) -o \
\( \( -type d -a -exec chmod 750 {} \; \) -o \
\( -type f -a -exec chmod 640 {} \; \) \)
It's OK to admit that the above command scares you. We could simplify this operation by doing it in stages, and actually, it's fine if you just lazily type chmod -R in the privacy of your own home. I will not tell on you. Sloppiness isn't really a major issue unless you are working professionally as a sysadmin. If that is the case, the above command is nothing less then job security. Just make sure you run this whenever your boss is looking over your shoulder.
To illustrate user and group management, we create a new user, bob, a new group, manhat, assign Bob to this group, set the new user and group as owners for some files in /var/manhattan, and lastly make these files secret for anyone outside of the Manhattan project:
# adduser # bob
# groupadd manhat
# usermod -a -G manhat bob
# chown -R bob:manhat /var/manhattan
# chmod 770 /var/manhattan
# adduser # bob
# pw groupadd manhat
# pw groupmod manhat -m bob
# chown -R bob:manhat /var/manhattan
# chmod 770 /var/manhattan
# adduser # bob
# groupadd manhat
# usermod -G manhat bob
# chown -R bob:manhat /var/manhattan
# chmod 770 /var/manhattan
# useradd -m bob
# groupadd manhat
# usermod -G manhat bob
# chown -R bob:manhat /var/manhattan
# chmod 770 /var/manhattan
# useradd -m bob
# groupadd manhat
# groupmod manhat -U +bob # Oracle Solaris
# usermod -G manhat bob # Illumos
# chown -R bob:manhat /var/manhattan
# chmod 770 /var/manhattan
As you can see, there is a disturbing amount of variation in user management across UNIX systems (chown and chmod however work the same). And these differences are somewhat incompatible. For instance usermod -G manhat bob is fine in OpenBSD and NetBSD, but that command will nuke Bob's membership in any group besides manhat in Linux and Solaris, whereas FreeBSD will, "thankfully", only give you a not found error. It would also have been nice if we could call our group manhattan, but surprisingly enough, some Unixes (eg. Solaris) cannot handle long group names. But I suppose our little Manhattan project illustrate that system administration, often involves a level of surprise, pain, embarrassment, and occasional atrocities.
Beware that there are other, more subtle, peculiarities between these systems as well. BSD's have the unique concept of login classes, and Solaris has a thing about "profiles", for instance. As always, the sysadmins primary task is to read the manpages carefully!
PS: It is possible to adjust user and group settings by hacking the files in /etc/passwd and /etc/group as root, and in the heyday of AT&T UNIX this was in fact the only way to add new users to the system. But this kind of Wild West approach is considered bad practice today, however inconsistent and ugly, its still better to use dedicated sysadmin tools for such tasks.
It turns out that our top secret Manhattan project above isn't quite as secure as we may think. You see, UNIX systems have an all-powerful user called root. The root user can do anything, no restrictions whatsoever (of course the root user can set restrictions on himself, but he can also rescind them). This concept is a lot more powerful then a "system administrator", that you will see on some non-UNIX'y systems. The root user is not only able to configure the system any way he wants, but he can engage in deep brain surgery and wanton destruction as he pleases. In fact, anytime you see root in a UNIX context, replace it with the LORD, and you will get the right idea. Beyond file encryption it is impossible to hide anything from root. Windows users may find this disconcerting, but in actuality they have the exact same issue, it's just that their "root" user is an unknown, plausibly evil, corporate entity called Microsoft (MacOS and Android (and Ubuntu for that matter), being UNIX after all, have a real root account - but like Microsoft, they try very hard to hide this fact from their users).
You can switch to the root user by typing the command su (or su - to simulate a full login), followed by the root password. By the way, you can use this command to switch to any user you have the password for, eg. su bob. It goes without saying that your user passwords should be a well guarded secret. But the root password is something you do not share even with your closest confidant! To quote Micah 7:5:
«Trust ye not in a friend, put ye not confidence in a guide: keep the doors of thy mouth from her that lieth in thy bosom.»
With careful group management, it is quite possible to delegate system administration tasks to users without giving them root access. Users working on a web server may be part of the www group, in order to work on files in /var/www for instance. If these users also need to run some database, or other web server maintenance program, the sysadmin can just run chgrp www on these programs, and set permissions to 550, in order to allow www users to run them, but no one else. But root is so powerful, that the system administrator himself should not use it, if it can be avoided. To illustrate the danger, suppose you meant to type rm -rf $HOME/* but mistyped it: rm -rf $HIME/*. Congratulations! You have now deleted all files on your computer.
Another, more subtle example: Suppose you are working in /var/www/mysite, and decide to delete the whole mess in an act of righteous indignation: rm -rf ../* Permission denied! Oh really, you think smugly to yourself, and do a full root login with: su -, then rm -rf ../*. Congratulations! You have now deleted all files on your computer (do you see why?). The point? Don't use root! This is where sudo comes into play. In it's simplest form, you can edit /etc/sudoers, and uncomment the line that says #%wheel ALL=(ALL) ALL, that is, remove the # sign. Then make sure that your user is a part of the wheel group, eg. usermod -a -G wheel myuser in Linux (some distros use sudo instead of wheel here).
PS: Whenever you need to run some dangerous command, it's a good idea to check first with echo. For instance, the two disastrous file munching commands mentioned above, could have easily been averted with a simple sanity check: echo rm -rf ...
With this in place, you can run sudo command, to execute this one command "as root". That still makes you disturbingly powerful of course, but it is marginally better then monkeying about blindly as the Almighty root, at the very least it makes you open up the briefcase before you hit the big red button. sudo also logs any commands executed by it. So assuming the computer survives the attempt, you will know who to blame when the dust settles. The true beauty of sudo however, is that it allows you do define very fine grained privileges. The above line could have been bob 192.168.0.12 = (operator) /bin/kill, /usr/bin/lprm, for instance. Meaning that the user bob can run the commands kill and lprm on the machine 192.168.0.12, but only as the operator user (eg. sudo -u operator kill). You can define aliases for users and commands and such, in order to administrate things further. If we defined Host_Alias CSNETS = 192.168.0.12, 128.138.204.0/24, 128.138.242.0 for example, we could have written CSNETS instead of 192.168.0.12, to allow bob access to this list of machines.
PS: Not all variants of UNIX come with sudo by default, and some have their own sudo'y kind of commands (eg. pfexec in Solaris and doas in OpenBSD). But all UNIX systems have sudo in their repositories at least.
About a thousand years ago, when Vikings roamed the sea and pillaged unsuspecting monasteries, people used telnet to login to their remote UNIX machines. It was truly a barbaric time when "security" was largely an unknown concept. Today of course we use ssh to connect safely, and civilly, to our remote boxes. But simply having ssh will not magically make the world around you safe. You need to use this tool correctly. In a word, this means: disable password authentication.
By default, an ssh server will accept a password login from anyone who has an account on the machine. Naturally this has to be the default, it would be impossible to set up a remote server otherwise. But the very first thing you should do on your shiny new server, is to set up public key authentication and then DISABLE password authentication! Passwords are a good way to protect your laptop from benevolent coworkers, it is NOT a safe way to protect your server out there on the hostile internet. There exists a network of bots on the web, dubbed the "Hail Mary Cloud", that will systematically probe any ssh server it can find. This cloud will try to login to your server a few times from one IP address, then try again with another IP, and so on and so on... If you allow password authentication on your remote ssh server, it will get compromised, not if, but when.
The answer to this problem is public keys. To understand why, we can use a simple analogy: Suppose you have set up a secret Pirate club in downtown New York, with a big flashy neon sign that says "Secret Password Required Upon Entry!" Well, daily you'll be pestered with kids trying to guess the password, and eventually one of them will get it right, before long your top secret Pirate den is full to the brim with script kiddies. The point? Use keys. Anyone who now wants to sneak into your club, must first accost a Pirate and steal his key. That can still happen, but you are likely safe from kids at least. Naturally, the bouncer should still ask for a secret password, for added security against such an event. To create an ssh key on your laptop run ssh-keygen. Once this is done, copy your key over to the server with: cat ~/.ssh/id_rsa.pub | ssh myserver 'cat >>~/.ssh/authorized_keys'. When you now log on to your server with ssh, it should say: Enter passphrase for key.... When you have verified in this way that your key is working, go ahead and disable password authentication on your ssh server. Edit /etc/ssh/sshd_config, and change these options:
ChallengeResponseAuthentication no
PasswordAuthentication no
PubkeyAuthentication yes
From the server you can copy your authorized_keys file to other ssh servers with: ssh-copy-id, to allow your laptop to access these remote machines using the same key. However, you want to create unique keys for each physical laptop/workstation that you are using. It is possible to use one key for multiple machines, just as it is possible to use one physical key for many houses, but it is dumb thing to do! It is also possible to set up your ssh key without requiring a passphrase. This is actually more secure than using password authentication, but it is nevertheless bad practice. The passphrase isn't used to authenticate against the server, the key is used for that, rather, the passphrase is used to decrypt a key locally. The reason for this security measure is to protect your key in case it is stolen. If your laptop is compromised, and they are easily compromised, the intruder can use unencrypted keys to access your remote servers. If your keys are encrypted with a passphrase however, the keys are of no use to the intruder (just as a stolen Pirate key is of no use to a rug rat unless he can also manage to guess the bouncer's secret password).
Once you have enabled key authentication, you can simplify things a bit further by using an ssh agent. On the console, you can type ssh-agent sh (or use whatever shell you prefer), and then ssh-add. You will be asked to type in your passphrase. Once done, the decrypted ssh keys will be saved in memory, and subsequently used whenever you run an ssh command. So in effect, you type the passphrase once, and use ssh throughout the day without typing in any passwords. This convenient approach is not safe if you share your computer with multiple users. Before taking a break at work, type ssh-add -D to flush the keys, then retype ssh-add when you get back from lunch (and obviously - shut down the computer before you leave for the day).
PS: Unfortunately the method of starting an ssh agent varies considerably between desktop environments and login managers, not to mention Windows. So consult your operating systems/desktops documentation, if you want to set up an ssh agent outside of the text console (also, the ssh-agent command must be executed on each virtual console you wish to use it on). To be clear: it's not very hard to set up, but it's hard for me to document, since it varies so.
You can do a lot of stuff with ssh. You can use it to secure network traffic that isn't otherwise encrypted. You can do remote desktop work with it. You can set up your own private VPN, and you can mangle the network in all kinds of other interesting ways. These topics are well outside the scope of this article, if you are interested, check out SSH Mastery by Michael W. Lucas. Lucas has a wide array of other high quality books that security minded UNIX sysadmins will likely find both beneficial and humorous. Some suggestions are: Sudo Mastery, PAM Mastery, PGP & GPG and many others. Check out his website if you are interested.
Like SSH, things are not magically secure just because you have a password. You need to use passwords correctly. There are two steps involved. First, you want long passwords that isn't easy to break. A good way is to find some obscure text that is very unique to you, an old Christmas card from a long gone relative, a really obscure magazine from the attic of your grandfather, or just some random sentence that only you can understand. Do not use short passwords, however obfuscated, nor anything from pop culture. "He's dead Jim" or "p455w0rd" are not safe passwords! "Høyesterettsjustitiariussjefsassistenten e en bondeknølbavianbajas fra Hålogalandbispedømmes ytterste gudsførlatte feskevær!!!"* is a reasonably safe password (and easily remembered if you happen to be Northern-Norwegian). Seriously though, use obscure sentences for good password protection!
Secondly, and most importantly, use different passwords for different things. Do NOT use the same password over and over again (and for the love of MikeOS - do NOT write down your passwords on post-it notes!). But I can't remember my passwords! you may think. Of course you can't, nobody can. If you are trying to remember your passwords, you are doing it wrong! Use a password manager. A password manager is like a secure post-it note. It requires a password to open, and once opened, you can freely read and edit a list of your passwords. So in effect, you use one password to rule them all, and in the encryption bind them. If you are working on a graphical desktop, I recommend that you use KeePass, or one of it's variants (eg. KeePassXC). If you are working on the console, you can easily make your own password manager.
Traditional UNIX utilities are monochrome and boring. As we talked about in the configuration section, you can easily change the default font color to green or whatever, but that just makes all-grey command output appear all-green, big whoop. This vexes newbies to no end, and they often hunt frantically after new and fancy alternatives to standard utilities. That is of course one way to solve the problem, but it's fairly easy to add colors to standard utilities if that's what you want.
First of, the utilities you use may actually support colorful output, check for --color in the manpage. For example, the GNU (ie. Linux) commands cal, diff, dmesg, grep, hexdump and ls, among others, support colorful output. If these commands don't produce colors on your system, try running something like grep --color=auto. If that works, you can set this behavior permanently by adding an alias to your shell startup file.* Eg. add these lines to ~/.bashrc:
alias grep='grep --color=auto'
alias egrep='egrep --color=auto'
alias fgrep='fgrep --color=auto'
alias diff='diff --color=auto'
alias ls='ls --color=auto -G'
Other UNIX'es will not necessarily have such features. FreeBSD's diff, grep and ls commands does, as does ls in Solaris, grep in NetBSD,* and none in OpenBSD.** Nevertheless, you can usually install the GNU versions of these commands, check the repository for ggrep or gnu-grep or some such.
Colorful ls and grep is great, but what if you don't want to run GNU software? (since you hate scruffy beards so much) And what about the hundreds of other graybeard tools out there, how can we put some color in them? Well, one easy alternative is supercat. (many repos have it, and it's easy to compile yourself) The tool provides a handful of predefined color config files, for highlighting C source code, diff output and log files for instance. These defaults may or may not work well for you, but the real beauty of supercat is that you can easily write your own config files.
For example, suppose you want to sprite up the old top command, and make it look more like htop. Add the following file in ~/spcrc/spcrc-top:
Grey blk b ([a-z_][a-z0-9_/-]*|[KMG%])
Green grn (PID|USERNAME|PRI|NICE|SIZE|RES|STATE|WAIT|TIME|CPU|COMMAND)
Cyan cya ([0-9])[0-9][0-9][0-9][KMG]
Cyan cya (CPU[0-9s]|Real|Free|Cache|Swap|load averages|processes| up )
# add some contrast to the header section
Cyan cya b ,[[:space:]]+([0-9\.]+),
Cyan cya b ^([0-9]+) processes
Cyan cya b up ([0-9]+) days ([0-9:]+)
To use this color config run the command top | spc -t top. (after testing, you can add something like alias top='top | spc -t top' to your ~/.bashrc file)* You can also generate the output as a webpage, for example, top | spc -n -w -t top will produce this output. This command will only produce a static page of colorized top output. Could we perchance put this in a while loop and make our very own htop? We could... but you may as well use htop if you want something dynamic.
The supercat config files can look a little cryptic, but it comes with an excellent manpage (man spc) and annotated examples. The fields are; HTML color, terminal color, attribute (eg. b for bold, blank is regular), number of matches (eg. 2, blank is match all), type of matching (eg. s for string, blank is regex). The regular expressions will color anything inside parenthesis, and the matches are applied sequentially. But enough talk, let's color some more commands:
PS: The supercat config files uses a lot of white space and long regular expressions must be typed in a single line. In most of the following examples I've condensed the code and broken up long lines with escape+newline, for better readability. Copy pasting may not work, follow the format specified in the config files that supercat provides.
# ~/.spcrc/spcrc-df
Grey blk b ([A-Za-z_/][A-Za-z0-9_/-]*|[KMG%])
Green grn (Filesystem|K?-blocks|Size|Used|Avail.*|Capacity|Mounted on)
Cyan cya ([0-9])[0-9][0-9][0-9][KMGT]
# highlight thousands, millions, billions, G and T bytes
Cyan cya ([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya ([0-9])[0-9][0-9]([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya ([0-9])[0-9][0-9]([0-9])[0-9][0-9]([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya [0-9]+(G)
Cyan cya b [0-9]+(T)
# ~/.spcrc/spcrc-du
Grey blk b (.+)
White whi (^[0-9.]+)
# highlight thousands, millions, M and G bytes
Cyan cya ([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya ([0-9])[0-9][0-9]([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya [0-9]+(M)
Cyan cya b [0-9]+(G)
# ~/.spcrc/spcrc-ps (for openbsd ps with -alx flags)
Grey blk b (.+)
Green grn (UID|PID|PPID|CPU|PRI|NI|VSZ|RSS|WCHAN|STAT|TT|TIME|COMMAND)
White whi ([0-9]+)
# add some contrast
Cyan cya [[:space:]]([0-9])[0-9][0-9][0-9][[:space:]]
Cyan cya [0-9]([0-9])[0-9][0-9][0-9][[:space:]]
Grey blk b ^[[:space:]]+[0-9]+[[:space:]]+[0-9]+[[:space:]]+[0-9]+\
[[:space:]]+([0-9-]+[[:space:]]+[0-9-]+[[:space:]]+[0-9-]+)
# make columns before/after time black
Grey blk b ([^\ ]+)[[:space:]]+[0-9]+:[0-5][0-9].[0-5][0-9][[:space:]](.+)$
# ~/.spcrc/spcrc-dmesg (for openbsd)
Brown yel ^([^:]+):
Brown yel ^([^\ ]+):[[:space:]]
Red red b [^\ ]+(:[[:space:]].*[Ee]rror.*)
# ~/.spcrc/spcrc-dig
Brown yel s CNAME
Green grn b [^A-Za-z0-9_-](A)[^A-Za-z0-9_-]
Blue blu b (^[A-Za-z0-9_\.-]+)
# ~/.spcrc/spcrc-diff
Cyan cya (^[0-9,]+[dc][0-9,]+)
Cyan cya (^\-\-\-[[:space:]]+$)
Red red (^<.*)
Green grn (^>.*)
# context diff (bug: color codes reversed)
Cyan cya (^\*\*\*\*+)
Cyan cya (^\! .+)
Red red (^\-\-\-[[:space:]]+[^\ ]+.+)
Green grn (^\*\*\* .+)
# unified diff
Cyan cya (^@.+)
Red red (^-[A-Za-z0-9_\.].+)
Green grn (^\+.+)
# ~/.spcrc/spcrc-calendar
Brown yel ^([A-Za-z]+ [0-9]+ )
Red red b s IMPORTANT
Yellow yel b ([01]?[0-9]:[0-5][0-9])
Green grn (file:.+)
These config files are not fancy, I have just aped the colors of htop, dog and GNU dmesg/diff. You may also need to adjust some of these examples to suit your own systems idiosyncrasies, the ps and dmesg configs in particular. But hopefully the examples illustrate that it's fairly easy to sprite up basic commands with a little regex know-how. The last example here is worth a comment. calendar will let you specify the date in a very flexible way. It would be a nightmare to write a regex to match all possible date formats, so we don't. We just assume that the user writes all dates as Month date (eg. Jan 1). We also highlight strings such as "IMPORTANT", "10:20" and "file:misc/memo.pdf" (and we assume this last string is always at the end), even though calendar has no awareness of such conventions. The point? Coming up with a meaningful convention and sticking to it, is often more powerful in UNIX-like environments then finding a fancy new tool.
Let's try something a little more dynamic though. The GNU cal command can highlight the current date, a very handy feature that classic UNIX cal can't. In order to highlight cal with supercat we need to write a script that writes the supercat config file, since it will change depending on the date, here is one solution: (as an exercise - how would you scan upcomming calendar(1) events and mark the calendar with these dates?)
#!/bin/sh # hcal - highlight current day in cal(1) # usage: hcal date=$(date +%e) mkdir -p $HOME/.spcrc cat <$HOME/.spcrc/spcrc-cal Red red b ^($date)[[:space:]] Red red b [[:space:]]($date)[[:space:]] Red red b [[:space:]]($date)$ EOF cal | spc -t cal
Another strategy is to write a wrapper script that extends the functionality of a command, but otherwise leave it intact. For example, we could colorize the default output of date, but print the output verbatim if the user specifies any arguments:
# ~/.spcrc/spcrc-date (the config file)
Brown yel ^[A-Za-z]+ ([A-Za-z]+ [0-9]+)
Yellow yel b ([0-2][0-9]:[0-5][0-9]:[0-5][0-9])
# ~/bin/date (the wrapper script)
#!/bin/sh
# date - a colorful wrapper for date
# usage: see date(1)
if [ $# = 0 ]; then
/bin/date | spc -t date
else
/bin/date $*
fi
Generally though, it's better to invent a new command name for such additional features. (eg. hdate) Other programs way well use date without your knowledge, and get confused when it gets a bucket load of escape codes. But I do hope you've found these examples inspiring. Of course, colorizing everything yourself will quickly become tedious, and really, supercat is not well suited for very complex tasks, such as adding syntax highlighting for more languages then the provided C and Python. To illustrate the point, we can port nano's syntax highlighting of shell scripts and save it to ~/.spcrc/spcrc-sh:
# shell syntaxt highlighting, based on nano's syntax file
# function declaration
Green grn (^[A-Za-z0-9_-]+)\(\)
# keywords
Green grn [^a-z](break|case|continue|declare|do|done|elif|else\
|esac|eval|exec|exit|export|fi|for|function|if|in|let|local|read|\
return|select|shift|then|time|until|while)[^a-z]
Green grn [^a-z](break|case|continue|declare|do|done|elif|else\
|esac|eval|exec|exit|export|fi|for|function|if|in|let|local|read|\
return|select|shift|then|time|until|while)$
Green grn ^(break|case|continue|declare|do|done|elif|else\
|esac|eval|exec|exit|export|fi|for|function|if|in|let|local|read\
|return|select|shift|then|time|until|while)[^a-z]
Green grn ^(break|case|continue|declare|do|done|elif|else\
|esac|eval|exec|exit|export|fi|for|function|if|in|let|local|read\
|return|select|shift|then|time|until|while)$
# symbols, comparisons
Green grn ([][{}\(\);:|`$<>!=&\])
Green grn [[:space:]]-(eq|ne|gt|lt|ge|le|ef|ot|nt)[[:space:]]
# short and long options
Magenta mag b [[:space:]](-[A-Za-z]|--[A-Za-z-]+)[[:space:]]
# common commands
Blue blu b [^a-z](awk|cat|cd|chgrp|chmod|chown|cp|cut|echo\
|env|[ef]?grep|head|install|ln|make|mkdir|mv|popd|printf|pushd|rm\
|rmdir|sed|set|sort|tail|tar|touch|umask|unset)[^a-z]
Blue blu b [^a-z](awk|cat|cd|chgrp|chmod|chown|cp|cut|echo\
|env|[ef]?grep|head|install|ln|make|mkdir|mv|popd|printf|pushd|rm\
|rmdir|sed|set|sort|tail|tar|touch|umask|unset)$
Blue blu b ^(awk|cat|cd|chgrp|chmod|chown|cp|cut|echo\
|env|[ef]?grep|head|install|ln|make|mkdir|mv|popd|printf|pushd|rm\
|rmdir|sed|set|sort|tail|tar|touch|umask|unset)$
Blue blu b ^(awk|cat|cd|chgrp|chmod|chown|cp|cut|echo\
|env|[ef]?grep|head|install|ln|make|mkdir|mv|popd|printf|pushd|rm\
|rmdir|sed|set|sort|tail|tar|touch|umask|unset)[^a-z]
# variable names
Red red b (\$[-@*#?$!0-9]|\$[[:alpha:]_][[:alnum:]_]*)
Red red b (\$\{[#!]?([-@*#?$!]|[0-9]+|[[:alpha:]_]\
[[:alnum:]_]*)(\[([[:blank:]]*[[:alnum:]_]+[[:blank:]]*|@)\])?\
(([#%/]|:?[-=?+])[^}]*\}|\[|\}))
# comments
Cyan cya [[:space:]](#.*)
Cyan cya ^(#.*)
# strings (bug: no multilines)
Brown yel b ("([^"\]|\\.)+"|'([^'\]|\\.)+')
As you can see, this is fairly horrid. Worse, it is also, like nano's highlighting, a bit buggy. (it doesn't handle multiline strings for instance) vim handles shell syntax highlighting much better, but it also uses ~1000 lines of code to make it work. Multiply that with the ~1000 different programming languages it supports highlighting for, and you get a an inkling of why some editors don't do syntax highlighting. So in conclusion, save yourself a lot of hassle and just use vim or bat if you need colorful code ;)
PS: If you do write a config file for highlighting a programming language, make sure to email the developer! (you'll find contact information on his web page)
Although supercat is a handy utility for quickly adding sloppy colors to your output, it does suffer from supporting only rudimentary colors and for being somewhat buggy. There are times when you need a more hands on approach. As we already touched on in the configuration sections above, you can manually add colors by writing special escape sequences to the terminal. To illustrate the point further, we can write yet another implementation of lolcat using only awk and printf:
#!/bin/sh
# lolcat.sh - convert input to color gradient
# usage: lolcat.sh < input
awk '
BEGIN {
col[1]="\\e[1m\\e[32m\\e[40m" # green bold
col[2]="\\e[1m\\e[33m\\e[40m" # yellow bold
col[3]="\\e[0m\\e[33m\\e[40m" # yellow
col[4]="\\e[0m\\e[31m\\e[40m" # red
col[5]="\\e[1m\\e[35m\\e[40m" # magenta bold
col[6]="\\e[0m\\e[35m\\e[40m" # magenta
col[7]="\\e[1m\\e[34m\\e[40m" # blue bold
col[8]="\\e[0m\\e[36m\\e[40m" # cyan
col[9]="\\e[1m\\e[36m\\e[40m" # cyan bold
initc = int(9 * rand()) + 1 # random start color
FS = "" # count every character
}
{
# advance the colors a little bit each line
i = NR
c = initc
if (NF > 1)
while (i > 10) {
i = i - 10
c++
if (c > 9)
c = 0
}
printf("%s", col[c])
# cycle through the colors every 10th character
for (j=1; j<=NF; j++) {
i++
if (i > 10) {
i = 0
c++
printf("%s", col[c])
}
if (c > 9)
c = 0
printf("%s", $j);
}
printf("\n");
}' | sed 's/%/%%/g' | xargs -0 printf
# reset terminal colors
printf '\e[0m'
We can improve on this design; ANSI color codes are used for portability, although modern terminals support greater variety. (see generic Linux configuration above) xargs will choke with copious amount of input, and the script will only colorize static stuff. (so you need to look elsewhere to pimp your steam locomotive) Finally, backward slashes ("\") in the input will confuse printf.
Yet it's hard to beat the simplicity here, and it competes well with the superfast C version of lolcat. Surprising, I know, but this isn't the first time I've seen shell scripts hold their own against C. 40 years ago you could count on shell scripts being 4 times slower then your C code, but swelling libraries and stupidly fast hardware in modern times have eroded the difference. Of course the shell has its limitations, but for workaday problems a shell script shouldn't be your last solution, though it may well be your first and last.
Another problem with using the console as a desktop is that things aren't interactive, there's no menu for finding applications or clickable icons for launching your favorite programs, even just a flippin progress bar for your cp would be nice! Settle down, there are solutions for all of this and more...
Although you cannot create graphical icons for launching a program in the console, you certainly can create shortcuts. For example, you can add this to your ~/.profile (or ~/.bashrc if you plan to use them from the desktop):
alias web=links
alias edit=nano
alias files=mc
alias play=mplayer
alias picture=fbi
alias game=myman
You can now edit files by typing edit, play video and music with play, view your files with files, and so on. If you are having trouble remembering your own aliases, just run the command alias to see a list of them. If you are having trouble remembering that, write it down in ~/help, and make this alias:
alias help='cat ~/help'
Aliases are also useful for running a command with a default set of flags, for example I usually set alias lynx='lynx -accept_all_cookies -assume_charset=utf8 -tagsoup', which makes the console browser lynx behave nicer. Aliases are nice for oneliners, but for serious automation you need shell scripts. A good tip is to place this in your ~/.profile (or other appropriate place): export PATH=$HOME/bin:$PATH. This line adds $HOME/bin to your list of default program paths, you can now write your own private shell scripts in the bin directory of your home folder, and launch them from anywhere. Note that we placed $HOME/bin first, this will let us easily overwrite system defaults. (which may or may not be a good idea depending on your skill)
Serious powerusers do not use menus, like an efficient dictator in a banana republic, he simply states his will in clear unmistakable language and watches it unfold, he does not ask to see the menu first! But suppose you are a sysadmin in a company, and the above set of aliases are just too hard for your colleagues to remember, and the help alias too boring. The only way to get these poor fellows to launch an application is to provide a menu. Can we accommodate them?
Certainly, if all you need is a static menu, then select might do the trick (PS: not all UNIX versions of sh support select - and not all versions of select behave in the same manner). We'll call our application launching menu apps:
#!/bin/sh
# apps - launch applications
# usage: apps
selectfile(){
echo -n "File: "
read file
if [ ! -f "$file" ]; then
echo "Error: file doesn't exist!"
exit
else
exec "$1" "$file"
fi
}
select prog in Web Edit Files Play Picture Game; do
case $prog in
Web) echo -n "URL: "
read url
exec links $url
;;
Edit) selectfile nano ;;
Files) exec mc ;;
Play) selectfile mplayer ;;
Pict*) selectfile cacaview ;;
Game) exec vitetris ;;
*) echo "Please choose a number!"
esac
done
Even if your work colleagues are reluctant to type help, you can force them to read instructions in various ways. You can write a welcome message that all users will see as they login to the system, by putting the text in /etc/motd (motd: "message of the day"). If a user needs specific instructions, the sysadmin can add something like this to his ~/.profile (or ~/.bashrc), when the account is initially created: cat /path/to/some/instructions, or even just: echo type apps to launch applications
If you want a more "GUI-like" experience though, dialog is your friend. This program can create a large set of standard GUI widgets with ncurses graphics, such as radiolists, progressbars, file and calendar selections, etc... You can even play Microsoft and create a whole chain of "are you sure", "are you REALLY sure" yes-or-no boxes to make that professional vibe and annoy your users to no end. The select menu example in dialog would be:
#!/bin/sh
# thing - launch applications, using ncurses
# usage: thing
tmp=/tmp/menu
while true; do
dialog --menu "Startup Menu" 0 0 0 1 "Web" 2 "Edit" 3 \
"Files" 4 "Play" 5 "Picture" 6 "Game" 2> $tmp
if [ $? = 1 ]; then
exit
fi
selectfile(){
dialog --title "Files" --fselect $HOME/ 0 0 2>$tmp
if [ $? = 0 ]; then
file=$(cat $tmp)
exec "$1" "$file"
fi
}
prog=$(cat $tmp)
case $prog in
1) dialog --title "URL" --inputbox \
"Enter website address:" 0 0 2> $tmp
if [ $? = 0 ]; then
url=$(cat $tmp)
exec links $url
fi
;;
2) selectfile nano ;;
3) exec mc ;;
4) selectfile mplayer ;;
5) selectfile cacaview ;;
6) exec vitetris ;;
esac
done
There are of course even more alternatives. whiptail is a simplified dialog, whereas gum is a radically new and cool tool, written from scratch in Go.
Most UNIX console programs follow "the rule of silence". This rule states that the users time and concentration is valuable, so do your work silently and only report back if something went horribly wrong, don't pester him with trivia. To be frank, this concept is very hard for a casual desktop user to grasp. You assume that my time is valuable, what kind of lame excuse is that, how dare you give me the silent treatment?!? Imagine shouting at a respectful butler for delivering your letters like you asked him to, without calling every other second to report how many steps he had taken towards the mailbox! Truthfully, modern computer systems have become little more then narcissistic training boxes. Experienced UNIX users actually enjoy a system that doesn't constantly second guess them or bother them with irrelevant information. The tranquility and peace of mind that the rule of silence provides is in fact one of the main benefits of using UNIX. Sometimes however you need to know how a command is progressing, is there a progressbar or something for the console?
Yes. First though, if you are lucky enough to use FreeBSD, you can hit Ctrl + t at any time, to see how a program is progressing. GNU (read Linux) utilities does not have this feature, but there is a program, appropriately named progress, which will tell you the progress of GNU coreutils, such as cp or mv. Of course, if you just need to verify that a lengthy dd job is actually doing something to the disk, iostat and other monitoring apps should suffice.
You can also create your own progressbar with pv (pipe viewer), it functions like cat, but reports its progress. So pv bigfile > bigcopy, will copy the file with a progressbar. You can also stick pv in the middle of a pipeline and see how it's progressing, eg. cat file | pv -s $(du file | awk '{ print $1 }') | nc -w 1 somewhere.com 3000. In this example we need to tell pv how large the file is with -s $(du file | awk '{ print $1 }'). Without this pv will still report write speed, but it will not know when the job is nearing it's completion.
Pipelines can be interactively manipulated in other ways too. For example percol and fzf (fuzzy finder) will let you interactively select elements from the pipe input and send it down the line, eg. ls | percol | du -h. A somewhat similar program from the moreutils package, vipe, will let you edit the input with vi, once you exit the editor, the modifications will be sent down the pipe.
You must know how to read and write in order to use the console, but contrary to popular belief, you don't need to write much. Use the Tab key to auto-complete filenames. So don't type cd /really-long-AND-obscure_path/to_my_file-SomeWhErE, just type cd /r then hit Tab, if it doesn't complete fully, hit Tab again to get a list of alternative matching files, type one or two letters more and hit a final Tab. You should also learn to use some basic regex; Instead of typing the tedious command rm "Aqua/I'm A Barbie Girl.mp3" "Aqua/Doctor Jones.mp3", just run rm Aqua/*.mp3.
The bash-completion package will let your bash shell auto-complete more then just filenames. But for serious auto-completion power I recommend switching your default shell to zsh. (and for your grandma I recommend fish ;^)
Alright, you're thinking, so you can do quite a lot of functional things with the console, but to be honest: The reason I still want to use the desktop, is because it's fun! I want to goof around with wallpapers, screensavers, games, 3D desktop cubes, watch youtube videos, and you know what, I really enjoy constantly resizing my windows. Seriously, I get a kick out of it. You just can't do that in the console! Oh really? Check out the links. If on the other hand, you just want to go on the web, keep reading.
Running ifconfig is a quick way to check your network settings, pay particular attention to your network card names. In Linux the wired network card is often called eth0, and the wireless card wlan0, but not always. Other UNIX operating systems use different names. Pinging a website (eg. ping www.wikipedia.org) is a quick way to check that you're online, but remember that this usually doesn't work from a virtual machine. Connecting to the web over a wired network is often done with dhclient eth0 or dhcpcd eth0.
You may be using a version of Linux that doesn't have ifconfig, in which case you need to use ip instead. Here is a quick ip to ifconfig cheat sheet:
ifconfig ip a
ifconfig eth0 up ip link set eth0 up
ifconfig eth0 down ip link set eth0 down
ifconfig eth0 add 10.0.0.1/24 ip a add 10.0.0.1/24 dev eth0
ifconfig eth0 del 10.0.0.1/24 ip a del 10.0.0.1/24 dev eth0
How about connecting to a wireless network? On Linux the easiest way to do this is usually with nmcli, a command line front-end to NetworkManager, which should be available by default. You can list available networks: nmcli dev wifi list, or connect to a network: nmcli dev wifi con mynetwork password mypassword, and do other network managy things with it. (there's also nmtui an ncurses front end to NetworkManager)
If you aren't using Linux, or your version of Linux doesn't have NetworkManager (or you don't want to use it), you can use wpa_supplicant to connect to your wifi. First write a default configuration file in /etc/wpa_supplicant.conf:
network={
ssid="my home network"
scan_ssid=1
key_mgmt=WPA-PSK
psk="mypassword"
}
With this in place we can make our connection. First make sure your wifi card is activated: sudo ifconfig wlan0 up. Then start wpa_supplicant: sudo wpa_supplicant -B -c/etc/wpa_supplicant.conf -iwlan0. And finally request an IP address form the wireless router: dhclient wlan0. You should now be connected!
If you only plan on ever connecting to a single wireless network, then this is enough (actually in such a case you should probably invest in an ethernet cable). But if you need to connect to wireless networks arbitrarily, manually editing the wpa_supplicant.conf file every time is tedious. There are a couple of workarounds, firstly, you can use wpa_cli (as root) to reconfigure wpa_supplicant on the fly:
# wpa_cli
> scan # scan for available networks
OK
> add_network
0
> set_network 0 ssid "Buddys Impenetrable Wifi"
> set_network 0 psk "p4ssw0rd"
> enable_network 0
However, you might like me, find this solution somewhat ironic, since we are trying to reduce the tedium. It is easy enough to write a simple script, lets call it wifi, that reconfigures wpa_supplicant:
#!/bin/sh
# wifi - connect to a wifi network
# usage: wifi network password
wlan=${wlan:-wlan0}
sed -i -e "s/ssid=.*/ssid=\"$1\"/" -e "s/psk=.*/psk=\"$2\"/" /etc/wpa_supplicant.conf
ifconfig $wlan up
if (ps -e | grep wpa_supplicant | grep -qv grep); then
wpa_cli -i $wlan reconfigure
else
wpa_supplicant -B -c/etc/wpa_supplicant.conf -i$wlan
fi
dhclient $wlan
You can now connect to your buddy's network with: sudo wifi "Buddys Impenetrable Wifi" p4ssw0rd (or wlan=wlan1 sudo wifi "Buddys Impenetrable Wifi" p4ssw0rd if your device isn't wlan0). There are a few problems with this script though: It does not check for errors, such as badly written arguments. Running sudo wifi Buddys Impenetrable Wifi p4ssw0rd would fail without giving you any obvious error messages (do you see why?). It doesn't list available wifi networks, for that you probably need iwlist. And it stores your password as plain text, you should probably use wpa_passphrase to encrypt it. Finally, you may also need to use dhcpcd instead of dhclient. All of these issues can be addressed, but I leave that as an exercise for you.
As a side note, you should never configure your wireless router to accept connections without a password, or to use WEP encryption, which is more or less the same thing. If you do need to connect to a WEP router, use iwconfig not wpa_supplicant.
Update: Actually it turn out that you shouldn't use wpa/wpa2 either... hm... Maybe IP over carrier pigeons is the safe option? Of course that method has security implications too, such as cats.
You can use a password manager to store network passwords, and use a simple script to automatically search this encrypted database and connect to your network, as discussed in the password manager section below.
With the exception of OpenBSD, all the BSD's use wpa_supplicant to connect to wireless networks, as described above. Although the mechanism is basically the same, some of the particulars vary. The first issue in FreeBSD is simply enabling the wireless network card in the first place. You can run sysctl net.wlan.devices to get the name of your card, it might be something like iwm0. You can get more information by analyzing the output of pciconf -lv, or just by running dmesg | grep Wireless. This last command might return something like this:
iwm0: <Intel(R) Dual Band Wireless AC 8265> mem 0xec000000-0xec001fff irq ...
Now that we know we are using an Intel iwm 8265 wireless network card, we can go ahead and configure FreeBSD to load up the correct firmware and other necessities during boot, by adding these values to /boot/loader.conf (if you're unsure which firmware you need, you can just add all the ones listed in the iwm(4) manpage - substitute for your network card! - you also don't need the "legal" lines below unless you happen to use an ipw or iwi card):
if_iwm_load="YES"
iwm8265fw_load="YES"
legal.intel.ipw.license_ack=1
legal.intel.iwi.license_ack=1
wlan_wep_load="YES"
wlan_ccmp_load="YES"
wlan_tkip_load="YES"
Boy, that was a lot of work! From here on things get easier. The next step is to write a /etc/wpa_supplicant.conf file for your wireless network, it might look something like this:
network={
ssid="mynetwork"
psk="mypassword"
}
Finally, we can configure the system to start wpa_supplicant at boot. (sysrc, like rcctl in OpenBSD, provides a safer way to edit /etc/rc.conf)
# sysrc wlans_iwm0="wlan0"
# sysrc ifconfig_wlan0="WPA SYNCDHCP"
# service netif restart
You should now be connected to the wireless network! If not, try running ifconfig wlan0 up, service wpa_supplicant reload and service dhclient restart wlan0 a few times. This should not be necessary, but is, at times.
PS: Wifi support in FreeBSD is surprisingly bad, noticeably worse then the other BSD variants, even some alternative OS's. Perhaps this is due to FreeBSD's near exclusive server-side focus. (in contrast to OpenBSD's "eat-your-own-dogfoot" approach) In any case, FreeBSD documentation is upfront about this weakness, and the developers are working on it. If you're having a lot of problems, you can test the wifi in a GUI setup with Xfce and networkmgr, and look at wifibox for running the wifi through a virtual Linux machine. If you want a lappy that "just works" however, and you don't care about wine, ZFS and bluetooth, you can always give OpenBSD a try.
Use ifconfig wlan0 list scan to list available wireless networks. You can switch to another wireless network manually by editing /etc/wpa_supplicant.conf, and then run service netif restart again. You can use the aforementioned wpa_cli tool to do this, or we can make a short wifi script:
#!/bin/sh
# wifi - connect to a wifi network (FreeBSD edition)
# usage: wifi network password
sed -i '' -e "s/ssid=.*/ssid=\"$1\"/" -e "s/psk=.*/psk=\"$2\"/" /etc/wpa_supplicant.conf
service netif restart
# the following lines should not be necessary
ifconfig wlan0 up
service wpa_supplicant reload
service dhclient restart wlan0
PS: The process of connecting to a wireless network in DragonFly BSD is exactly the same, with one exception: The DragonFly kernel is preconfigured to use any wireless devices currently supported, so just write your /etc/wpa_supplicant.conf and add a few values in /etc/rc.conf, and you are good to go!
Setting up a wireless network in NetBSD is easy-peasy, compared to FreeBSD at least ;^) You can auto-configure your wireless connection during installation, but in case you need to set things up manually, here's how; First write a configuration file for your wireless network in /etc/wpa_supplicant.conf, it might look something like this:
network={
ssid="mynetwork"
psk="mypassword"
}
Now add the following values to /etc/rc.conf (this example assumes your network card is called "iwn0", it may be called something else):
ifconfig_iwn0="up"
dhcpcd=YES
dhcpcd_flags="-qM iwn0"
wpa_supplicant=YES
wpa_supplicant_flags="-B -s -i iwn0 -D bsd -c /etc/wpa_supplicant.conf"
Now reboot, and you should be connected. Use ifconfig iwn0 list scan, or wpa_cli scan + wpa_cli scan_results, to list available wifi networks. To change to a different wireless network, you can edit /etc/wpa_supplicant.conf, and run service wpa_supplicant reload again. Naturally you can use wpa_cli to do this, or we can automate it with a NetBSD version of the above wifi script:
#!/bin/sh
# wifi - connect to a wifi network (NetBSD edition)
# usage: wifi network password
sed -i -e "s/ssid=.*/ssid=\"$1\"/" -e "s/psk=.*/psk=\"$2\"/" /etc/wpa_supplicant.conf
service wpa_supplicant reload
If you feel confused after reading the above sections, you should be! It is worth mentioning that OpenBSD is the only operating system in the UNIX world that connects to a wireless network in a sensible way: ifconfig iwn0 nwid 'my home network' wpakey mypassword followed by dhclient iwn0. And how do you scan for wireless networks? ifconfig iwn0 scan. (iwn0 is a typical wireless card name in OpenBSD, and em0 is a typical ethernet card name, but your devices may have different names). Other operating system developers could learn a lot from the OpenBSD weirdos :^)
Update: As of OpenBSD 7.0 you should run ifconfig iwn0 autoconf, instead of dhclient iwn0. You can also add something like this to /etc/hostname.iwn0, and it will just autoconnect to those networks whenever you are around them:
join workwifi wpakey allworkandnoplaymakesjackadullboy
join homewifi wpakey 12345
join cafewifi wpakey doubledecafnonfatsugarfreevanillalatte
inet autoconf
inet6 autoconf
As for Illumos it has it's own "NetworkManager" called NWAM, which if running will try to connect you to any known wifi or ethernet links automatically. You can also scan for wireless networks with dladm scan-wifi. To connect to a network, first register a password for it: dladm create-secobj -c wpa mypass You will be prompted for the password, mypass is just the key label, not the actual password. Once that is done, connect to the network like so: dladm connect-wifi -e 'my home network' -k mypass wpi0 (you don't need specify the wireless card if you only have one, wpi0 is a typical wireless card name in Solaris, and e1000g0 is a typical ethernet card name, but your devices may have different names).
It is possible to disable NWAM and use ifconfig manually. For instance, these commands will disable NWAM and manually request an IP address from your DHCP server (to switch back to NWAM, disable "default" and enable "nwam"):
svcadm disable network/physical:nwam
svcadm enable network/physical:default
# you may need to run these commands first (only once):
dladm show-phys # list physical network devices
ipadm create-if wpi0
cp /etc/nsswitch.dns /etc/nsswitch.conf
# now connect with DHCP:
ifconfig wpi0 dhcp
PS: If you're planning to use illumos/Solaris with wifi support, I'd recommend OpenIndiana, or the OpenIndiana (not OmniOS) version of Tribblix. The other illumos/Solaris variants are server oriented with low wifi priority.
The modern web browser is a beast, and not in a good way. In fact Firefox has about 25 million lines of code, and is bigger then the Linux kernel and the entire KDE4 suit of applications combined (as of 2019, as time goes by these statistics will only grow worse)! Expecting something like this to work in the text-based UNIX console is hopelessly unrealistic. Sadly this is one area where the console does not, and cannot, impress.
Having that said there are a number of basic HTML browsers available for the console. The classic choice is lynx, it's a bit like vi in that you need to learn a whole set of key-bindings before it becomes useful. But like vi it is very fast and efficient once learned. You can check these key bindings by starting lynx and typing ?. A more user-friendly alternative is links. It has a nice drop down menu accessible with Esc.
PS: None of these browsers support anything like JavaScript or Flash!
For a more graphical experience (Linux only) you can run links -g, if the framebuffer has been configured. w3m is also capable of rendering images in the framebuffer (in Debian you want the links2 and w3m-image packages).
There are a few examples of more advanced textual/framebuffer browsers, but none of them are really recommendable. elinks adds many advanced features to links, such as limited CSS and JavaScript capabilities. It has a troubled history, eventually being banned from OpenBSDs repository due to poor maintenance and security. Renewed interest in the project has resurfaced on GitHub in recent years however. netsurf-fb used to be a very nice graphical browser for the console, but Wayland dependence in later years has made it a desktop only option. Finally, browsh renders websites in ASCII art using Firefox as its back end. (use Ctrl-l to type in a new URL, Ctrl-w to quit) This too is only usable in a graphical desktop, and it's more of a gimmick then a useful tool.
Do not be too quick to dismiss lynx though, even if many sites will not work and you find the whole experience boring! The big graphical browsers out there are buggy, bloated and more often then not, boneheaded! lynx on the other hand is an invaluable tool for debugging your web server. It often helps to see what your website looks like in pure HTML, blind people using screen readers, for instance, will often see you site exactly as lynx presents it (shame on the web developers who discriminate blind folk and UNIX nerds alike!). This venerable old tool also does a supreme job of rendering sites as pure text. So if you want to print out a hard copy of the Wikipedia article about Slackware for instance, you could run the command: lynx -dump https://en.wikipedia.org/wiki/Slackware | lpr
Still not convinced? Here is a short demonstration of a simple web app printing today's weather:
$ lynx -dump http://weather.yahoo.com/united-states/illinois/chicago-2379574/
...
Current conditions as of 1:54 pm EDT
Mostly Cloudy
Feels Like:
32 °F
Barometer:
30.13 in and rising
...
$ cat sedcond
/IL, United States/{
n
p
}
$ cat sedtemp
/Feels Like/{
p
}
$ cat weather
#!/bin/bash
# weather - extract the current weather for Chicago, IL
# usage: weather
URL="http://weather.yahoo.com/united-states/illinois/chicago-2379574/"
LYNX=$(which lynx)
TMPFILE=$(mktemp tmpXXXXXX)
$LYNX -dump $URL > $TMPFILE
conditions=$(cat $TMPFILE | sed -n -f sedcond)
temp=$(cat $TMPFILE | sed -n -f sedtemp | awk '{print $4}')
rm -f $TMPFILE
echo "Current conditions: $conditions"
echo The current temp outside is: $temp
$ ./weather
Current conditions: Mostly Cloudy
The current temp outside is: 32 °F
Of course Yahoo might change their website and break our script. The point here is to understand the potential of lynx. Once you convert a webpage to pure text, you can feed it to the standard UNIX tools such as sed, awk, bc, etc...
Let's explore another example. This Web App prints daily quotations from quotationspage.com. It checks if the URL is valid or not, and writes a download log in /tmp/quote.log. You can run this script as a daily cron job, and configure your shell to print the daily quote, if you want:
#!/bin/sh
# dquote - download daily quote to /tmp/daily_quote.txt
# usage: dquote
quote_url=www.quotationspage.com/qotd.html
check_url=$(wget -nv --spider $quote_url 2>1&)
if (echo $check_url | grep -s '*error404*'); then
echo "Bad web address"
echo "$quote_url invalid"
echo "Exiting script..."
exit
fi
wget -o /tmp/quote.log -O /tmp/quote.html $quote_url
sed 's/<[^>]*//g' /tmp/quote.html |
grep "$(date +%B' '%-d,' '%Y)" -A2 |
sed 's/>//g' |
sed '/ /{n ; d}' |
sed 's/ //g' |
tee /tmp/daily_quote.txt > /dev/null
exit
Much of the details here has to do with cleaning up the HTML code, and printing only the text we want. If your having trouble following the logic here, I suggest you do the steps manually, one by one, and analyze how it transforms the output.
wget is great for retrieving data from the web, but it's cousin curl can also send data. In this example we use curl to send an SMS (ei. text) message to our phone (PS: This will only work within the US):
#!/bin/sh
# sms - send an sms
# usage: sms number message...
phone=$1
shift
SMSrelay_url=http://textbelt.com/text
curl -s $SMSrelay_url -d \
number=$phone \
-d "message=$@" > /dev/null
exit
If you happen to have a boring staff meeting in 30 minutes, you can send this message to your phone in 35 minutes, and thus get a convenient excuse to leave, like so: at -f 'sms 5553334444 Emergency - need you in the office now!' Now + 35 minutes
As the name suggests, netcat (or just nc), is really a "cat" for networks. It's an invariable tool for testing and debugging your servers. In this example we are going to recreate the classic UNIX write command, a dead simple chat program, using netcat on both machines. On the first machine run nc -l 1234, then on the second run nc <ip_address> 1234. That's it!
Just as the old write command our chat program is crude beyond words, it does not identify who is talking, and the text is garbled if both users write at the same time. The old UNIX write convention was that the user who initiated the conversation starts typing and ends with "(O)" for over, then the second user responds and ends with "(O)", and so on, until someone finally ends the conversation with "(OO)" for over and out. Of course it is possible to program a much more elaborate chat client using netcat, that identifies the users and handles simultaneous input, but that would kind of spoil the fun now wouldn't it ;^)
There is a whole host of web tools available for the console that we haven't covered. But I hope these examples have wet your appetite, and perhaps demonstrated how useful the console tools can be on the modern web. At least you don't have to learn some horrid JavaScript framework to use them.
In Linux land you often use wget to download single files from the web, and an FTP client like lftp to download files from an FTP server. In BSD the included ftp program handles both tasks. Torrents can be downloaded with a number of programs, such as transmission-cli (the command line frontend for transmission) or rtorrent.
wget can also be used to do quite complex downloads. For instance this command continues a previous ripping of a website, pretending to be Firefox and only downloading files at random intervals: wget --continue --random-wait -r -p -e robots=off -U mozilla www.somesite.com
PS: Piracy is a crime, anyone found guilty of it will be keelhauled!
The classic way to chat over the internet on a console is by IRC. Many UNIX developers and powerusers still use this old technology, and it's a great way to get free support. Two popular IRC clients for the console are irssi and weechat.
There are other clients that can handle other chat protocols. For instance, pidgin is a popular chat client on the desktop that can handle most chat protocols, such as Google talk, Facebook Messenger, MSN and what not. There are also Skype and Telegram plugins for it (text only though). Pidgin has a command line front end called finch, which uses the same libraries and configuration files, so any account and contact information made on one will also appear in the other. If you are into Slack, there is a terminal client for that chat service, called slack-term. And there are other clients besides, too numerous to mention, choose whatever trash can that fits your social garbage.
The standard program for reading and sending email in UNIX is mail. Like vi you need to learn a handful of keybindings to use it, but these are quite mnemonic. The mail commands can either be written in long format, such as print or short as in p. When you open mail it presents a numbered list of all your emails. You can print message 2 like this: print 2. When you are done reading it you can: delete it, reply to it, save /path/to/file it or move to the next email. You can also list the email headers again, that is, print a list of your email inbox. When you are done you can quit.
To type an email run mail -s subject person@email.address, you can then write your email and end it by hitting Ctrl + d (which in UNIX means "end of file"), or mail -s subject person@email.address < myemail. You can do other thing as well such as attach files to email messages, mark an email as junk, delete all junk mail, and so on... Read the manpage for the specifics. Of course this all assumes that you have configured your machine to function as an email server. The method of doing so is depressingly complicated, and highly dependent on what system you are using, and what choices you make. It is well beyond the scope of this article. (if you want to dig into that topic though, I recommend Run Your Own Mail Server by Michael W. Lucas)
You can however use offlineimap to sync your external email, such as GMail, to a local directory, and then use mail or mutt (or the newer fork neomutt) to read them. You can also use msmtp to send email via an external provider such as GMail. The exact configuration here depends on what kind of external email you are using, and other details, but it's relatively straight forward. You can find good examples out there on the net, and the Arch Linux wiki, as always, have some really good tips on the subject. The aforementioned mutt email client is also capable of using an external email provider all on its own. The following example shows how you can configure mutt to use GMail (Ps: I do not recommend using Google services, this is just an example). Put the following in ~/.muttrc:
# IMAP Settings
set imap_server = 'myuser'
set realname = 'My Real Name'
set from = 'myser@gmail.com'
set imap_pass = 'mypassword'
# Remote GMail Folders
set folder = 'imaps://imap.gmail.com:993'
set spoolfile = '+INBOX'
set postponed = '+[Gmail]/Utkast'
set trash = '+[Gmail]/Papirkurv'
set record = '+[Gmail]/Sendt e-post'
# Local folders for cached headers and certificates
set header_cache = ~/.mutt/cache/headers
set message_cachedir = ~/.mutt/cache/bodies
set certificate_file = ~/.mutt/certificates
# SMTP Settings
set smtp_url = 'smtp://myuser@smtp.gmail.com:587/'
set smtp_pass = 'mypassword'
# Securing
set move = no
set imap_keepalive = 900
# Handle HTML emails with w3m
auto_view text/html
And we need to create a couple of directories and add a mailcap line for this to work:
$ mkdir -p ~/.mutt/cache/bodies
$ echo 'text/html; w3m -I %{charset} -T text/html; copiousoutput;' >> ~/.mailcap
You can now read your GMail account from the console, we have even configured mutt to use the w3m console web browser for reading HTML emails.
There are a few rss/atom news client's available for the console, such as newsbeuter (aka. newsboat) and rssowl. Some UNIX systems also have the ancient news command available, this command reads any new text files added to /news or a similar place. If your system doesn't have a news command, you can easily create one. You can then either manually download news items and save them to the news directory, or you can write a script that automatically syncs this directory with some online news service and add it to your crontab. Here is the script:
#!/bin/sh
# news - read latest from /news
# usage: news
for i in $(ls -t /news/* $HOME/.news_time 2>&1); do
case $i in
*' No such file') ;;
*/.news_time) break ;;
*) set X$(ls -l $i)
echo "
$i: ($3) $6 $7 $8
"
cat $i
esac
done
touch $HOME/.news_time
This short script adapted from The UNIX Programming Environment on page 164, can teach you a great deal about shell scripting. It has a bug though, you must have at least one file in /news for the program to work properly. Naturally, this script is only useful in the extremely rare situation where colleagues are working on a common server, and actually use that server for communication and project management. This is the only sensible way to work with computers of course, but nobody outside of Bell Labs seems to be aware of it. If you like this kind of workflow, check out the Hipster Media section below.
In Linux there are command line clients for Facebook, Google, Readit and Twitter: fbcmd, googlecl, rtv, ttytter (and others). Of course real hackers update their twitter account like this: curl -u myuser:mypassword -d status="Twitting from my shell" http://twitter.com/statuses/update.xml Similar clients come and go, and those that do exist are naturally susceptible to breakage whenever the upstream service changes. Some service providers are even so radical that regular HTML browsers, such as links or w3m, can use them. But that is rare.
There are very good reasons why you shouldn't hand over your data to commercial companies that refuse to give you any control or insight into what they do with it. If the postman snoops in your private mail, you have valid grounds to call the police. But calling the police on Google isn't quite so easy. And besides sifting through the social garbage on Facebook, Instagram, Imgur and what not, will likely cause brain damage. Is there a way to create a local social network on the console, where the users are fully in control?
Sure, in fact the technical aspect of sharing data and communicating with a group of people, is remarkably easy. That's what UNIX was designed to do! Simply give your friends user accounts on your server with adduser and let them access it securely with ssh. No one can snoop on them now! (well, except big brother root)
You can see who is presently online with who, change and check your online profile with chfn and finger (finger will also print the users ~/.plan and ~/.project files if they exists), send them emails with mail. Sending emails internally usually doesn't require any special configuration (or at least minimal configuration), just make sure that a mail server, like sendmail or exim, is running in the background. Share files with cp or mv, control who has read and write access to what with chmod (if your collaborating on a project make a group for it and chmod -R myuser:mygroup /path/to/project).
Run a centralized news service by putting plain text news items in /news. You can even chat interactively with write or talk. PS: Users can enable or disable this service with mesg, and control whether or not they should be notified about incoming email with biff. talk may require some configuration (and there are more advanced clients for this ancient protocal). Of course the social challenge of getting your friends to use these programs will be borderline impossible. Don't expect to get many "likes" on your hipster network, but then again that wouldn't be the point now would it :^)
In Linux systems with alsa-utils installed, you can adjust the volume interactively with alsamixer, or set it statically with amixer -c 0 set Master 75%. (to set volume on sound card 0 to 75%) This should also work fine with PulseAudio or PipeWire as your sound server. It is possible to adjust PulseAudio settings directly with pactl/pacmd (see also pamix) or PipeWire with wpctl (wireplumber package), but you may as well use the old graybeard stuff. Other systems use other commands:
FreeBSD uses the mixer command, which adjusts volume in decimals from 1 to 0. So you can for instance do this: mixer vol.volume=0.75 (75%), or mixer mic.volume=+0.10 (+10%). We can make a simple wrapper script for this, so that we can write volume 75 instead of the clunky line above:
#/bin/sh
# volume - set audio volume
# usage: volume percent
output=$(echo "scale=2; $1 / 100" | bc)
mixer vol.volume=$(printf "%1.2f" $output)
You need to tweak /boot/loader.conf to get audio working in DragonFly, try:
snd_hda_load="YES"
snd_uaudio_load="YES"
If you still don't get sound, try loading all the audio drivers with kldload snd_driver as root (see sound(4)). If that works, you need to change the snd_hda_load line to match your audio driver. Once audio has been correctly configured, you can adjust the volume like so: mixer vol 75 (to set output to 75%)
There is a nice alsamixer-like package available in ports called cmixer. Otherwise you can adjust volume and other sound settings with mixerctl. For instance mixerctl outputs.master=255 will set the speakers to maximum output. This simple script will let you specify volume in percentage (eg. volume 75):
#/bin/sh
# volume - set audio volume
# usage: volume percent
output=$(echo "(255 * $1)/100" | bc)
mixerctl outputs.master=$output
NetBSD also has it's own alsamixer-like command called aiomixer, and like OpenBSD, it too has a mixerctl command that works much the same way. The command will not list the mixer settings if you run it without any flags though, instead you must run mixerctl -a. And to change settings you must use the -w flag: mixerctl -w outputs.master=195. Similarly to OpenBSD the NetBSD mixerctl specifies volume from 0 to 255, you can easily make the above mentioned OpenBSD volume command for NetBSD by tweaking the example a little.
You can change audio configuration with the audioctl command. For instance you should be able to set the volume to 50% by doing something like this: audioctl set-control volume 50. Exactly what controls are available may depend on your hardware, you can check with audioctl show-control (Read the full man page for further details).
As a side note: I had some hardware problems on my test machine. The first audio device /dev/sound/audiohd:0 did not work, but the second one /dev/sound/audiohd:1, did. Unfortunately the system choose the wrong card as the default! Well, no problem, this quick fix solved it: ln -sf /dev/dsp1 /dev/dsp. You can test your audio with audiotest.
If you have configured your framebuffer correctly in Linux, you can watch videos in the console with vlc -I ncurses movie.mpeg. You might get a garbled screen when the movie is finished playing, if so type reset. Another fine alternative here is mplayer. Like vi you need to learn a few key-bindings before you can truly appreciate this program. You can play a movie in the framebuffer like so: mplayer -vo fbdev2 -vf scale=1366:768 movie.mpeg. The resolution must match exactly the resolution of your framebuffer, this is usually the same as your maximum X resolution (you can check this in X with the command xrandr).
Some key-bindings to help you get started with mplayer: You can pause and unpause the movie with Space, change volume with 0 and 9, jump forward or backward with arrow keys or pageup or pagedown, and quit with q.
PS: MPlayer is an old multimedia player that has seen better days, much of the development effort in recent years has moved over to the MPV fork. In theory you should be able to play movies on the console with mpv, but you probably need to recompile mpv with --enable-sdl and --enable-sdl2, and you may need to recompile sdl2 with --enable-video-directfb. I have not tested this myself.
There are a couple of ways you can watch youtube movies in the Linux console. One is to navigate to the video you want with a console browser, such as lynx. When you have found the right URL, hit G, now left click and mark the URL (assuming you have configured gpm). Hit Ctrl + c to quit lynx. You can now type youtube-dl and middle click to paste the URL and download the video. vlc and mpv can actually play such YouTube streams directly, and we can make a wrapper script for mplayer:
#!/bin/sh
# ytplayer - play YouTube stream
# usage: ytplayer url
vid=$(youtube-dl -q -f 18 $1)
mplayer "$vid"
Another program you could use is youtube-viewer. You can either run youtube-viewer -d to download the movies instead of watching them, and then later play them with vlc or mplayer. Or you can edit ~/.config/youtube-viewer/youtube-viewer.conf and set the appropriate mplayer or vlc flags to play the movies directly in the framebuffer.
Update: youtube-dl seems dormant, you might have better luck with the yt-dlp fork.
Besides vlc or mplayer, there is a ton of music players available for the console! A very simple one which I really like is moc ("music on console", sometimes renamed mocp to avoid a naming conflict with the MPD client mentioned below). Adjust volume with , (comma) and . (dot), display available key-bindings with ?. You can open a directory with i, and start playing music files here by selecting it and hitting Enter. Or you can add the song to your playlist with a, clear the playlist with C, when the playlist is populated correctly switch to it by Tab and hit Enter. Shuffle with S and repeat with R. cmus and mp3blaster are popular alternatives, and there are plenty of others. There are also several more basic audio players available. The sox package discussed below for instance, includes the play command, which is quite capable of playing any kind of audio file.
You can play internet streams with mplayer and other media players, but this requires you to find the correct URL streams. This is more tricky then you might think since streaming websites will usually not display these URL's directly. Often you need to snoop around in the website HTML code in order to find them. A good place to look for such streams is somafm.com. Here is a sample radio script written in dialog to get you started:
#!/bin/sh
# radio - listen to internet radio
# usage: radio
tmp=/tmp/radio
player="mplayer -really-quiet -playlist"
while true; do
pkill mplayer # stop any radio stream that might be playing
dialog --menu "Internet Radio Stations" 0 0 0 1 "In-Sound" \
2 "Bossa Beyond" 3 "DEF CON Radio" 4 "Groove Salad" \
5 "Drone Zone" 6 "Suburbs of Goa" 7 "Sonic Universe" \
8 "Left Coast 70s" 9 "Underground 80s" 10 "Lush" \
11 "PopTron" 12 "Secret Agent" 13 "Heavyweight Reggae" \
14 "Folk Forward" 2> $tmp
if [ $? = 1 ]; then
exit
fi
prog=$(cat $tmp)
case $prog in
1) station="In-Sound"; feed=insound;;
2) station="Bossa Beyond"; feed=bossa;;
3) station="DEF CON Radio"; feed=defcon;;
4) station="Groove Salad"; feed=groovesalad;;
5) station="Drone Zone"; feed=dronezone;;
6) station="Suburbs of Goa"; feed=suburbsofgoa;;
7) station="Sonic Universe"; feed=sonicuniverse;;
8) station="Left Coast 70s"; feed=seventies;;
9) station="Underground 80s"; feed=u80s;;
10) station="Lush"; feed=lush;;
11) station="PopTron"; feed=poptron;;
12) station="Secret Agent"; feed=secretagent;;
13) station="Heavyweight Reggae"; feed=reggae;;
14) station="Folk Forward"; feed=folkfwd;;
esac
# play selected radio stream
$player http://somafm.com/${feed}.pls &
dialog --msgbox "Playing from $station, hit OK to stop" 0 0
done
You can also set up your own radio stream with mpd, and play it locally with mpc, ncmpcpp or another MPD client.
Here is an example setup, place it in /etc/mpd.conf for a system wide service, or ~/.config/mpd/mpd.conf if you plan to run it as a user process:
db_file "~/.config/mpd/database"
log_file "syslog"
music_directory "~/music"
auto_update "yes"
playlist_directory "~/.config/mpd/playlists"
pid_file "~/.config/mpd/pid"
state_file "~/.config/mpd/state"
sticker_file "~/.config/mpd/sticker.sql"
Run mkdir -p ~/.config/mpd/playlists if this directory does not exist. You're now all set, just run mpd, and then you can launch an MPD client, such as ncmpcpp to play your music.
There are command line clients for these online services: despotify was a good Spotify client, but seems to have been abandoned, ncspot is another alternative. For LastFM and podcasts, you can use shell-fm and bash-podder (of course for podcasts you could just manually download them and use your favorite music player). These commands are rare to see outside Linux repositories, but they shouldn't be too hard to compile yourself.
You can also install Mopidy (Linux only), which can be configured to play music from Spotify and other providers as an MPD service, you can then listen to Spotify with one of the many MPD clients. For example, after mopidy and mopidy-spotify is installed, you can add this to /etc/mopidy/mopidy.conf (or ~/.config/mopidy/mopidy.conf if run as a user process):
[audio]
output = tee name=t ! queue ! autoaudiosink t. ! queue ! udpsink
port=5555
[spotify]
enabled = true
username = [username] # Must have Spotify Premium
password = [password]
bitrate = 320
You can then configure the MPD client ncmpcpp to use this, by adding this to ~/.ncmpcpp/config:
visualizer_fifo_path = "/tmp/mpd.fifo"
visualizer_output_name = "my_fifo"
visualizer_sync_interval = "30"
visualizer_in_stereo = "yes"
visualizer_type = "spectrum"
visualizer_look = "+|"
You can now launch the Spotify client like so: nohup mopidy & ; mkfifo /tmp/mpd.fifo ; while : do yes $'\n' | nc -lu 127.0.0.1 5555 > /tmp/mpd.fifo done & ; ncmpcpp If you don't care about visualization, and mopidy is already running as a service, it would suffice to just run ncmpcpp.
You will not find a massive studio software suit, complete with a pointy-clicky interface, for the console, but you can do basic productive audio/video work nonetheless.
To begin with, you can record audio with rec, which is included in the sox package, or you can use a more native approach:
For recording video see the screen capturing section.
The Jack Audio Connection Kit sound server is available on practically all modern Unixes, and can be used from the command line to organize and mangle your audio equipment. ffmpeg and sox can be used to edit audio and video in various ways. Here are some examples:
If you plan to do a lot of audio/video work in the command line, you may find it useful to create some aliases for these commands, or whatever variations of them you may need.
Ripping audio CD's can be done with cdparanoia (OpenBSD has it's own excellent cd tool, cdio), and vobcopy is a good DVD ripper (see the cd's, dvd's and bluerays section for more information). If you want to rip a radio stream, streamripper is a good candidate. Finally the mplayer package also includes mencoder, which can rip any media that mplayer can play, you can also use it to edit the video/audio stream in various ways.
A few image viewers can be used in combination with the framebuffer on Linux. My favorite is fbi. You can open a picture with fbi picture.jpeg, or a directory of pictures with fbi pictures/*, and navigate back and forth with n and p, and zoom in or out with - and +.
Some people recommend zgv to view images in the console, but I have never managed to get this program to work on my machines.
There is no "PhotoShop" for the console, but you can easily "make" pictures with a camera, and work with these on a console. Just use a memory card reader to retrieve the pictures, view them with fbi, and organize them into whatever directories and naming scheme you like.
The process of mounting a memory card is similar to mounting a USB memory stick, although the device names are different. Stick your memory card in the reader and run dmesg | tail to find out what the device is called. On my Linux box the command gave me this output:
[ 20.443125] mmc0: new ultra high speed SDR50 SDHC card at address e624
...
[ 20.493617] mmcblk0: p1
Don't worry too much about all the details here, the important thing to notice is that a new SD card was detected, and that a new device called mmcblk0 (with partition p1) was created. We can now mount the device and copy over the images:
$ sudo mount /dev/mmcblk0p1 /mnt/hd
$ cp /mnt/hd/DCIM/100NIKON/* Pictures/vacation_2018
As you can see my camera organized its pictures in DCIM/100NIKON, but yours may do things differently. Once mounted just use ls and find out (PS: If you are having trouble you can always buy a cheap usb memory card reader on ebay, you can then mount your memory card like an ordinary usb memory stick).
For scanners see the printers and scanners section.
You can also edit photos from the console, using the ImageMagick collection of tools (a good alternative here is GraphicsMagick). Here are some of its utilities:
These tools give you quite a lot of scripting power to manage your photos. It is well worth learning, even if you are only interested in using a desktop environment. You can find many good tutorials for this software suit online.
The fbi (aka. fbida) package mentioned above usually include fbipdf (a good alternative is fbpdf). This program can be used to view PDF's on the framebuffer like so: fbipdf document.pdf. Some old versions of fbi do not include fbipdf, in such cases you can usually use the inferior fbgs program.
If you don't have a framebuffer available, you can convert a PDF to HTML or plain text with pdftohtml and pdftotext, both of which are a part of the poppler (aka. poppler-utils) package. You can also use pdfimages from the poppler tools to extract images in a PDF. To demonstrate:
$ pdftohtml -s -i document.pdf
$ lynx document-html.html
$ pdftotext -layout document.pdf
$ less document.txt
Exactly how well this works depends very much upon the PDF in question. If you are lucky, the conversion is merely bad, often though, it's unreadable. Some PDF's are simply a collection of JPEG's of text, the poppler tools cannot convert such documents. You can try with ocrmypdf or other OCR solutions (see discussion below).
The poppler utils can do some basic editing as well, such as extracting pages from a PDF or merging pages into a single PDF, checking the metainfo and more. An alternative PDF editing tool is pdftk. Of course non of these tools are interactive, so they are mostly useful for basic batch editing and scripting.
Sadly OCR support, that is scanning a document and converting the image to plain text, is not well supported in the open source world. The only widely available tool is tesseract, here is a short demonstration of its usage:
$ scanimage --mode grey --resolution 300 > scan.pnm
$ unpaper -b 0.5 -w 0.8 -l single scan.pnm scan1.pnm
$ convert scan1.pnm scan.tif
$ tesseract scan.tif scan.txt
This example uses commands from the sane-backends, unpaper and ImageMagick packages. The results of this OCR depends much upon the quality of the scan, the higher the resolution the better, and it also requires you to have installed the correct language packages for tesseract (eg. tesseract-ocr-fra or similar if the document is French).
For the geeky console user, ASCII art can provide much fun. It's entirely frivolous of course, non-Linux consoles without video playback that could use a textual representation of graphics, either have no ASCII art support at all, or they don't have good enough font support to make it practical. If you install the libcaca (sometimes called caca-utils) package, you can view images with cacaview or convert them to text files with img2txt. The related libaa is used for black and white ASCII art, it includes the fun bb demo. You can even play videos in ASCII art with something like this: mplayer -vo caca myvideo.mkv, or vlc -V aa https://youtube.com/some_url. Speaking of video, hasciicam will let you play video output from your webcam in ASCII!
PS: The quality of the ASCII art depends on what fonts you use. The smaller the font, the greater the resolution. So before playing a video, or displaying a picture in ASCII, set your console to use the smallest possible font you have available (see console configuration section on how to do this).
Although your artistic painting skills may be limited on the console, you can freely draw ASCII art. There are a few programs designed specifically for this purpose, such as cadubi, but real pros do their ASCII painting in vi! (of course ed users scoff at such newbs)
The previously mentioned ASCII art section is strictly speaking a digression, since you really need fairly advanced consoles to use it well (either a Linux console or a terminal emulator running in X). But since we have already digressed, lets digress a bit further and briefly mention two retrofuturistic terminal projects that are truly amazing:
The first is cool-retro-term. This is a terminal emulator running under X, similar in functionality to, say xterm. But it allows you to emulate the look of ancient computer screens, such as the Apple II or the IBM 3278, and you can manually tweak effects, such as scanning lines, flickering, screen curvature and more. If you want a cool retro terminal, look no further! You will find this package in most UNIX systems repositories (on Solaris you need to compile it yourself - good luck with that).
The second project I'd like to mention, is notcurses. We have used curses, and the newer ncurses for a long time now to create pseudo-graphics in our terminals. Applications like mc, nethack and tmux are good examples of what I mean by "pseudo-graphics." But modern terminals can actually take things a lot further with color blending, adjusting transparency and allowing a full set of UTF-8 characters (including Braille characters and what not). The ncurses library cannot use such advanced features since it needs to be compatible with all kinds of ancient cruft. This is where notcurses comes into the picture. It's a library that pushes heavily on modern capabilities to produce surprisingly good graphics directly in the terminal. Check out the demo video and book from the developer. Even for non-programmers, this library can provide a lot of fun (it is available in the repos of several Linux distros and FreeBSD). The notcurses package includes the video player ncplayer, the neofetch-like ncneofetch, the ls command ncls which also renders pictures, and notcurses-demo that can display many different demos (eg. notcurses-demo x). For some truly ludicrous retrofuturism, you can run these applications in cool-retro-term, and thus make the really modern stuff look really old!
See the disk management section for more details about formatting USB memory sticks, in this section we will only talk about mounting such devices. The device name used in the examples are typical for these operating systems, but yours may be different (it may be sdb1 not sdd1 for example). After attaching your memory stick you can run dmesg | tail to check what the system called the device, or to see if there were any errors. Finally, depending on your security setup, you may need to use sudo to run some of these commands.
$ mkdir -p /mnt/usb # make a directory to mount the usb device in
$ mount /dev/sdd1 /mnt/usb
$ cp -r /mnt/usb $HOME/Downloads # use the usb device like a regular directory
$ umount /mnt/usb # safely unmount the device
$ mount -t msdosfs /dev/da0s1 /mnt/usb
$ umount /mnt/usb
$ disklabel sd0 # check the partition name
$ mount /dev/sd0i /mnt/usb
$ umount /mnt/usb
$ rmformat # check the partition name
$ mount -F pcfs /dev/dsk/c0t0d0p0:c /mnt/usb
$ umount /mnt/usb && eject /dev/dsk/c0t0d0p0
Ripping CD's and DVD's can be done with cdparanoia and vobcopy. Writing CD's and DVD's can be done with cdrecord (or dvd+rw-format) and growisofs, and making iso files can be done with mkisofs (or xorrisofs). OpenBSD has it's own program for ripping and writing CD's, cdio. We have already talked about playing music and videos in the Multimedia section above. But let's say a few words about finding and playing a DVD track. Unfortunately the way DVD's are organized vary greatly, and more often then not, defy conventional wisdom. lsdvd is a nice program that lists DVD tracks, and is helpful for sorting out this mess:
$ lsdvd
opening /dev/rcd0c for title
Disc Title: STAR_TREK_TOS_D6
Title: 01, Length: 00:00:30:160 Chapters: 01, Cells: 01, Audio ...
Title: 02, Length: 00:48:28:200 Chapters: 06, Cells: 06, Audio ...
Title: 03, Length: 00:48:24:800 Chapters: 06, Cells: 06, Audio ...
...
$ vlc dvd://2 # play the first episode on the dvd
Since the first track only had a 30 second length, we know that the first real episode is on the second track. We can use such logic to create a simple dvd script, that will automatically play the nth track from a DVD with a sizable length (~20 minutes or longer), using only mplayer:
#!/bin/sh
# dvd - play nth dvd track
# usage: dvd n
if [ ! $# = 1 ]; then
echo Usage: dvd n 2>&1 && exit 1
fi
track(){
mplayer -noconfig all -cache-min 0 -vo null -ao null -frames 0 \
-identify dvd:// |
grep '[0-9]_LENGTH=[0-9][0-9][0-9][0-9]' |
sed -n "$1p" | sed 's/.*_\([0-9][0-9]*\)_.*/\1/'
}
mplayer dvd://$(track $1)
We can now play our first Star Trek episode with dvd 1. Of course our elegant mplayer script ignores some messy details; To play this in the console mplayer need some more flags, as mentioned in the Multimedia section. You may also want to detect if there is a VIDEO_TS directory around, and add -dvd-device ., to play DVD's you have ripped to your harddisk with vobcopy. Finally, as mentioned, DVD organization is disturbingly illogical at times, so our script will not always work as intended.
But enough about playing DVD's, lets talk about making them. The following examples show how to create and mount ISO files, and how to burn such data to a CD/DVD:
$ mplayer cdda:// # play audio CD
$ mplayer dvdnav:// # play DVD video
$ eject # eject CD/DVD
$ cdparanoia -B # rip audio CD
$ mkisofs -o mydisk.iso /my/path # make a data ISO
$ mkdir -p /mnt/cd
$ mount /dev/cdrom0 /mnt/cd # mount CD
$ mount /dev/dvd0 /mnt/cd # mount DVD
$ mount -o loop mydisk.iso /mnt/cd # mount ISO
$ umount /mnt/cd # unmount CD/DVD/ISO
$ cp -r /mnt/cd $HOME/Downloads # rip ISO contents
$ cdrecord -scanbus # check name of CD/DVD burner
$ cdrecord mydisk.iso
$ cdrecord -dao -useinfo *.wav # write an audio CD
$ mkisofs -R -J -udf -iso-level 3 -o mydvd.iso /my/path # make a DVD ISO
$ growisofs -dvd-compat -Z /dev/cd0=mydvd.iso # write a DVD
There are software alternatives to the above examples, but you can do much the same operations with other UNIX systems, although the device names may be a little different. And they might require you to create a virtual device for the ISO file before you can mount it:
$ mdconfig -a -t vnode -f mydisk.iso -u 0 # make a dev for the ISO
$ mount -t cd9600 /dev/md0 /mnt/cd # mount ISO
$ mount /dev/cd0 /mnt/cd # mount a real CD/DVD
$ cdio cdplay # play audio CD (cdio is OpenBSD only)
$ cdio tao mydisk.iso # write data CD/DVD
$ cdio cdrip # rip audio CD
$ cdio tao -a *.wav # write audio CD
$ vnconfig vnd0 mydisk.iso # make a dev for the ISO
$ mount /dev/vnd9c /mnt/cd # mount ISO
$ mount /dev/cd0c /mnt/cd # mount a real CD/DVD
$ growisofs -dvd-compat -Z /dev/rcd0c=mydvd.iso # write a DVD
CD's should be auto mounted in /media, you can double check with rmformat.
$ mount -F hsfs mydisk.iso /mnt/cd # mount ISO
Practically all modern Unixes use CUPS and SANE to manage printers and scanners. Installing a printer from the console is tricky if you don't know what you are doing. So for this one task it's probably better to use a graphical desktop and configure CUPS using a modern web browser, such as Firefox. Just go to http://localhost:631 and enter your root password. Many UNIX systems have large collections of printer drivers available by default, but if you can't find your model, head over to openprinting.org and search for a driver. If you cannot find your exact model try to find a driver for a similar device. For example, I could not find a driver for my HL-2035 Brother printer, but I did find one for the HL-2030 that worked just fine. Once the printer is set up, installing it on other UNIX machines using the console is easy. Just copy the printer configuration file /etc/cups/printers.conf, and optionally other configuration files that might be relevant, such as the drivers in /etc/cups/ppd.
If you have configured your printer as the server default, you can use the following commands to print documents:
$ lpr document.pdf document.ps document.txt
$ enscript document.txt # print as postscript (looks nicer)
$ lpq # list printing jobs
$ lprm # remove a print job
PS: Some UNIX systems may already have a native lpr and related suite of applications. In OpenBSD for example, the native print command is /usr/bin/lpr, while the CUPS print command is in /usr/local/bin/lpr. In FreeBSD the CUPS command is called lpr-cups, to distinguish it from the native lpr command. Run which -a lpr to check if your system has more then one print command.
Scanners are also easy to use from the console, using the SANE collection of tools, for instance: scanimage >image.pnm. Most scanners should be plug-and-play, but check the documentation or Google around if you're having problems.
Don't expect the triple A games on your Steam account to work in the console! The console is not a popular gaming platform for obvious reasons, but there are a few simple games that will work. Some of my favorites are myman, nudoku and vitetris (or you can try basted if you like getting frustrated). There is also cpat for all kinds of solitaire fun, and the "racing" games moon-buggy and ztrack. ASCIIpOrtal is an amazingly cool game, even if it's quite obscure and unmaintained. You can find a lot more suggestions, and plenty of online TTY games here. The classic bsdgames and gnuchess (eg. type d2d4 then show board) might also provide some fun for meganerds.
There are precious few strategy games for the console, but here are a couple of good ol' classics: starlanes is an economic conquest game, and is great fun to play hotseat with a couple of friends. vms-empire (or just empire) is a lot more elaborate, it's sort of basic Civilization for the terminal. Its vi-like interface takes some getting used to though, and you will have to skim through the manual a few times. Here are some quick starting points: After typing a to enter "auto-mode" at the beginning of the game, and a again to produce armies in your city, type y to command your new unit to auto explore. To change a units orders type j to enter "edit-mode". From here you can cancel a units orders with k, or change a city's production with b, hit o to go out of "edit-mode". When you want to quit the game, hit o to go out of "auto-mode", and into "command-mode". This will take effect once you have completed your turn, so move any remaining units until the game finally asks you for a command. At this point you can type s to save, and q to quit. A final tip, you want to start this game with vms-empire -d0! Have Fun :^)
If you are running Linux a few SDL games can actually run in the framebuffer. One example is dosbox, this DOS emulator can run pretty much any old game from the early 90's. If you're a retro gamer like me, dosbox opens up a world of gaming for the console! You need to tweak the ~/.dosbox/dosbox-*.conf setting a bit for programs to work well. These are the settings I use:
fullscreen=true
fulldouble=false
fullresolution=1680x1050
windowresolution=1680x1050
output=overlay
usescancodes=false
keyboardlayout=us
aspect=false
scaler=none
Note that the resolutions must match your framebuffer resolution, which is usually the maximum resolution in X. (if the DosBox screen is pushed to the far left of a wide screen, you can try to reduce the resolution to center it - Eg. 1400x900 may work better on a 1600x900 screen) And output must be overlay with no fancy scaler, otherwise fullscreen resolution will not work. Also be sure to disable usescancodes and set a keyboard layout manually, since usescancodes do not work in the framebuffer!
scummvm might also works in the framebuffer (it did not in Debian), but getting the resolution right is not easy. You can try something like scummvm -g hq3x, hopefully this will work well enough.
The big gaming genre on the UNIX console is of course roguelikes. Roguelikes are 2D dungeon exploration RPG games. The name comes from the game rogue, which was the first game in this genre, created way back in the 80's (in fact curses was originally developed for rogue). Sometimes the ancient rogue, hack or larn games are included in the bsdgames package, but players today usually prefer more modern alternatives.
The arch typical roguelike of all time is nethack, which might very well be the greatest game ever created! It has been in continuous development for over 30 years (40 if you include the predecessors it is based on), and has surprising depth. You owe it to yourself to die at least once playing this murderously difficult dungeon crawler before you snuff it for good in RL. Other popular alternatives is the action focused crawl, the meta-world games of adom and angband, and the full blown strategy/survival games dwarffortress (Linux and FreeBSD only) and cataclysm (ps: to play dwarffortress in the console, set [PRINT_MODE:TEXT] in ~/.df/data/init/init.txt).
There are a great many alternative roguelikes out there. Some of the more obscure ones are also the most interesting, my own personal favorite for instance is an abandoned scify spin of nethack called zapm. You can browse around in roguebasin for more options if you are curious.
Way back in the 70's the legendary game developer Infocom produces a host of interactive fiction games. These games are all text, like an interactive novel. Now, I know what you are thinking: "What?!? No graphics? That cannot be fun!" Well, that's a bit like saying that a book without pictures cannot be fun. It's OK to say that if you're 4, not if you're 40. To quote one of the Infocom developers from the Get Lamp (2010) documentary:
«It's all good with high resolution graphics and surround sound, but it is still far away from reality. So what if you make a virtual reality game with 3D glasses? That's a lot better, but you don't feel anything. So how about emerging your body into a sensory simulation capsule. Now we are getting somewhere! But it might not feel like the real thing. So what if we make a direct neural connector to our brain, feeding it our simulation directly? Now we can really live out our fantasy! Of course our imagination could have replaced this high tech solution all along.»
Or, to quote Sheldon Couper from The Big Bang Theory; interactive fiction games runs on the most powerful graphic chip known to man, your imagination! These old games can be played in the command line with frotz (getting the Infocom games legally might be a problem though). Interactive fiction games are still being created today, surprising I know, and most of the newer titles can be obtained for free. There are even yearly IF competitions!
As fun as interactive fiction is, it's an introspective and solitary kind of joy, much like enjoying a good novel. Multi User Dungeons, or MUD's, on the other hand, are multiplayer interactive fiction that has been around since the 80's. Many of these MUD's are still around, in fact some are huge virtual realms that will take years to explore, with decades of political history - in real time! Beware: MUD's are insanely addictive!
One suggestion is Discworld MUD, this MUD recreates Terry Pratchets world in startling detail (check out the Ankh-Morpork city map - each dot here is a playable location! - and this is just one of the cities in the game). You can play the game from the command line with: telnet discworld.starturtle.net. If you want to have even more enjoyment out of this game, you can buy any of the 41 Discworld novels on Amazon. Other MUDS are available on mudconnect.com
There isn't much edutainment software for the console, but there are some. The bsdgames collection contains a few, such as arithmetic and hangman, which are fairly self explanatory. Another example is quiz, which asks you questions and evaluates your responses.
By default quiz contains questions for various topics, such as European capital or Presidential terms. You can either run quiz European capital to get the European nations and guess their capitals, or quiz capital European to get the capitals and guess their European nations. All quizzes can be reversed in this way. The default questions however are usually very dated, even using geography names from the Cold-War era, but you can easily supply your own quiz files, and this radically increases the programs usefulness as a learning tool!
Say you wanted to write a quiz to memorize the console commands used in this howto, you could add this line to /usr/share/games/quiz/index (it might be located in a different place on your system - check with locate quiz):
/usr/share/games/quiz/console:console_task:command
Now in the /usr/share/games/quiz/console file add these entries:
Watch a video:vlc|mplayer
Play some music:play|mocp
Browse the web:lynx|w3m|links
Read your e-mail:mutt|mail
Read a PDF file:fbi?pdf
View some images:fbi
...
You can now brush up on your console skills with quiz console_task command. Notice that the quiz database allows regex, so the answer to "Browse the web" is either lynx, w3m or links, and the answer to "Read a PDF file" is either fbipdf or fbpdf. You may of course write other more useful quizzes, such as vocabulary trainers when learning another language, or quizzes that help you prepare for an exam.
Beyond bsdgames, edutainment software for the console is rare, but one fine exception is typespeed, which is kind of a touch typing game. And i suppose the most useful kind of edutainment is simply reading, a great source of free online books can be found at gutenberg.org. You can read these online with lynx or download them and read them in less. Of course if you really want to challenge your brain, learn to program, and write some edutainment software yourself :^)
You can have much geeky fun in the console beyond pure gaming. fortune is a classic command from the bsdgames collection that produces random quotations and humorous snippets. It's quite popular to add this program to /etc/motd or your shell configuration file, so that you are always greeted with a random welcome message when you log in. You can also pipe this output (or other text) to various fun filters such as:
If you want to produce a random quote with a random cow (cowsay has many different "cows" to choose from), you can do so with this script:
#!/bin/sh
# cowfortune - fortune with random cow
# usage: cowfortune
COWDIR=/usr/share/cowsay/cows
COWFILE=$(ls -1 $COWDIR | shuf | head -n 1)
fortune | cowsay -f $COWFILE
PS: Beware that some systems include offensive ASCII art with sexual content, whereas others have censored out these cows from the collection. So printing a random "cow" may not be what you want. Of course, you can always manually delete any of the cow files that you find offensive.
You could also do something like this: fortune | pirate | cowsay -f $(ls -l /usr/share/cowsay/cows | shuf | head -n 1) | lolcat Have fun :^)
If you're into ASCII art you will probably enjoy the "screensavers" cmatrix and asciiquarium. The bsdgames collection also has a few that you can try out, such as worms -t -d 50 or rain -d 150.
Another fun command is sl, which animates an ASCII art steam locomotive traveling across your screen. The joke is that ls is probably the command you type the most, it's easy to mistype it as sl, and when you do the steam locomotive will grieve you for several seconds.
FreeBSD has a nice program in its repo called coffeebreak. It's just a shell script that runs a bunch of important looking sysadmin commands that does nothing. The idea is to run coffeebreak at work whenever you need a break, it looks like you are just waiting for something important to finish. Other operating systems do not have this program, but seasoned sysadmins can easily create such a script themselves. Here is one simple way to do it: Install and run ttyrec. Now compile a bunch of stuff, and type exit when you are done. You can then, mv ttyrecord ~/.coffeebreak, and add this to ~/.profile (or ~/.bashrc): alias coffeebreak='while true; do ttyplay ~/.coffeebreak'; done You now have your very own coffeebreak script! To quit the playback, hit Ctrl + c.
There are also various Easter eggs in Linux land. A popular example is to add Defaults Insults in /etc/sudoers, which will make sudo rather obnoxious if you mistype your password. Naturally you can make your own internal dad jokes too, I often have the following in my ~/.bashrc:
# quirky humor
alias playdead=halt
alias lazarus=reboot
alias woman=man
alias dog=cat
...
Judging from various terminal screenshots on the web, many people seem to like neofetch and tty_clock (OpenBSD has its own version of this called grdc). It is easy enough to write scripts with similar functionality yourself though. You could for instance just create an ASCII art logo, possibly with the aid of the aforementioned figlet. You can then colorize it with lolcat. As for the digital clock you can add the command cliclock, which does much the same as tty_clock, and topclock, which will add a digital clock to the upper right corner for your terminal (you can still use your terminal normally), by adding the following aliases to your shell configuration (eg. ~/.profile or ~/.bashrc):
alias cliclock='while sleep 1; do clear; date "+%H:%M:%S" | figlet -f smslant; done'
alias topclock='while sleep 1; do tput sc; tput cup 0 $(($(tput cols)-8));
date "+%H:%M:%S"; tput rc; done &'
You can do a whole lot of fun with telnet too, even in 2018! Here are some suggestions:
The arch typical program for reading documents on the command line is less, and for many UNIX users it is a well known tool for text reading. What may come as a surprise though is that less can read a lot more than plain text files. This is due to the fact that less allow a user to specify a preprocessor with the LESSOPEN environment variable. Thus, any file that can be converted to text by some other program, can be viewed automatically in less with the right configuration in place. On most UNIX systems this preprocessor is set to /usr/local/bin/lesspipe.sh or a similar script, which allows less to do a few fancy things like read the contents of a tar archive. You can manually expand the lesspipe.sh script, or even wholesale replace it, to suit your viewing needs.
A nice github project that exploits this is here. When you run the provided ./configure script, it checks to see if you have tools such as: antiword to read Microsoft Word documents, xlhtml to read MS Excel documents, pdftotext (from poppler-utils) to read PDF documents, and more. When it has figured out what tools you have available, it builds a corresponding lesspipe.sh script and a few helper scripts. You can then install them simply by running sudo cp lesspipe.sh sxw2txt code2color tarcolor /usr/local/bin, or other suitable location (make sure your LESSOPEN variable points to the correct script. eg. add export LESSOPEN="|/usr/local/bin/lesspipe.sh %s" to your ~/.profile or ~/.bashrc).
With this in place you should be able to read quite a few documents with less. But be aware that the quality of document to text rendering varies. There is excellent support for old Microsoft office documents for instance, but not for new ones. The rendering of PDF's will vary depending on the file in question, but it's usually not very good. If you have a framebuffer device available, your best bet is probably to convert your office documents to PDF and read them directly with a PDF reader such as fbipdf.
The above script has one flaw, it is rather dated. Worse, the tools needed to convert Microsoft Office documents are even less maintained. Luckily LibreOffice has excellent support for converting different document formats, and you can use it without the pesky GUI. Here are some examples:
$ soffice --headless --convert-to pdf mydoc.docx
$ soffice --headless --convert-to 'html:XHTML Writer File:UTF8' mydoc.docx
$ soffice --headless --convert-to 'txt:Text (encoded):UTF8' mydoc.docx
$ soffice --headless --convert-to epub mydoc.docx
$ soffice --headless --convert-to odt mydoc.docx
Epubs are actually just zipped HTML files, so in theory you can unzip the file and read the results with lynx or another browser. In practice though it can be quite a mess to figure out which files correspond to the chapters you want. Here is a crude script that does the job fairly well in most cases: (and some conversion scripts that build on it)
#!/bin/sh
# epub2html - convert epub to html
# usage: epub2html file.epub
# bugs: only one epub at a time
# set some defaults
CWD=$(pwd)
usage(){
echo Usage: epub2html file.epub 1>&2
exit 1
}
if [ ! $# = 1 ]; then
usage
fi
file=$1
if [ ! -f $file ] || (echo $file | grep -qiv '.*.epub$'); then
usage
fi
if (echo $file | grep -vq '^/'); then
# convert relative path to absolute path
cd $(dirname $file)
file=$(pwd)/$(basename $file)
cd $CWD
fi
name=$(basename ${file:%.[Ee][Pp][Uu][Bb]})
dir=${name}_files
# determine directory name of toc file
ops(){
ops=$(ls $1 | grep -i '^o.*ps')
if [ -z "$ops" ]; then
echo $1
else
toc=$(ls $1/$ops | grep -i 'toc.ncx')
if [ -z "$toc" ]; then
echo $1
else
echo $1/$ops
fi
fi
}
# extract epub and chapter information
mkdir -p $dir && cd $dir
unzip -qo $file
ops=$(ops $CWD/$dir) && cd $ops
cat [Tt][Oo][Cc].[Nn][Cc][Xx] | sed -n '/<navPoint/,/<\/navPoint/p' |\
sed -n 's/.*<text>\(.*\)<\/text>.*/\1/p' > chaps
cat [Tt][Oo][Cc].[Nn][Cc][Xx] | sed -n '/<navPoint/,/<\/navPoint/p' |\
sed -n 's/.*src="\(.*\)".*/\1/p' | sed 's/%20/ /g' > links
# generate html and text files
cat <<'EOF' > $CWD/$name.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Contents</title>
</head>
<body>
<h1>Contents:</h1>
EOF
for i in $(seq $(cat links | wc -l)); do
link=$(sed -n ${i}p links)
chap=$(sed -n ${i}p chaps)
echo " <a href=\"$ops/$link\">$chap</a><br>" \
>> $CWD/$name.html
done
cat <<'EOF' >> $CWD/$name.html
</body>
</html>
EOF
#!/bin/sh
# epub2pdf - convert epub to pdf
# usage: epub2pdf file.epub
# depend: epub2html, libreoffice, ghostscript
# bugs: html to pdf conversion is fairly slow
# set some defaults
if [ ! $# = 1 ]; then
exit 1
fi
name=$(basename ${1%.[Ee][Pp][Uu][Bb]})
if [ ! -f $name.html ]; then
# only keep html intermediary if it already exists
trap 'rm -rf $CWD/$name.html $CWD/${name}_files' 0 1 2 15
epub2html $* || exit 1
fi
CWD=$(pwd)
temp=/tmp/epub2pdf-$$
mkdir $temp && cd $temp
IFS='
'
# convert extracted epub to pdf
for file in $(awk -F\" '/<a/ { print $2 }' $CWD/$name.html); do
soffice --headless --convert-to pdf $file 2>/dev/null 1>&2
done
gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite \
-sOutputFile=$CWD/$name.pdf *.pdf
rm -rf $temp
#!/bin/sh
# epub2txt - convert epub to text
# usage: epub2txt file.epub
# depend: epub2html, w3m
# set some defaults
if [ ! $# = 1 ]; then
exit 1
fi
name=$(basename ${1%.[Ee][Pp][Uu][Bb]})
if [ ! -f $name.html ]; then
# only keep html intermediary if it already exists
trap 'rm -rf $name.html ${name}_files' 0 1 2 15
epub2html $* || exit 1
fi
IFS='
'
# convert extracted epub to text
> $name.txt
for file in $(awk -F\" '/<a/ { print $2 }' $name.html); do
w3m -dump -T text/html < $file | fmt >> $name.txt
done
#!/bin/sh
# eread - read an epub directly in w3m
# usage: eread file.epub
# depend: epub2html, w3m
CWD=$(pwd)
file=$1
if (echo $file | grep -vq '^/'); then
# convert relative path to absolute path
cd $(dirname $file)
file=$(pwd)/$(basename $file)
cd $CWD
fi
name=$(basename ${file:%.[Ee][Pp][Uu][Bb]})
# extract epub to /tmp then read it
cd /tmp
trap 'rm -rf /tmp/$name*' 0 1 2 15
epub2html $file || exit 1
Other ebook formats may not be that easy to read manually. If you have installed calibre, it comes with the ebook-convert command line tool, which you can use to convert the ebook file into other formats. Some examples:
$ ebook-convert mydoc.mobi mydoc.pdf ; fbipdf mydoc.pdf
$ ebook-convert mydoc.mobi mydoc.html ; lynx mydoc.html
$ ebook-convert mydoc.mobi mydoc.txt ; less mydoc.txt
CBR comic books are just images archived with rar, you can read them in the console by unrar'ing the file and then view the images with fbi if you have a framebuffer device available (CBZ files are zipped images, so use unzip on them).
As already mentioned in the Editor Wars section, the classic text editors in UNIX are nano, vi and Emacs, or one of their many variants. There are many other excellent editors besides, far too many to mention. If you just need a quick place to start however, nano has your back. But for serious coding, I recommend learning either vi or Emacs. Both of these editors will require some effort to learn, but they are extremely useful when mastered.
You can spell check your files interactively with aspell -c myfile, or aspell -l lang -c myfile, for languages other then English.
Not all Unixes come with aspell (or ispell/hunspell/etc...) by default, but virtually all come with the classic spell checker spell. spell is a very simple program. It checks if your words are found in the systems plain text dictionary (usually in /usr/share/dict/words, you might need to install such a dictionary if it isn't already included, eg. apt install wamerican), if not it just prints the presumably misspelled word. By the way you can manually check for words in this dictionary with the look command. Using spell may at first seem to be very inefficient, since it just hands you a long list of words that you need to manually check and correct yourself. But for precisely that reason it is actually very efficient at teaching you correct spelling.
By the way both vim (vi improved) and Emacs have internal spell checkers you can use, :set spell for vim, and Alt + x ispell-buffer for Emacs. But it's just as easy to use aspell from within vi/vim by running :!aspell -c myfile (the Emacs ispell command actually uses aspell as its backend).
WordNet is a very good English dictionary, you can lookup a word from the command line like so: wn flabbergasted -over (over means "overview"). Sadly there aren't many such alternatives for languages other then English, but you may be able to find a free online dictionary, and if you are very lucky it might even work in lynx.
Way back in the 80's UNIX came with the Writer's Workbench suit of tools. Today much of this functionality has been continued with the GNU tools diction and style (both will be included if you install the diction package). diction does basic sanity checks of your documents, such as checking if two identical words are written right next to each other. And style analysis your prose. Of course the tools cannot actually make you a good writer, but they are handy utilities nonetheless (they only support English and German). And like all the classic UNIX tools, it pays great dividends to split your work into many small files (eg. cat chap* > book), otherwise the output from these commands will be overwhelming.
Although the GNU project has done some work in translating their manpages to Non-English languages, that is the exception to the rule. Documentation in UNIX is almost always in English, and any noteworthy book on the subject certainly is. I don't mean anything condescending by that, I am not a native English speaker myself, it's just the way things are.
Of course, you can work with Non-English languages. The Linux Framebuffer section demonstrates how you can use fbterm to display any exotic characters on the console that you can on the desktop. But this is not always an option. (you'd be hard pressed to run fbterm in a serial console) And worse, although you might be able to display Non-English text, it is another matter to type it. It is possible to change the keyboard layout as we discussed in the Configuring the Console section, but it is almost always desirable to use regular English-friendly ASCII input by default, even if you occasionally need to type in something exotic. Here is a simple tool that I find useful when I need to type in something Norwegian in the console:
#!/bin/sh
# no2utf - convert ASCII Scandinavian representations to UTF-8
# usage: no2utf < in > out
sed -e 's/\\aa/å/g' \
-e 's/\\AA/Å/g' \
-e 's/\\Aa/Å/g' \
-e 's/\\oa/å/g' \
-e 's/\\OA/Å/g' \
-e 's/\\oA/Å/g' \
-e 's/\\ae/æ/g' \
-e 's/\\AE/Æ/g' \
-e 's/\\Ae/Æ/g' \
-e 's/\\"a/ä/g' \
-e 's/\\"A/Ä/g' \
-e 's/\\\/o/ø/g' \
-e 's/\\\/O/Ø/g' \
-e 's/\\"o/ö/g' \
-e 's/\\"O/Ö/g' \
With such a tool I can write "\/Onskeliste" in a file and convert it to "Ønskeliste" with no2utf < file > temp && mv temp file, or do so interactively with cat | no2utf. You should be able to echo it too if you quote the string, eg. echo '\/Onskeliste' | no2utf. (if not, try echo -E or /bin/echo) We can easily make similar tools for Spanish (eg. s/\\~n/ñ/g), German (eg. s/\\ss/ß/g), etc. We can even write more elaborate converters for things like Greek, Cyrillic, Arabic or Hebrew. (the last two need to piped through frididi to get right-to-left order) Here is a ASCII to Greek converter for instance:
#!/bin/sh
# gre2utf - convert ASCII Greek representations to UTF-8
# usage: gre2utf < in > out
# bugs: uses arbitrary non-standard conventions
# annoyingly, a final sigma must be typed \s, not s.
#
# guide:
# Most of the Greek letters are mapped to their vocalization
# equivalents in the Latin alphabet, eg: A, B, G, D, E, Z, I, K, L, M,
# N, X, O, P, R, S, T, U, F and Q map more or less perfectly to Α, Β, Γ,
# Δ, Ε, Ζ, Ι, Κ, Λ, Μ, Ν, Ξ, Ο, Π, Ρ, Σ, Τ, Υ, Φ and Χ. Finally, a few
# letters are represented by Latin characters that look like the Greek
# alternatives, such as H, C, Y and W, corresponding to Η, Θ, Ψ and Ω.
# (C is rather forced, you can also write it with \O)
#
# Some special characters must be written with an initial escape, "\",
# character, these include Iota/Upsilon with diaeresis, a final sigma
# and Theta/Phi/Kappa scripts. So \I, \i, \U, \u, \s, \c, \f, \k map to
# Ϊ, ϊ, Ϋ, ϋ, ς, ϑ, ϕ and ϰ. You can also write the name of the letter
# after an escape, eg: \DELTA and \omega for Δ and ω.
sed -e 's/\\ALPHA/Α/g' \
-e 's/\\alpha/α/g' \
-e 's/\\BETA/Β/g' \
-e 's/\\beta/β/g' \
-e 's/\\GAMMA/Γ/g' \
-e 's/\\gamma/γ/g' \
-e 's/\\DELTA/Δ/g' \
-e 's/\\delta/δ/g' \
-e 's/\\EPSILON/Ε/g' \
-e 's/\\epsilon/ε/g' \
-e 's/\\ZETA/Ζ/g' \
-e 's/\\zeta/ζ/g' \
-e 's/\\ETA/Η/g' \
-e 's/\\eta/η/g' \
-e 's/\\THETA/Θ/g' \
-e 's/\\theta/θ/g' \
-e 's/\\IOTA/Ι/g' \
-e 's/\\iota/ι/g' \
-e 's/\\KAPPA/Κ/g' \
-e 's/\\kappa/κ/g' \
-e 's/\\LAMBDA/Λ/g' \
-e 's/\\lambda/λ/g' \
-e 's/\\MU/Μ/g' \
-e 's/\\mu/μ/g' \
-e 's/\\NU/Ν/g' \
-e 's/\\nu/ν/g' \
-e 's/\\XI/Ξ/g' \
-e 's/\\xi/ξ/g' \
-e 's/\\OMICRON/Ο/g' \
-e 's/\\omicron/ο/g' \
-e 's/\\PI/Π/g' \
-e 's/\\pi/π/g' \
-e 's/\\RHO/Ρ/g' \
-e 's/\\rho/ρ/g' \
-e 's/\\SIGMA/Σ/g' \
-e 's/\\sigma/σ/g' \
-e 's/\\TAU/Τ/g' \
-e 's/\\tau/τ/g' \
-e 's/\\UPSILON/Υ/g' \
-e 's/\\upsilon/υ/g' \
-e 's/\\PHI/Φ/g' \
-e 's/\\phi/φ/g' \
-e 's/\\CHI/Χ/g' \
-e 's/\\chi/χ/g' \
-e 's/\\PSI/Ψ/g' \
-e 's/\\psi/ψ/g' \
-e 's/\\OMEGA/Ω/g' \
-e 's/\\omega/ω/g' \
-e 's/\\O/Θ/g' \
-e 's/\\o/θ/g' \
-e 's/\\I/Ϊ/g' \
-e 's/\\i/ϊ/g' \
-e 's/\\U/Ϋ/g' \
-e 's/\\u/ϋ/g' \
-e 's/\\s/ς/g' \
-e 's/\\c/ϑ/g' \
-e 's/\\f/ϕ/g' \
-e 's/\\k/ϰ/g' \
-e 's/A/Α/g' \
-e 's/a/α/g' \
-e 's/B/Β/g' \
-e 's/b/β/g' \
-e 's/G/Γ/g' \
-e 's/g/γ/g' \
-e 's/D/Δ/g' \
-e 's/d/δ/g' \
-e 's/E/Ε/g' \
-e 's/e/ε/g' \
-e 's/Z/Ζ/g' \
-e 's/z/ζ/g' \
-e 's/H/Η/g' \
-e 's/h/η/g' \
-e 's/C/Θ/g' \
-e 's/c/θ/g' \
-e 's/I/Ι/g' \
-e 's/i/ι/g' \
-e 's/K/Κ/g' \
-e 's/k/κ/g' \
-e 's/L/Λ/g' \
-e 's/l/λ/g' \
-e 's/M/Μ/g' \
-e 's/m/μ/g' \
-e 's/N/Ν/g' \
-e 's/n/ν/g' \
-e 's/X/Ξ/g' \
-e 's/x/ξ/g' \
-e 's/O/Ο/g' \
-e 's/o/ο/g' \
-e 's/P/Π/g' \
-e 's/p/π/g' \
-e 's/R/Ρ/g' \
-e 's/r/ρ/g' \
-e 's/S/Σ/g' \
-e 's/s/σ/g' \
-e 's/T/Τ/g' \
-e 's/t/τ/g' \
-e 's/U/Υ/g' \
-e 's/u/υ/g' \
-e 's/F/Φ/g' \
-e 's/f/φ/g' \
-e 's/Q/Χ/g' \
-e 's/q/χ/g' \
-e 's/Y/Ψ/g' \
-e 's/y/ψ/g' \
-e 's/W/Ω/g' \
-e 's/w/ω/g' \
In addition to typing text on the console, you may want to write documents. The following section talks about that topic, but we can make an extra addendum: Neatroff is a superb tool for making documents in exotic languages, including right-to-left alphabets. This modern troff version supports custom colors, fonts and links. You probably won't find it in the repository, but it's an easy compile.
The closest console equivalent to a typical "word processor" is wordgrinder. It is basically Word Pad for the terminal, so it lets you add a few basic extras to your text documents, such as underlined text, chapter headers or bullet points. Just hit Esc to bring up the main menu. It uses it's own file format by default, but you can export this to other formats, such as ODT, LaTeX or HTML. The problem with word processors though, as with food processors, is that it takes delicious looking input, and returns a messy goo as its output. It may technically be edible, but it ain't pretty! A cleaner, more tasteful, way to write documentation is to use a markup language, and then convert that to a document format, such as PDF or HTML.
You can write sloppy HTML quickly with Markdown, Pearlpod, and an assortment of other "shortcut" languages, but, generally, I don't recommend it. Firstly, these tools tend to produce exactly the kind of messy goo that we are trying to avoid. Secondly, writing Troff is just as easy. TeX or HTML requires more effort, but once you go beyond the very basics, quick and dirty tools tend to have subtle edge cases, requiring just as much knowledge and effort to work around, as it would to write the low level markup language in the first place. My advice is this: If you want to write in a markup language, learn a real markup language, otherwise, use a word processor.*
PS: If you have to write Markdown,
there are many alternative interpreters,
python3-markdown is one of them
(the executable may be called markdown_py,
or something else,
it can vary from system to system).
Another famous alternative,
on the console at least,
is pandoc.
This last Markdown interpreter extends the language quite a bit,
and tries to be compatible with all document formats known to man.
The idea is to quickly write a Markdown document,
that you can then convert to PDF,
HTML, LaTeX, DocBook, DOCX, ODT or what have you.
The resulting output may not be pretty for a obsessive-compulsive purist like me,
but it's a quick and dirty way to get the job done.
Troff is the markup language used to write man pages in UNIX, and as such it isn't very impressive. The man pages are just plain text with little formatting finesse. This however is not because troff is primitive, but rather, because the terminal is. By default troff writes output as postscript for printing. If you read the postscript edition of a manpage, you will see that it actually looks like a professional document:
$ cp /usr/man/man1/ls.1.gz /tmp && cd /tmp
$ gunzip ls.1.gz
$ groff -man -T pdf ls.1 > ls.pdf
$ fbipdf ls.pdf
Of course it still looks like a man page, as it should since it's written with the man(7) macro package (pure troff markup is very low level, you usually write documents with higher level macro commands). But troff is capable of producing other document types as well. Several computer text books have been written in troff. Naturally this includes the UNIX classics such as The C Programming Language and The UNIX Programming Environment, but also some modern works such as The Design and Implementation of the FreeBSD Operating System and The Go Programming Language. These books contain indexes, footnotes, graphs, diagrams, pictures, tables and most confections you would expect to find in professional literature (troff's complete lack of comic sans and clip-arts reinforces the professional vibe).
A popular troff macro package for general purpose documents is ms. Here is an example letter written in ms:
.DS L
To: Archduke Poggle of Geonosis
23 Insectoid Str.
Hive
103133
GEONOSIS
From: Emperor Palpatine
Imperial Palace
PO 000001
Senate District
CORUSCANT
.DE
.SH
Dear Archduke
.PP
The so called
.I
undefeatable
.R
Death Star was blown to bits by a bunch of teenagers yesterday.
I must say I am disappointed!
We need to construct a new planet killer ASAP,
and this time lets try to avoid an
.B
Achilles heel
.R
in our design shall we?
.PP
I have some other ideas for further improvements.
First of all we need a
.I
menacing
.R
throne room with a view...
The syntax here is very simple. You first write an ms command, which all begin with a dot, such as .SH (a section header). This command is then applied to the following text, until you encounter another ms command. You will find troff included by default on virtually every UNIX system, and it is a simple document system that is indispensable for system administrators, and surprisingly useful for casual use. Another advantage of troff is that its syntax fits very well with the line based UNIX tools, for instance to convert a troff file to plain text you only need to do this: cat mydoc.ms | sed '/\..*/d' | fmt
You can convert the above ms source file to various document formats, such as HTML or PDF: groff -ms -T pdf mydoc.ms > mydoc.pdf. It will look like this:
Some important ms macros:
PS: You can write inline troff commands if you need to; eg. "normal text \fIitalic text \fBbold text \f(CWconstant width text \fRroman, ei. normal, text." But generally speaking, in ms documents at least, you write troff commands on a line by itself, and content verbatim after it.
Ironically the markup language used to write UNIX manuals, troff, is not well documented. This has mainly to do with the fact that it did not catch on as a general purpose documentation tool outside of Bell Labs. Still, the old Bell Labs papers on these tools are very useful references, even for modern versions of these programs. PS: Linux systems today use groff and BSD systems use mandoc, but both are essentially backwards compatible with the original UNIX troff. Mandoc however can only render manpages, so install groff as well on BSD systems if you plan to write generic troff documents. If you need to write an actual man page, check man 7 man. There are also a number of troff preprocessors that can add specific functionality, most of them have fairly decent man pages. In the list below you will also find links to the old Bell Labs papers on these tools:
Pro tip: You can actually write troff in Markdown, using the aforementioned pandoc. Run pandoc myfile.md -t ms -o myfile.pdf, and it will use GNU troff (the -t ms part), instead of LaTeX, to generate the output. This will even render advanced stuff like code blocks with syntax highlighting, images (if they are in EPS format, eg. convert pic.png pic.eps), and embedded LaTeX code (pandoc will translate it to troff code). Check out Gavin Freeborn's YouTube channel for more tips!
Perhaps the most widely used document system on UNIX, besides XML, is TeX (LaTeX is a dialect of TeX). It was originally written because it's author was annoyed with the limitations of Troff. TeX has become quite popular in the scientific community, and it is not unusual for universities to require students to send in their papers in this format. Of course you can also use TeX to write your personal letters and todo lists. Although TeX is not normally included by default in UNIX systems, you can usually install a TeX distribution, like the texlive-base or a similar package, with the systems package manager (PS: Unlike troff, TeX is a large package, it can take more than a Gigabyte of diskspace!). The above example letter in TeX would be written like this:
\documentclass{article}
\begin{document}
\begin{verbatim}
To: Archduke Poggle of Geonosis
23 Insectoid Str.
Hive
103133
GEONOSIS
\end{verbatim}
\begin{verbatim}
From: Emperor Palpatine
Imperial Palace
PO 000001
Senate District
CORUSCANT
\end{verbatim}
\section*{Dear Archduke}
The so called \emph{undefeatable} Death Star was blown to bits by a bunch
of teenagers yesterday.
I must say I am disappointed!
We need to construct a new planet killer ASAP,
and this time lets try to avoid an \textbf{Achilles heel} in our design
shall we?
I have some other ideas for further improvements.
First of all we need a \emph{menacing} throne room with a view..
\end{document}
As you can see the TeX syntax is more free form, where you can intersperse normal text with formatting commands. You can convert the TeX source file to more useful document types, such as a PDF:
$ latex mydoc.tex
$ dvipdf mydoc.dvi
(PS: Use pdflatex if you need pictures, also, TeX is very particular about its syntax, it will fail to compile the document for instance if you leave out the \end{document} line at the end, or make some such easy mistake). It will look like this:
The TeX equivalents to the above mentioned ms macros are:
You can probably find a useful TeX tutorial with a quick Google search (although "latex" might give you some unwarranted results), and there are several useful books on the subject available on Amazon.
The serious competitor to TeX (in terms of popularity at least), is DocBook. It is basically HTML for books. As such, it is only slightly more horrible then plain HTML, if such a thing were possible.* ** *** Here is an example of how you could write the above letter:
<!DOCTYPE article PUBLIC "-//OASIS/DTD DocBook XML V4.5//EN"
"/usr/share/xml/docbook/xml-dtd-4.5/docbookx.dtd">
<article>
<section>
<para>
<literallayout>
To: Archduke Poggle of Geonosis
23 Insectoid Str.
Hive
103133
GEONOSIS
</literallayout>
</para>
</section>
<section>
<para>
<literallayout>
From: Emperor Palpatine
Imperial Palace
P0 000001
Senate District
CORUSCANT
</literallayout>
</para>
</section>
<section>
<title>Dear Archduke</title>
<para>
The so called <emphasis>undefeatable</emphasis> Death Star was
blown to bits by a bunch of teenagers yesterday. I must say I am
disappointed! We need to construct a new planet killer ASAP, and
this time lets try to avoid an <emphasis role="bold">Achilles
heel</emphasis> in our design shall we?
</para>
<para>
I have some other ideas for further improvements. First of all we
need a <emphasis>menacing</emphasis> throne room with a view...
</para>
</section>
</article>
We can now convert this source document into HTML or other formats, for example assuming you saved this letter as letter.xml you can do this:
$ STYLEST=/usr/share/xml/docbook/xsl-stylesheets-1.78.1/html/docbook.xsl
$ xsltproc -o letter.html $STYLEST letter.xml
The result will look similar (browser dependent) to this:
There is a bewildering plethora of different DocBook tools available, and the method used to produce DocBook documents can vary greatly. This is just an example. Also note that the docbookx.dtd and docbook.xsl files used here can be located in different places on different systems, check with locate docbookx.dtd docbook.xsl (You might need to install a package or two to get them).
The DocBook equivalents to the above mentioned TeX commands (as with HTML, DocBook tags such as <para> must end with a closing tag, in this case </para>):
Writing left-aligned paragraphs, numbered paragraphs, footnotes or linebreaks, can be done with various clever hacks, that is anything but intuitive. Buy a book an Amazon or search the web for tutorials to get the gory details, or better yet, choose a different markup language to write your documents in.
PS: We all know that XML is horrid and all that, but the "good" thing about it is that it's used everywhere! The MS/LibreOffice formats DOCX/ODT uses XML for instance, and the above xsltproc command could have been -o letter.docx or -o letter.odt. You can also unzip an ODT file and inspect/edit the XML with xsltproc/xmllint.
Of course, you don't need to write documents in pure DocBook (eg. XML), no one in their right minds would. Much professional documentation however, including many O'Reilly books and the FreeBSD Handbook, is written in AsciiDoc. To compile the example below to HTML for instance, just run asciidoc letter.txt*
To: Archduke Poggle of Geonosis
23 Insectoid Str.
Hive
103133
GEONOSIS
From: Emperor Palpatine
Imperial Palace
P0 000001
Senate District
CORUSCANT
== Dear Archduke
The so called _undefeatable_ Death Star was blown to bits by a bunch of
teenagers yesterday. I must say I am disappointed! We need to construct a
new planet killer ASAP, and this time lets try to avoid an
*Achilles heel* in our design shall we?
I have some other ideas for further improvements. First of all we need a
_menacing_ throne room with a view...
As you can see, AsciiDoc is very similar to Markdown. The latter is usually prefered by web developers who don't care much about publishing, the former by writers who don't care much about web development. Your mileage may vary.
sc is quite a decent spreadsheet for the console. It has its own way of doing things though, so initially you will want to hit ? and spend some time learning the ropes. sc uses its own file format by default, but it can export to LaTeX or tbl (ei. troff) too. Of course sc is a poor substitute for Excel, but then again, Excel is a poor substitute for a database. And there are many professional databases available for UNIX, such as MariaDB and PostgreSQL. For private use though SQLite is probably sufficient:
Let's apply a popular spreadsheet pass time to our SQL database, making a budget and doing our accounting:
$ sqlite3 account.db
sqlite> CREATE TABLE account ( id INTEGER PRIMARY KEY, date TEXT DEFAULT CURRENT_DATE,
...> category TEXT DEFAULT 'food', price INTEGER NOT NULL, desc TEXT );
sqlite> INSERT INTO account ( category, price, desc )
...> VALUES ( 'sweets', 1.25, 'coca cola' );
sqlite> INSERT INTO account ( category, price, desc )
...> VALUES ( 'sweets', 2.95, 'candy bar' );
sqlite> INSERT INTO account ( date, price ) VALUES ( '2018-04-25', 20.65 );
sqlite> .headers on
sqlite> .mode column
sqlite> SELECT * FROM account ORDER BY date;
It will produce this output:
| id | date | category | price | desc |
| ---------------- | ---------------- | ---------------- | ---------------- | ---------------- |
| 3 | 2018-04-25 | food | 0.65 | chewing gum |
| 1 | 2018-04-26 | sweets | 1.25 | coca cola |
| 2 | 2018-04-26 | sweets | 2.95 | candy bar |
With this database in place you can list particular types of expenses: SELECT * FROM account WHERE category = 'sweets';, or summarize them: SELECT sum(price) AS total FROM account WHERE category = 'sweets';. Type .help for more instructions, and .quit to exit SQLite. Using SQL databases from a shell script is also quite trivial. Here is a simple example:
#!/bin/sh
# sqlaccount - manage an sql account
# usage: sqlaccount [-d date] [-c category] [-s category] [amount comments...]
database=$HOME/account.db
if [ ! -f $database ]; then
echo "CREATE TABLE account ( id INTEGER PRIMARY KEY, date \
TEXT DEFAULT CURRENT_DATE, category TEXT DEFAULT 'food', price \
INTEGER NOT NULL, desc TEXT );" | sqlite3 $database
fi
if [ $# = 0 ]; then
echo "SELECT * FROM account ORDER BY date;" | sqlite3 -column $database
exit
fi
date=$(date "+%Y-%m-%d")
category=food
for args in "$@"; do
case "$args" in
-d) date=$2 ; shift 2 ;;
-c) category=$2 ; shift 2 ;;
-s)
if [ $# = 1 ]; then
echo "SELECT sum(price) AS total FROM account;" |
sqlite3 -column $database
else
echo "SELECT sum(price) AS total FROM account WHERE \
category = '$2';" | sqlite3 -column $database
fi
exit ;;
esac
done
price=$1
shift
echo "INSERT INTO account ( date, category, price, desc ) VALUES \
( '$date', '$category', $price, '$@');" | sqlite3 $database
You can now do something like this:
$ sqlaccount -c sweets 1.25 coca cola
$ sqlaccount -c sweets 2.95 candy bar
$ sqlaccount -d 2018-04-25 20.65
$ sqlaccount
3 2018-04-25 food 20.65
1 2018-04-26 sweets 1.25 coca cola
2 2018-04-26 sweets 2.95 candy bar
$ sqlaccount -s
24.85
$ sqlaccount -s sweets
4.2
There is a whole heck of a lot you can do with an SQL database, we have only scratched at the very surface here. There are many fine books on this subject on Amazon, and I am sure you can find a useful tutorial if you Google for it. It does require some effort to learn SQL, but once you have done so, you will never need a spreadsheet again!
Sidenote: There are a number of alternative super lightweight databases out there for personal use, one uncommon but good choice is Plan9Ports ndb.
You may feel that the above example was far too complicated, and you are absolutely right. In fact if you ever see anyone doing their personal accounting in an SQL database, slap them! (one exaggeration deserves another) It is in fact quite simple to just type in your account information in a plain text editor, and summarize them or query the accounts with sed or awk. Consider this example:
2018-04-25 food 20.65
2018-04-26 sweets 1.25 coca cola
2018-04-26 sweets 2.95 candy bar
You can list all account records for April 2018 by: sed -n '/^2018-04/p' account.txt, or summarize the March expenses by: awk '/^2018-04/ { sum+=$3 } END { print sum }' account.txt. Or if you only want to summarize your candy expenditures for April: awk '/^2018-04.* sweets/ { sum+=$3 } END { print sum }' account.txt. A more elaborate example of an accounting shell script can be found in the Accounting section.
Do not be quick to dismiss awk, it can do much more then you think. In section 4.3 of the classic book The AWK Programming Language, for example (if you don't have this in your bookshelf - get it!), the authors demonstrate how a ~50 line awk script enables you to create a relational database spread across multiple files, using column name tags instead of numbers. awk is also fast, unless you are planning to develop a website with hundreds of requests per second, industry grade databases will not give you any significant speed improvements. Best of all, when you have mastered awk, you will find a million other uses for this versatile utility!
It is odd that businessmen place so much emphasis on presentations, when they are really just a slideshow of pictures, lightly sprinkled with semi-irrelevant text. There are presentation specific software for the console, such as tpp, you can also create slides with TeX or even troff. Another approach is to create a bunch of images with sequential names (eg. 001.jpg, 002.jpg, etc...), and view them with an image viewer like fbi (you can add text to the pictures with ImageMagick if you like).
If you want to give tpp a go, read the instructions in /usr/share/doc/tpp/README (it might be located somewhere else on your system). It's fairly straight forwards. tpp uses its own file format, but it can also export to LaTeX. The above tpp screenshot was created with this tpp file:
--bgcolor white
--fgcolor magenta
--newpage
--heading A Short Demo of TPP
--withborder
--center Presentations are written as plain text,
--center but you can manipulate the text in various ways.
--center For example, you can change
--boldon
--color red
--center font style and color
--boldoff
--## ulon means "underline on", this is a comment btw
--ulon
--color blue
--center in various ways
--uloff
--color magenta
--center and you can play around with
--huge ----Figlet----
--center Thats about it
--center Use the arrow keys or space to move around in the slide show.
--newpage
A more modern alternative is mdp, which will let you write presentations in slick Markdown. (I'd go for the latter)
For mathematical tasks, bc is the goto tool. You can use it interactively or within your scripts. Just be aware that this calculator doesn't do floating point arithmetic by default, you can either use the -l flag to set the scale to 20 and enable some advanced mathematical functions (sine, cosine, arctangent, logarithms, etc.), or you could just set the scale to the accuracy you need. To demonstrate:
$ bc -q
20.2 / 10
2
scale=2
20.2 / 10
2.02
quit
$ echo pi = $(echo "4*a(1)" | bc -l)
pi = 3.14159265358979323844
Of course there are other mathematical tools available. For instance, dc is a reverse-polish desk calculator, if you happen to be a 100 years old and like that sort of thing.
As for graphs, the goto program is gnuplot. This program can do any imaginable thing relating to graphs, and then some. But just to get you started, let take a very simple example:
Suppose you have used the above mentioned sqlaccount script to check how much expenses you had from January to December in 2017, and you added these numbers to a text file like so:
01 2104
02 2260
03 1987
04 2053
05 2242
06 2321
07 2107
08 2134
09 2032
10 1934
11 2143
12 2355
You can then use gnuplot to make a graph out of this:
$ gnuplot
gnuplot> set term pngcairo # you may need to use pdfcairo
gnuplot> set title 'Expenditures 2017'
gnuplot> set out 'expenditures.png'
gnuplot> unset key
gnuplot> set grid
gnuplot> plot [1:12] 'months.txt' with line
gnuplot> quit
You can put these instructions in a file and call gnuplot with gnuplot -c mycommands to use this program non-interactively. If you want gnuplot to export a plain ASCII graph, use set term dumb instead of cairopng. You can see a full list of output formats by typing set term in the gnuplot prompt. The notation [1:12], tells gnuplot to begin drawing the graph at x coordinates 1 (January), and end at x coordinates 12 (December). Most of the other commands here control what the graph will look like. Feel free to experiment with these settings. There are many ways to incorporate gnuplot graphs into your documents. The simplest is just to export the graph as a picture, as we did in the above example, and include this picture in your document (exactly how will depend on what document type you want to create). You can do a lot with gnuplot, and most distributions also bundle some nice demos that you can play with.
You can make simple drawings with document markup languages, such as TeX or troff (using pic). However, you might not be so interested in the presentation of mindmaps, as the functionality of them. Well, you could just use the filesystem: If you think about it, the hierarchical filesystem in UNIX is a mindmap of sorts. And tools such as ranger and tree can help you visualize it. See the getting organized section for ideas.
Before we delve into this topic, let me tell you a little story: I have a good friend who is a wiz at front-end web development. He had spent the better part of his youth making a customer support database using ReactJS and CouchDB. The system was fantastically complex, so much so that it never actually replaced the pen-and-paper interim solution. One day we sat down and talked about what his goals with the project were. It boiled down to two things:
So I asked him: Why not simply make directories of the geographic areas where you have customers, and a file for each customer. You can put contact information at the top of the file and add dates followed by a support log after that (eg. date >> */customer ; nano */customer). You can then:
My friend was a mite peeved at having his whole life reduced to a set of oneliners, but being a humble fellow, our friendship survived. (the last example here lists the customer files from oldest to newest) The point? Don't look for tools to get organized, get organized. Even the simplest tools go a long way with wise conventions. As a sidenote, do not underestimate the value of pen and paper. You should always have a notebook at the ready for short-term tasks and quick brainstorming, even if you use a computer (including a smartphone) to organize long-term data.
Personal Information Management, or PIM, is a modern buzz word which simply means "getting organized". What this entails in practice depends on your needs and tastes. For the minimalist vi is really all you need, but at the other end of the spectrum you have full-blown-command-line-PIM-suits, such as taskwarrior (aka. task, there is also a nice front-end for this, vit). Of course if all you need is a simple todo list, there are programs for that too, such as tdl and many others (org-mode for Emacs is a kind of todo list on steroids).
The following sections demonstrate how to write your own simple PIM scripts. These are crude, the point here is to show you some tricks and perhaps inspire you to create your own variations to suit your needs. But to reiterate, even basic tools can be powerful if you follow careful conventions and keep your files organized, many of these examples (or variations of them) are scripts that I use on a daily basis.
We'll start off by creating a simple 2do list:
#!/bin/sh
# 2do - simple 2do list manager
# usage: 2do [-c] project [ nr... | task... ]
# set some defaults
DIR=${DIR:-$HOME/.2do}
mkdir -p $DIR
# parse arguments
if [ $# = 0 ]; then
ls $DIR && exit
elif [ $# = 1 ]; then
nl $DIR/$1 && exit
elif [ $1 = -c ]; then
> $DIR/$2 && exit
fi
FILE=$1 ; shift
# nr: remove task; task: add it
if (echo "$@" | egrep -q '^[0-9\ ]+$'); then
cmd=$(echo "$@" | sed 's/\([0-9][0-9]*\)/\1d;/g')
sed -i "$cmd" $DIR/$FILE
else
echo "$@" >> $DIR/$FILE
fi
And here is a short demonstration of its use:
$ for thing in milk eggs 'NEED toilet paper ASAP!'; do
> 2do buy $thing
> done
$ 2do work finish damn project
$ 2do work start other project
$ 2do
shop work
$ 2do buy
1 milk
2 eggs
3 NEED toilet paper ASAP!
$ 2do work
1 finish damn project
2 start other project
$ 2do buy 3 1
$ 2do buy
1 eggs
This 2do script is very basic, it does not support priorities or tags for instance. And though you can remove multiple tasks at once, you can only add one task at a time. There is no mechanism for deleting a 2do list, but you can always do that manually: rm $HOME/.2do/mylist. Naturally, these 2do lists can also be opened in any plain text editor, eg. vi $HOME/.2do/mylist.
It is possible to augment this functionality with simple conventions, to demonstrate:
$ 2do call :$(date +%Y-%m-%d):10:family:grandma, not on her cell phone!
$ 2do call | sort -k 2 | awk -F: '$3>5 && $4=/family/ { print $1, $5 }'
1 grandma, not on her cell phone!
In this example we are adding a task to our call list, with todays date, a priority of 10 and the tag "family". Next, we list the call tasks, from oldest to newest, that have a priority greater then 5 and the tag "family".
If you need such gruesome PIM (mis)management, if might be better to extend the 2do script. You might want to add dates automatically for example. If you comment out tasks instead of deleting them, you might devise a way to print a project ETA based on how many tasks you have completed since the initial start date. You could add default priorities and tags, and then sort the lists by priority first, then by date. You might even design a mechanism for automatically incrementing the priority over time, so that older tasks will get more and more urgent. Feel free to experiment with it. And once you have made all of these enhancements, and more, take a step back and compare your improved version to the original. Was the extra complexity worth it?
Here is a very simple script, that I find useful in various situations. It basically keeps track of a queue, a plain text list, where each line is a task:
#!/bin/sh
# que - a simple queue tracker
# usage: que [-cp][-f file | file ]
# set some defaults
dir=${dir:-$HOME/.queues}
mark=${mark:-<--}
# parse arguments
if [ $# = 0 ]; then
ls $dir && exit
fi
while getopts :cpf: opt; do
case "$opt" in
c) clear=yes ;;
p) prcur=yes ;;
f) file=$OPTARG ;;
'?') echo "Usage: que [-cp][-f file | file ]" >&2 && exit 1 ;;
esac
done
shift $(($OPTIND - 1))
file=${file:-$dir/$1}
if [ ! -f "$file" ]; then
echo "Error: Queue $file does not exist" >&2 && exit 1
fi
# update the queue
if (grep -q $mark $file); then
task=$(grep -n $mark $file | sed 's/:.*//')
else
task=1
fi
if [ "$prcur" = yes ]; then
if [ $task -gt 1 ]; then
sed -n "$(($task - 1))p" $file
fi
exit
fi
sed -i "s/$mark//" $file
if [ "$clear" = yes ]; then
exit
fi
sed -n "${task}p" $file
sed -i "$(($task + 1))s/$/$mark/" $file
To use this script you must first write a queue file in $HOME/.queues. (the -f flag allows you to supply a queue file in a non-standard directory, the -p flag prints the current ongoing task without advancing the queue) It could for instance be called "homework", and look like this:
Math: Read chapter 3.12
Math: Do exercises 3.12.1 through 3.12.20
Math: Prep for exam
English: Read the poem from Keats
English: Write essay on Keats
...
If you now run que homework it will print Math: Read chapter 3.12, and the 2nd line of this text file will change to: Math: Do exercises 3.12.1 through 3.12.20<--. And the next time you run que homework, this line will be printed, and the arrow will be moved to the 3rd line, and so on. The concept here is similar to a 2do list, but useful in situations where you need to do tasks in a sequential order, such as reading chapters of a book. I find this script to be very useful if I want to automate such a task. For instance, are you following a number of TV shows? You could write a queue file for Star Trek for instance, and add this line to your ~/.profile (or ~/.bashrc): alias tos='mplayer $(que tos)'. So that the next time you want to watch Star Trek, just type tos, and it will automatically play the next episode for you.
The cal command can print a calendar if you need it. To schedule appointments however, the classic UNIX choice is calendar. Practically every version of UNIX has this command, with just one notable exception: Linux. Just one of those little reminders that this operating system does not have deep roots in the UNIX world. Speaking of reminders, remind is a good alternative to calendar, and you can find it in most package repositories. remind also has a nice ncurses front-end, wyrd.
To use calendar (or remind) write your reminders in a plain text file in ~/calendar. Your appointments must start with a date, followed by a tab (not a space), and finally a description that must fit into exactly one line (it can be as long as you want though). For example:
Jan 03 Give borrowed towel back to neighbor
Apr 27 Dentist appointment - flee the country!
May 04 Star Wars marathon!!!
If you now run the command calendar, and it happens to be April the 26th, it will print Apr 27 Dentist appointment - flee the country! Which will, at the very least, give you a days notice to buy plane tickets. The command will print appointments for today and tomorrow, or on a Friday, from Friday through Monday. Naturally, the trick to making this useful, is to set things up so that reminders are automatically printed whenever you log in to your machine, for example, by adding this command to ~/.profile or ~/.bashrc (the command remind -t1 $HOME/calendar will essentially do the same as the above mentioned calendar command).
PS: One weakness of calendar is that it only gives you a couple of days notice before an event. Every so often, I feel the need to look further ahead and see what upcoming events are coming in the next few weeks. We'll discuss alternatives to calendar in the next paragraph, but if all you want is to see the next, say 20, upcoming appointments, a simple awk script will suffice:
#!/bin/sh
# calmon - check calendar(1) appointments upcoming month-ish
# usage: calmon
# bugs: calendar(1) db must be formated "Mon d\tEvent..."
# no output if no appointment in current month
cal=$HOME/calendar
next=20
# stop 19 lines after first appointment in current month
awk "/$(date +%b)/ && \$2 > $(date +%e) && ! stop { stop=NR+$next-1 }
stop { print }
NR == stop { exit }" $cal
The classic calendar command is great in all its simplicity, but there is something gratifying about crossing out dates on a visual calendar in front of your eyes. Sure you can run cal manually, and work out when your appointments are due, and then run date to double check that you remembered todays date correctly. But it is all a bit cumbersome, are there more user friendly alternatives? You bet!
A good ncurses calendar and 2do list program is calcurse. Besides being a nice looking curses app, it can export and import to ical formats, which means that you can use it to manage your Google Calendar. Of course, we could also write a script ourselves that wraps a nicer interface around the commands cal, calendar and date. Here is one suggestion:
#!/bin/sh
# mcal - print marked cal(1), date(1) and calendar(1)
# usage: mcal [-f file]
# bugs: can only print current date
# set some defaults
FILE=$HOME/calendar
if [ $1 = -f ]; then
FILE=$2
shift 2
fi
month=$(date +%b | tr A-Z a-z | sed \
-e 's/a/[Aa]/;s/b/[Bb]/;s/c/[Cc]/;s/d/[Dd]/;s/e/[Ee]/;s/f/[Ff]/' \
-e 's/g/[Gg]/;s/i/[Ii]/;s/j/[Jj]/;s/l/[Ll]/;s/m/[Mm]/;s/n/[Nn]/' \
-e 's/o/[Oo]/;s/p/[Pp]/;s/r/[Rr]/;s/s/[Ss]/;s/t/[Tt]/;s/u/[Uu]/' \
-e 's/v/[Vv]/;s/$/.*/')
day=$(date +%d | sed 's/^0//')
# print marked calendar
for dt in $(awk '/'$month'/ { print $2 }' $FILE | uniq); do
mdate="$mdate $dt"
done
cal | awk -v day=$day -v mdate="$mdate" '
BEGIN { split(mdate, mdt) }
NR <= 2 { print }
NR > 2 { split($0, line)
for(wd=1; wd<=NF; wd++){
if(line[wd] == "1" && NF < 7)
for(i=1; i<8-NF; i++)
printf("%2s ", " ")
for(dt=1; dt<=length(mdt); dt++)
if(line[wd] == mdt[dt] || line[wd] == day) markdt=1
if(markdt){
if(line[wd] == day)
printf("%2s ", "*")
else
printf("%2d ", line[wd])
markdt=0
} else
printf("%2s ", "-")
}
printf("\n")
}'
echo
date
echo
calendar -f $FILE
Lets ignore the gruesome complexity of this script for now, and just focus on how to use it: After adding your appointments to the calendar database, mcal will print cal with all the dates dashed out, except those who have a calendar appointment. It will also mark todays date with an *, and it will print todays date in full with any upcomming appointments. To demonstrate:
$ echo 'apr 1Ctrl+v TabDentist appointment - DIE!' >> ~/calendar
$ echo 'apr 2Ctrl+v TabApril fools day is finally over' >> ~/calendar
$ echo 'apr 27Ctrl+v TabReal dentist appointment - flee the country!' >> ~/calendar
$ mcal
April 2018
Su Mo Tu We Th Fr Sa
* 2 - - - - -
- - - - - - -
- - - - - - -
- - - - 26 - -
- -
Wed Apr 01 13:22:04 CET 2018
apr 1 Dentist appointment - DIE!
apr 2 April fools day is finally over
As you can see, mcal does not really give you much functionality beyond what date, cal and calendar already provides, rather it tries to build a convenient interface around them. The rather horrid month= line followed by a bunch of unreadable sed commands, converts a date, such as jan, into a regex pattern, in this case [Jj][Aa][Nn].*. The effect is that awk is able to find calendar appointments somewhat intelligently, Jan 1 or january 1 are both valid for instance. It is not intelligent enough however to allow the full range of date formats that calendar supports. Appointments must be written as an English months followed by a date (you'll note that we typed Ctrl+v Tab in the above example to make echo write a literal Tab character).
Next, we loop over the calendar database and make a variable called mdate, which holds all the dates with appointments (the "marked" dates). With that it place we execute our awk program. It prints the first two lines of cal as is, then it loops over the week days, prefixing the 1st date with spaces as necessary, printing the date as "*" if it is the current date, as the date number (line[wd]) if it has a calendar appointment, or "-" if it doesn't.
You can easily adjust this script to do the reverse, that is, dash out the dates with a calendar appointment, but keep the dates without an appointment untouched. This can be useful if you are not so much interested in reminders, as you are in the motivation of seeing dashed out dates for days when you managed to do whatever you wanted to do. Perhaps you even want to add a -v flag for reversing the default behavior of mcal, or you can make up some other scheme, such as adding the color codes red for todays date and yellow for appointments. The main point to take away from this, is that assembling basic UNIX commands together to form a more convenient interface is often fairly easy. A little awk knowhow goes a long way in that regard. Of course, there are even simpler ways to mark the calendar; You could just run cal 2018 > mycal, then edit this to your hearts content with your favorite editor.
There are a few password managing utilities out there for the console, but none are really ubiquitous (search your repository for "password"), and this is a very simple task to accomplish with a script anyway. Provided you have a tmpfs /tmp partition, you can safely edit a plain text database of passwords in /tmp, since this will only be written to memory, and then encrypt it to your disk using GnuPG (it might be called gpg or gpg2 on your system):
#!/bin/sh
# pass - a simple password manager
# usage: pass [-e] [ keywords... ]
# bugs: assumes a safe tmpfs on /tmp
# depends: GnuPG (either gpg or gpg2), EDITOR
# set some defaults
gpg=$(which gpg 2>/dev/null) || gpg=$(which gpg2 2>/dev/null) || exit 1
file=${file:-$HOME/.passwords.gpg}
temp=${temp:/tmp/.passwords}
edit=${EDITOR:-$(which vi 2>/dev/null} || exit 1
# no args: print password file
if [ $# = 0 ]; then
$gpg --decrypt $file
exit
fi
# -e encrypt; keyword search...
trap "rm -f $temp" 0 1 2 15
case $1 in
-e) $gpg --decrypt $file >$temp
$edit $temp
$gpg --symmetric $temp
mv $temp.gpg $file ;;
*) $gpg --decrypt $file | grep -i "$@" ;;
esac
You can write your password database however you want. Personally, I write a tab separated database using these fields: Category Name Username Password Email Website Comments.... But that's just me, however you choose to write your passwords, do so in a single line, and try to be consistent with yourself. You can now search for passwords with pass searchterm. If you later need to add or edit the password database, run pass -e, your changes will automatically be reencrypted to disk when you are done. The script will exit silently if you don't have GnuPG or vi (or other EDITOR) on your system.
PS: Make sure that your /tmp partition is a memory filesystem, since you will be editing your passwords in a plain text file. If /tmp is in memory, this isn't a big concern, since the passwords will only exist in memory while you are editing the database, and then safely removed afterwards. But beware of text editors making automatic backups! PPS: Shredding contents of files after writing them to disk is not safe! There is no way to guarantee that data is overwritten on modern harddisks.
This simple script can be used to automatically authenticate other services, as an example the following script can be used to search the encrypted database for wireless networks and connect to it:
#!/bin/sh
# con - manage wifi connections
# usage: con name
# bugs: assumes a safe tmpfs on /tmp
# depends: pass, wifi
# set some defaults
wlan=${wlan:-wlan0} # change to match your wifi device
temp=${temp:/tmp/$$-wifi}
touch $temp || exit 1
# query the database
trap "rm -f $temp" 0 1 2 15
pass "$@" > $temp
matches=$(cat $temp | wc -l | sed 's/^[ ]*//')
case $matches in
0) echo Error, cannot find network! ;;
1) name=$(awk -F\t '{ print $3 }' $temp)
pass=$(awk -F\t '{ print $4 }' $temp)
sudo wifi $name $pass
;;
*) awk -F\t '{ print $2, $3, $4 }' $temp | column -t ;;
esac
The script relies upon the conventions mentioned above, that you store the wifi passwords in a tab separated file encrypted by pass, and that the 3rd field is the network name, the 4th field the password, and so on. The script does not handle the details of actually connecting to a wireless network, since that greatly depends upon which system you are using. Instead it assumes you have a script called wifi, which knows how to do that given the network name and password as arguments. Se the wifi section above for clues on how to write such a script.
You can use ledger to manage your accounts, and we already discussed how to make an accounting script yourself with sqlite. But lets explore how to do so with a simple shell script:
#!/bin/sh
# account - basic account manager
# usage: account [-d YYYY-MM-DD][-c catg] [$$.CC [ comments... ]]
# depends: budget
# set some defaults
account=${account:-$HOME/.account}
budget=${budget:-$HOME/.budget}
date=$(date +%Y-%m-%d)
catg=food
for arg in "$@"; do
case "$arg" in
-d) date=$2 && shift 2 ;;
-c) catg=$2 && shift 2 ;;
esac
done
if (! echo $date | grep -s '^[12][0-9][0-9][0-9]-[01][0-9]'); then
echo Error: invalid date, use the form YYYY-MM-DD
exit 1
fi
month=$(echo $date | awk -F- '{ print $2 }')
year=$(echo $date | awk -F- '{ print $1 }')
# no args, print monthly report
if [ $# = 0 ]; then
echo Report for $month, $year
grep "^$year-$month" $account | awk -f $budget
exit
fi
# with args, add record to account
amount=$1
if (! echo $amount | egrep '^[0-9]+(\.[0-9][0-9])?$')>/dev/null; then
echo Error: invalid number, use the form 100 or 100.00 with no prefix
exit 2
fi
shift
if [ ! $catg = income ]; then
amount=-$amount
fi
echo $date $amount $catg $@ >> $account
This script simply stores account records. It has no understanding of currency, it just assumes all records are expenses and will convert them to negative integers, unless the category is income, in which case the number remains positive. The category defaults to food, since this is assumed to be your most frequently used expense. The date is set to your current date by default. Dates must follow the convention YYYY-MM-DD, but YYYY-MM will suffice if you just want a monthly budget report.
Monthly budget report you say? Ah yes, I haven't shown you how to do that yet. The account script calls an external awk script to produce a report of your expenses if you don't supply it with any arguments. Here is an example of what that script could look like:
BEGIN { printf("%-s\n", "------------------------------") }
/ income/ { income+=$2 }
/ rent/ { rent+=$2 }
/ food/ { food+=$2 }
/ car/ { car+=$2 }
{ sum+=$2 }
END {
printf("%-10s%10.2f %-10s\n", "income:", income, "of 2000")
printf("%-10s%10.2f %-10s\n", "rent:", rent, "of -500")
printf("%-10s%10.2f %-10s\n", "food:", food, "of -500")
printf("%-10s%10.2f %-10s\n", "car:", car, "of -500")
printf("%-10s%10.2f %-10s\n", "misc:",
sum - (income + (rent + food + car)), "of -500")
printf("%-s\n", "------------------------------")
printf("%-10s%10.2f\n", "Balance:", sum)
}
Of course I assume your own financial status is somewhat more complex then this, but the example given should be clear enough that you can expand and adapt it to your own needs. You could integrate this into the account script so that everything is neatly packed into a single file if you want, but if your budget changes or more than one person uses account, it might be better to keep these files separate (or you could write a smart piece of code that can interpret a simpler user supplied budget). Lets demonstrate how to use account:
$ account -d 2018-08-25 -c rent 500
$ account -d 2018-08-25 -c income 2000
$ account -d 2018-08-26 5 starbucks
$ account -c car 254.35
$ account 14.50 groceries
$ account -c cloths 20 jacket
$ account
Report for 08, 2018
------------------------------
income: 2000.00 of 2000
rent: -500.00 of -500
food: -19.50 of -500
car: -254.35 of -500
other: -20.00 of -500
------------------------------
Balance: 1206.15
$ cat .account
2018-08-25 -500 rent
2018-08-25 2000 income
2018-08-26 -5 food starbucks
2018-08-28 -254.35 car
2018-08-28 -14.50 food groceries
2018-08-28 -20 cloths jacket
$ grep '^2018-0[6-8]' .account | awk '/cloths/ { cloths+=$2 } END { print cloths }'
-20
The account script is very basic, it only supports a single account and has no tag support, but you can use comments for this and you can keep separate accounting files for different accounts. For example, a busy family provider could add this to his ~/.profile:
alias work="account=$HOME/.workaccount budget=$HOME/.workbudget account"
wife(){
account "$@" wife
}
kids(){
account "$@" kids
}
self(){
account "$@" beer
}
The last line of the previous demonstration also shows how to summarize all expenses used on cloths for June to August 2018, feel free to play around with awk and gnuplot to mine the database for more useful information. Since the account database is just plain text it is easy to convert and integrate it into other programs (such as SQL), for instance you could convert it to CSV and open it in LibreOffice: sed -e 's/ /,/' -e 's/ /,/' -e 's/ /,/' .account > account.csv ; localc account.csv
Again, this is stupefyingly easy to just do yourself (hint: nano and grep should suffice). If you really want a dedicated program for this abook is a good choice.
But lets elaborate. How would you sync an address book with the contact information in /etc/passwd? Suppose you have a private contact database in $HOME/.tel, which is just a plain text file with the entries "name, address, telephone (work,private), email" per line. You can create a tel command by putting this in $HOME/.bashrc:
tel(){
grep "$@" $HOME/.tel
}
You can now search this database with tel name (or you can search for a number or address or whatever). Now the following script will read all the user contact information in /etc/passwd, ignore any entries that do not contain email addresses (email's can be added by the sysadmin in the other (chfn -o) category) and update your local contact database:
#!/bin/sh
# updatetel - update tel db with /etc/passwd entries
# usage: updatetel
tel=${tel:-$HOME/.tel}
IFS='
'
for entry in $(awk -F: '/@/ { print $5 }' /etc/passwd |\
awk -F, '{ printf("%s\t%s\t%s,%s\t%s\n", $1, $2, $3, $4, $5) }'); do
name=$(echo $entry | sed 's/ .*//')
if (grep -q "^$name" $tel); then
sed -i "s/^$name.*/$entry/" $tel
else
echo $entry >> $tel
fi
done
As long as the sysadmin, or the employees themselves, keep the contact information correct here, the user can run updatetel periodically (eg. once a week with cron), and all the company contacts should be up to date. At the same time the user is free to add whatever private contacts he wants to his own database. And since this database is just a plain text file, it can be easily synced to his home computer, or phone. And it would be trivial to convert the data to a spreadsheet or SQL database, or whatever poison consumers may demand nowadays.
As we have seen, you can read text and PDF files, listen to music and watch video on the console. But this involves fiddling about finding the file first, and worse the program does not usually remember where you left of last time. Is there a way to bookmark PDF files and videos on the console, so that we can automatically continue from where we left of last time, like we expect from fancy desktop document readers and media players? Well, sort of, here is an example:
#!/bin/sh
# bm - bookmark manager
# usage: [ search... ][ -r file... ][ file page comments... ]
# set some defaults
bm=${bm:-$HOME/.bookmarks}
touch $bm
# parse arguments
if [ "$1" = -r ]; then
remove=yes && shift
fi
if [ $# = 0 ]; then
cat $bm && exit
elif [ $# = 1 ]; then
matches=$(grep "$1" $bm | wc -l)
if [ $matches = 0 ]; then
echo No matches found for $1 && exit
elif [ $matches -gt 1 ]; then
grep "$1" $bm && exit
else
if [ "$remove" = yes ]; then
file=$(echo $1 | sed 's/\//\\\//g')
sed -i "/$file/d" $bm; exit
fi
file=$(grep "$1" $bm | awk '{ print $1 }')
page=$(grep "$1" $bm | awk '{ print $2 }')
if (ps ax | grep X | grep -vq grep); then
# use these on the desktop
case $file in
*.pdf) exec xpdf $file $page ;;
*.ps) exec gv --page $page $file ;;
*.?htm?) exec firefox "file://$file#$page" ;;
(*.avi|*.mp*|*.ogg|*.wav) exec mplayer -ss $page $file ;;
esac
else
# use these on the console
case $file in
(*.pdf|*.ps) exec fbgs -fp $page $file ;;
*.?htm?) exec lynx "file://$file#$page" ;;
(*.mp3|*.ogg|*.wav) exec mplayer -ss $page $file ;;
# ps: Linux only, also adjust the resolution if necessary
(*.avi|*.mp*)
exec mplayer -vo fbdev2 -vf scale=1366:768 -ss $page $file ;;
esac
fi
# use these for both desktop and console
case $file in
*.t?xt) exec vim +$page $file ;;
(*.c|*.cpp|Makefile|*.pl|*.rb|*.py|*.php|*.sh)
exec vim +$page $file ;;
(*.el|*.cl|*.lisp|*.sbcl) exec emacs +$page $file ;;
*) echo I don't know how to open $file && exit ;;
esac
fi
fi
# resolve relative/absolute path
if (echo $1 | grep -s '^/'); then
file=$1
else
file=$(pwd)/$1
fi
page=$2 # ps: page number is required!
shift 2
# update/add entry in bookmarks
if (awk '{ print $1 }' $bm | grep -q $file); then
sed -i "s,^.*$file.*$,$file $page $@," $bm
else
$file $page $@ >> $bm
fi
And a little demonstration of its use:
$ bm Documents/p/plan9/fqa.pdf 177 I wonder if you can have Go in Plan9?
$ bm Documents/a/articles/example.html some_tag something I meant to publish...
$ bm Music/podcast/linuxvoice/lv_s05e17.ogg 1:02:02
$ bm Music/podcast/linuxvoice/lv_s05e18.ogg 30:00
$ bm
/home/dan/Documents/p/plan9/fqa.pdf 177 I wonder if you can have Go in Plan9?
/home/dan/Documents/a/articles/example.html some_tag something I meant to publish...
/home/dan/Music/podcast/linuxvoice/lv_s05e17.ogg 1:02:02
/home/dan/Music/podcast/linuxvoice/lv_s05e18.ogg 30:00
$ bm fqa # open fqa.pdf on page 177 with xpdf
$ bm publish # open example.html at tag "some_tag"
$ bm linuxvoice
/home/dan/Music/podcast/linuxvoice/lv_s05e17.ogg 1:02:02
/home/dan/Music/podcast/linuxvoice/lv_s05e18.ogg 30:00
$ bm lv_s05e18 # start playing lv_s05e18.ogg 30 minutes into the audio file
$ bm Music/podcast/linuxvoice/lv_s05e18.ogg 45:00
$ bm -r lv_s05e17
$ bm -r example
$ bm
/home/dan/Documents/p/plan9/fqa.pdf 177 I wonder if you can have Go in Plan9?
/home/dan/Music/podcast/linuxvoice/lv_s05e18.ogg 45:00
This script contains some Kong Fu ASCII puke, sed 's/\//\\\//g' is particularly jarring. And to be sure, it is a long and tedious script, but the logic is simple enough, and it should be fairly straight forward to edit the script and adjust your favorite application to their file types. PS: This example is not intended as a magic bullet, you really should edit it to suit your own workflow!
Also note, that like a real bookmark, you actually need to place a bookmark in the file you want to save, by supplying a page number (if it's an audio/video file the "page" is the time you stopped playing). You will find that, just closing the book and staring madly at the bookmark for not jumping to the correct page, is of little benefit. If you only want to find a file, use locate or find instead. In theory you could add support for online bookmarking and other things, if you feel hackish (the provided HTML support assumes a local HTML file).
Some modification tips: First of all only Linux can play video files in the framebuffer, but you probably need to adjust resolution flags and set things up (see the video section above). You may also wish to change some of the default programs, eg. vim to nano or less, both use the +linenr syntax (you might want to take a chance and just use vim or less if a file prefix wasn't recognized). You can also change both the PDF and Postscript viewer to okular --page or evince --page-index, or a different PDF reader, but be aware that not all viewers support this feature. Finally, you could use vlc --start-time instead of mplayer, but be aware that vlc only supports start time in seconds. You cannot type vlc --start-time 10:00, you must instead use --start-time 600. Of course you could always edit bm so that it converts minutes to seconds, but I leave that as an exercise for the reader.
Sidenote: A more elegant bookmarking solution might be to use the Plan9Port plumber.
You can easily check what the time is with date. Or if you prefer to keep a digital clock around, you can use tty_clock. And naturally tmux comes preconfigured with a digital clock in the lower right corner. If you need to schedule some task at a particular time, commands such as at, cron and anacron can help; if you need to run a one time command, run a server command regularly, or run a laptop command semi-regularly, respectively.
The time command is primarily used to measure how much time a program uses before it exits. But you can use it to time yourself. Just type time read, and hit Ctrl + d (this key combo means "end of file") when done. time prints three numbers; system time, relevant for kernel hackers. user time, relevant for hackers. And real time, relevant for real people.
Perhaps you need a more elaborate time keeping program though, to measure how much time you spend on various projects. Here is one solution:
#!/bin/sh
# track - keep track of time
# usage: [[-r][-s][ start|stop ] project comments... ]
dir=${dir:-$HOME/.track}
mkdir -p $dir
output=${output:-60} # ei. output in minutes
if [ $# = 0 ]; then
for file in $(ls $dir | grep -v ongoing); do
if [ -f $dir/${file}_ongoing ]; then
echo $file '(ongoing)'
else
echo $file
fi
done
elif [ $# = 1 ]; then
cat $dir/$1
elif [ $1 = -r ]; then
# remove file
rm $dir/$2
elif [ $1 = -s ]; then
# summarize time used on this project
awk '{ sum+=$2 } END { print sum }' $dir/$2
elif [ $1 = start ]; then
date "+%s" > $dir/${2}_ongoing
elif [ $1 = stop ]; then
file=$2
shift 2
start=$(cat $dir/${2}_ongoing)
rm $dir/${2}_ongoing
stop=$(date "+%s")
time=$(echo "scale=2; ($stop - $start) / $output" | bc)
echo $(date "+%Y-%m-%d") $time "$@" >> $dir/$file
fi
You can then use it like so:
$ track start work
$ track start solitaire
$ track stop solitaire all work and no play makes jack a dull boy
$ track
work (ongoing)
solitaire
$ track solitaire
2018-04-26 5.02 all work and no play makes jack a dull boy
$ track -s solitaire
5.02
Again, this script is fanatically simplistic. In particular it's perception of time is very naive. It reports time used in minutes with two decimal points, so 1 hour, 11 minutes and 12 seconds, will be reported as: "71.20." This stupidity is somewhat ironic since the purpose of this program is to help you keep an accurate track of time.
You can modify the output to show a more human readable time format with yet another script, for example:
$ humantime(){
> seconds=${1#*.}
> seconds=$(echo "($seconds * 60) / 100" | bc)
> minutes=${1%.*}
> hours=$(echo $minutes / 60 | bc)
> minutes=$(echo "$minutes - ($hours * 60)" | bc)
> echo $hours hours, $minutes minutes and $seconds seconds
> }
$ humantime $(track -s solitaire)
0 hours, 5 minutes and 1 seconds
Why isn't this functionality included in the script? After careful consideration I choose not to add it for a couple of reasons: First of all the simplistic format makes it very easy to use the tracking data with external programs, such as drawing a graph with gnuplot. Or you can easily generate reports with it in awk, calculating percentages, and even merge or split tasks if you need to. Such tasks would be massively more complex if track reported time as "1 hour, 11 minutes and 12 seconds."
Secondly, and most importantly, I wanted to write a simple example. Handling time correctly is anything but simple. Suppose you wanted to write it in the format "HH:MM:SS", how would you make sure the minutes are exactly two digits (eg. "02" not "2" minutes)? What about the hours? Suppose a project has taken 300 hours, should you split those into days? Of what length, should you use a literal 24 hour day or an 8 hour work day. If you use 8 hour work days, should you skip weekends and holidays? Do you begin to see the problem..? And thus, I lazily give you this crude script, and leave it as an exercise to you dear reader, to extend it to something that's actually useful ;^)
The purpose of this howto is to explain how to use the console as a desktop, so why are we talking about the X Window System (the classic graphics backend in UNIX)? Well, realistically you may need to dip your toes into a graphical desktop once in a while. Don't be ashamed, even Stallman uses a desktop sometimes. Sadly though there is little convergence between the UNIX console and its desktop, the reason being that the desktop was created later, and without much thought to existing design philosophy. Is it possible then to integrate these two worlds, and thus create an effective working environment also on the desktop?
The easiest way to move data between the console and the desktop is simply to redirect output to files. For instance say you wanted to import the output of ps ax to LibreOffice, you could do this:
$ ps ax | sed 's/^ *//' |
> sed -e 's/ */,/' -e 's/ */,/' -e 's/ */,/' -e 's/ */,/' > ps.csv
$ localc ps.csv
This simple but cumbersome sed line converts the first four spaces to commas (there's two spaces in 's/ */,/'), this transforms the data into a comma separated file, which can be opened nicely in LibreOffice Calc (ei. localc). This example illustrates that it is sometimes necessary to transform the text a bit before you can feed it other programs, especially programs that don't follow normal UNIX conventions. Tools such as sed, tr, fmt, not to mention awk, are essential for such tasks.
Redirecting text from a graphical program to a file can be more tricky though. The X desktop actually has two clipboards, the classic xclipboard works by first selecting text with left-click and drag, then middle click where you want to paste it. The marked text is automatically copied into the xclipboard, so you don't need to do a manual copy action. The command line tool xclip can be used to manipulate xclipboards. For instance you can find a nice Youtube video in Firefox, mark the URL (Ctrl-L), then type youtube-dl $(xclip -o) in a terminal to download it (of course you could also have typed youtube-dl and middle clicked).
The newer X selections, on the other hand, tries to mimic Microsoft Windows, where you first mark the text, then right-click and select "copy," then right-click and select "paste" (you can also use keyboard shortcuts for this). The command line tool for manipulating these X selections is xsel.
In addition tmux has quite advanced clipboard management that can be integrated with X in various ways. Look into it if your interested :^)
Sometimes you need to move a process, say from the console to a terminal running in the desktop (or vice versa). reptyr is your friend (Linux and FreeBSD only). Another example of its usefulness: Say its late at the office and you want to go home, but you started a long running process over SSH that cannot be interrupted, and you forgot to run tmux first. What do you do? Log in to the server again, run tmux, then use reptyr to move the process over to tmux, you can now kill the ssh session and head on home, the process will continue to run in the detached tmux session.
If you have framebuffer support (Linux only) you can use fbcat or fbgrab to take a screenshot of your console, and ffmpeg can be used to obtain a full video screencapture:
The last two examples are for desktop recording, but you should be able to tweak them in order to record the framebuffer console (hint use -f fbdev instead of -f x11grab). These last examples is very hardware demanding though, and will likely produce a bad result, a simple workaround is to capture a smaller resolution. Either set a smaller desktop resolution, or capturing a smaller nested X desktop (see Xephyr), or just record a corner of your desktop.
For recording plain text you can use ttyrec or the venerable script tool, note however that script will not work well with interactive applications, such as vi. If you need to record text editing, either use ed or record your vi sessions with ttyrec.
The reason working on the console is so efficient, is not because it lacks graphics per say, but rather that it's a text oriented, distraction free environment. Can you create such an environment in X?
Yes. Tiling window managers are the answer. These work essentially as tmux or screen on the console, and there are a great many to choose from. I would personally recommend dwm, its lightweight, easy to use and has all the features you will likely need (there are many other good alternatives). Just get it and read the manpage.
In addition you can replace some graphical applications, and desktop features, with alternatives that are keyboard friendly, and have minimal interfaces. Some suggestions:
Naturally you can also use all the console applications mentioned here in an X terminal. Be sure to check out the ASCII art and Retrofuturism sections as well, for some cool tips on making your desktop look retrotastic!
We have now talked a bit about moving the console into your desktop, but is it possible to do it the other way around? There are at least two options available for Linux. twin is a simple stacking window manager for the console using ncurses graphics, you can create new terminal windows from its drop down menu, resize windows and move them about with your mouse. Not very useful in my humble opinion, but cool non the less.
Speaking of not-so-useful-but-cool, the libCACA team has created an experimental fork of screen called neercs (that is "screen" spelled backwards). It works much like screen, but has some ridiculous extended features, such as an ASCII 3D spinning desktop cube. You will probably not find it in the repo, but you can get it like so: git clone git://git.zoy.org/neercs.git (PS: This program is buggy, slow and unmaintained. Like Compiz, I would not recommend it for daily use, but its fun to keep around in case you want to knock the socks off some unsuspecting Windows user).
Most Linux users are vaguely aware that the console exists, and many sysadmins use it daily to administrate their servers, but as we have seen in this article the console can work reasonably well as an everyday desktop. Tasks such as basic HTML browsing, email and chatting works very well in the console. As does playing music and text games. Lastly, office tasks such as writing documents, using spreadsheets and doing PIM related tasks is easy. On a Linux console with a graphical framebuffer enabled, you can even view pictures, PDF's, videos and play graphical games!
There are limitations though. Obviously graphic heavy tasks, such as triple A games and 3D movie production, is not well suited for the humble console. The most painful limitation of all is the lack of a modern web browser. Realistically this limitation cannot be solved, creating a text browser capable of rendering pages as well as Google Chrome, would literally require more effort then recreating the Linux operating system from scratch! Nevertheless, dipping your toes into the console, and learning how to use it efficiently, can greatly boost your productivity. These skills are absolutely useful to have, even on the desktop!
Most of all, it's great fun :^)
