HON95-wiki/config/general/linux-general.md

---

title: Linux General breadcrumbs:

• title: Configuration
• title: General --- {% include header.md %}

Resources

Security

• Linux Hardening Checklist (trimstray)
• The Practical Linux Hardening Guide (trimstray)

Information

Distros

Debian

• Nobody user and group: nobody:nogroup
• List of default groups: SystemGroups (Debian Wiki)
• Release info file: /etc/debian_version

RHEL

• Nobody user and group: nobody:nobody
• Release info file: /etc/redhat-release or /etc/centos-release

Miscellanea

• urandom VS random: random blocks when running out of entropy while urandom does not. For all practical purposes, urandom will almost never be less random than random and random may block at inappropriate times, so always use urandom.

Bugs

• Environment variables from */environment.d/*.conf aren't visible for login sessions when using systemd. */environment.conf works, though. See systemd#7641.

Commands

AAA

• Sudo:
  • Show sudo permissions for current user: sudo -l

Executables:

• Show type and info: file <executable>
• Show library dependencies (for glibc): ldd <executable>
  • It lists shared library file names and memory location to map it to, or "statically linked" if no dynamic dependencies.
  • Alternatively, run the executable with envvar LD_TRACE_LOADED_OBJECTS=1 set instruct glibc to print dependencies and exit. This is basically how ldd works internally, but with more options.
  • Warning: This might execute the program if not using glibc.
• Show printable character strings in the executable: strings -a <executable>
• Run application and show dynamic library calls: ltrace <executable> [args]
• Run application and show system calls: strace <executable> [args]
• Strip the symbol table and debug info from an executable: strip <executable>
  • Without certain options, it will still keep some useful info in the file.
• Rebuild the symbol table for a statically linked executable: See gensymtab.

File Systems and Logical Volume Managers

• Partition disk: gdisk <dev> or fdisk <dev>
• Create filesystem: mkfs.<fs> <dev>
• Modify fstab:
  • Test it with mount -a to make sure it doesn't have errors that may cause boot to fail.
  • Run systemctl daemon-reload to avoid having systemd remount stuff that was removed from fstab or other weird shit.
• Benchmark with IOzone:
  • Install (Debian): apt install iozone3
  • It uses the current dir.
  • Test with various record sizes and file sizes: iozone -a
  • Benchmark: iozone -t1 (1 thread)
  • Plot results: TODO It should be doable with gnuplot somehow.
• Shred disk with shred:
  • Example: shred -n2 -v <file>
  • -n<n> speficied the number of passes. This takes ages to begin with for large disks, so keep it as low as appropriate. 1 pass is generally enough, 2 to be sure.
  • --zero adds an extra, final pass to write all zeroes.
  • -v shows progress.

Files

• Search:
  • By UID: find / -user <UID>
  • Without a user: find / -nouser
  • With setuid permission bit: find / -perm /4000
  • Recursive search and replace: find <dir> \( -type d -name .git -prune \) -o -type f -print0 | xargs -0 sed -i 's/123/456/g'
    • -type d -name .git -prune skips .git directories and can be excluded outside of git repos.
• Space usage:
  • du -sh <dirs>
  • K4DirStat (GUI) (package k4dirstat)
• Shred files:
  • shred --remove --zero -v <file>

Fun

• Color text from STDIN: lolcat
• cowsay
• fortune

Monitoring (General)

• For more specific monitoring, see the other sections.
• htop:
  • ncurses-based process viewer like top, but prettier and more interactive.
  • Install (APT): apt install htop
  • Usage: htop (interactive)
• glances:
  • Homepage: nicolargo.github.io/glances/
  • Install (PyPI for latest version): pip3 install glances
  • ncurses-based viewer for e.g. basic system info, top-like process info, network traffic and disk traffic.
  • Usage: glances (interactive)
• dstat:
  • A versatile replacement for vmstat, iostat and ifstat (according to itself).
  • Prints scrolling output for showing a lot of types of general metrics, one line of columns for each time step.
  • Usage: dstat <options> [interval] [count]
    • Default interval is 1s, default count is unlimited.
    • The values shown are the average since the last interval ended.
    • For intervals over 1s, the last row will update itself each second until the delay has been reached and a new line is created. The values shown are averages since the last final value (when the last line was finalized), so e.g. a 10s interval gives a final line showing a 10s average.
    • The first line is always a snapshot, i.e. all rate-based metrics are 0 or some absolute value.
    • If any column options are provided, they will replace the default ones and are displayed in the order specified.
  • Special options:
    • -C <>: Comma-separated list of CPUs/cores to show for, including total.
    • -D <>: Same but for disks.
    • -N <>: Same but for NICs.
    • -f: Show stats for all devices (not aggregated).
  • Useful metrics:
    • -t: Current time.
    • -p: Process stats (by runnable, uninterruptible, new) (changes per second).
    • -y: Total interrupt and context switching stats (by interrupts, context switches) (events per second).
    • -l: Load average stats (1 min, 5 mins, 15 mins) (total system load multiplied by number of cores).
    • -c: CPU stats (by system, user, idle, wait) (percentage of total).
    • --cpu-use: Per-CPU usage (by CPU) (percentage).
    • -m: Memory stats (by used, buffers, cache, free) (bytes).
    • -g: Paging stats (by in, out) (count per second).
    • -s: Swap stats (by used, free) (total).
    • -r: Storage request stats (by read, write) (requests per second).
    • -d: Storage throughput stats (by read, write) (bytes per second).
    • -n: Network throughput stats (by recv, send) (bytes per second).
    • --socket: Network socket stats (by total, tcp, udp, raw, ip-fragments)
  • Useful plugins (metrics):
    • --net-packets: Network request stats (by recv, send) (packets per second).
  • Examples:
    • General overview (CPU, RAM, ints/csws, disk, net): dstat -tcmyrdn --net-packets 60
    • Network overview (CPU, ints/csws, net): dstat -tcyn --net-packets 60
    • Process overview (CPU, RAM, ints/csws, paging, process, sockets): dstat -tcmygp --socket 60

Hardware

• Show hardware: lshw
  • Run as root for more info.
  • Specify -X to show GUI (requires lshw-gtk).
• Show hardware topology: lstopo (requires hwloc)
  • lstopo will try to present as a GUI. Use lstopo-no-graphics to force console output.
• Show PCI devices: lspci
• Show block devices: lsblk
• Show USB devices: lsusb
• Show CPUs: lscpu

Software Installation

APT (Debian)

• Find packages depending on the package: apt rdepends --installed <package>
• Quickly add new repo: add-apt-repository <repo-line
  • It will add the line to /etc/apt/sources.list, where you can manually remove it again.
• Keys:
  • List: apt-key list
    • It will also show which file contains it.
  • Remvoe key: apt-key del <key-id>
    • The 8-digit hex key ID may either be found on pub line or as the last 8 hex digits on the continuation line.

Pacman (Arch)

• Full system upgrade: pacman -Syu
• Search package: pacman -Ss <package-name>
• Show package info: pacman -Si <package>
• Install package: pacman -S <packages>
  • Never use -Sy to avoid partial upgrades.
• Remove package: pacman -R <packages>
  • Remove orphans too: -s
  • Purge configurations: -n

Network

• Monitor usage:
  • nload <if>
  • iftop -i <if>
  • speedometer -t <if> -r <if> [...]
  • dstat -tcyn --net-packets 60
• Monitor per-process usage:
  • nethog
• Test throughput:
  • Internet: speedtest (from speedtest.net)
  • Internal: iperf3
• Show sockets (with ss):
  • Example: ss -tulpn
  • (Note) ss replaces netstat and is mostly option compatible.
  • Option tu: Include TCP and UDP sockets (no UNIX sockets).
  • Option l: Include listening sockets (no client sockets).
  • Option p: Show protocol (requires root).
  • Option n: Don't translate port numbers to common service names.
• Show misc. stats:
  • Show kernel SNMP counters: nstat
  • Show per-protocol stats: netstat -s
• Bring interface up or down:
  • (Note) Your network manager probably has a more appropriate way to do this.
  • Directly up or down interface: ip link set dev <if> {up|down}
• Traffic shaping and link simulation:
  • See tc to simulate e.g. random packet drop, random latencies, limited bandwidth etc.

ip

• General notes:
  • Subcommands may be shortened (e.g. as ip a instead of ip address)
  • For subcommands like address and link, it will default to show all elements if no further option is specified.
• General options:
  • Specified before the subcommand (e.g. ip -c a).
  • -c: Show colored output.
  • -4 or -6: Show IPv4 or IPv6 addresses only.
  • -s: Show stats (bytes, packets, errors, dropped, etc. for RX and TX).
  • -br: Show brief with one line per interface (MAC address and status for link, addresses for address).
  • -o: Show one line per interface (but all info unlike -br).
  • -j [-p]: Print as JSON. Add -p for pretty printing.
• Show L2/MAC addresses:
  • Command: ip link [show [<interface>]]
• Show L3/IP addresses:
  • Command: ip address [show [<interface> [scope <scope>]]]
  • Argument scope global: Only show global addresses (excludes localhost/host, link-local/link, etc.).
• Show neighbors:
  • Command: ip neighbor
• Show routes:
  • Command: ip route
• Show multicast addresses:
  • Command: ip maddress
• Show multicast routes:
  • Command: ip mroute

tcpdump

• Typical usage: tcpdump -i <interface> -nn -v [filter]
• Options:
  • -w <>.pcap: Write to capture file instead of formatted to STDOUT.
  • -i <if>: Interface to listen on. Defaults to a random-ish interface.
  • -nn: Don't resolve hostnames or ports.
  • -s<n>: How much of the packets to capture. Use 0 for unlimited (full packet).
  • -v/-vv: Details to show about packets. More V's for more details.
  • -l: Line buffered more, for better stability when piping to e.g. grep.
• Filters:
  • Can consist of complex logical statements using parenthesis, not/!, and/&& and or/||. Make sure to quote the filter to avoid interference from the shell.
  • Protocol: ip, ip6, icmp, icmp6, tcp, udp, ``
  • Ports: port <n>
  • IP address: host <addr>, dst <addr>, src <addr>
  • IPv6 router solicitations and advertisements: icmp6 and (ip6[40] = 133 or ip6[40] = 134) (133 for RS and 134 for RA)
  • IPv6 neighbor solicitations and advertisements: icmp6 and (ip6[40] = 135 or ip6[40] = 136) (135 for NS and 136 for NA)
  • DHCPv4: ip and udp and (port 67 and port 68)
  • DHCPv6: ip6 and udp and (port 547 and port 546)

Memory

• NUMA stats:
  • numastat (from package numactl)

Performance and Power Efficiency

• Set the CPU frequency scaling governor mode:
  • High performance: echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
  • Power save: echo powersave | ...
• Show current core frequencies: grep "cpu MHz" /proc/cpuinfo | cut -d' ' -f3

Profiling

• time (timing commands):
  • Provided both as a shell built-in time and as /usr/bin/time, use the latter.
  • Typical usage: /usr/bin/time -p <command>
  • Options:
    • -p for POSIX output (one line per time)
    • -v for interesting system info about the process.
  • It give the wall time, time spent in usermode and time spent in kernel mode.
• strace (trace system calls and signals):
  • In standard mode, it runs the full command and traces/prints all syscalls (including arguments and return value).
  • Syntax: strace [options] <command>
  • Useful options:
    • -c: Show summary/overview only. (Hints at which syscalls are worth looking more into.)
    • -f: Trace forked child processes too.
    • -e trace=<syscalls>: Only trace the specified comma-separated list of syscalls.

Security

• Show CPU vulnerabilities: tail -n +1 /sys/devices/system/cpu/vulnerabilities/*
• Generate xkcd-style (multi-word) passwords (using package xkcdpass): xkcdpass
• Generate SHA cryptographical hashes: sha{256,512} <files>

PGP/GPG

Using GPG (from package gnupg2 on Debian).

• Use a local or temporary keyring instead of default one:
  • This is useful if you need to verify a downloaded file with the signing pubkey, but don't want to permanently import the key.
  • Create the keyring and import the key: gpg --no-default-keyring --keyring ./tmp.keyring --import <pubkey> (example)
  • Use it by specifying -no-default-keyring --keyring ./tmp.keyring in the commands where you need it.
  • Delete it and the ~-suffixed backup of it when you no longer need it.
• Inspect pubkey:
  • Imported: TODO
  • File: gpg --show-keys <keyfile>
  • Key server: TODO
• Import pubkey:
  • Unless using a keyfile you know is trusted, always verify the fingerprint of imported keys against some trusted source.
  • Import to local/temporary keyring: See section about it.
  • Import from file: gpg --import <pubkey>
  • Import from key server: gpg [--keyserver <url>] --recv-keys <key-id>
• Check signature of file using a detached signature field (typically .asc), the publisher's signing pubkey file, and temporary keyring (complete example):
  1. Download the archive, archive signature and publisher pubkey.
  2. Import the key to a local keyring: gpg --no-default-keyring --keyring ./tmp.keyring --import <keyfile>
  3. Verify the archive: gpg --no-default-keyring --keyring ./tmp.keyring --verify <sig-file> <data-file>

Storage

• Test read speed: hdparm -t <dev> (safe)
• Show IO load for devices: iostat [-dxpm] [-t] [interval]
  • -d: Show only device usage.
  • -x and -p: Include extended attributes and partitions.
  • -t and interval: Show timestamp and repeat every x seconds.
• Show IO usage for processes: iotop -o [-a]

System

• Version info:
  • Release info files:
    • Debian (and Ubuntu): /etc/debian_version
    • RHEL: /etc/redhat-release
    • CentOS: /etc/centos-release
  • General release info: uname -a
  • Slightly more distro-specific release info: lsb_release -a
• Monitor system load:
  • uptime
  • iostat [-c] [-t] [interval]
• Monitor processes:
  • ps (e.g. ps aux or ps ax o uid,user:12,pid,comm)
• Monitor a mix of things: See the "general monitoring" section.
• Monitor interrupts:
  • irqtop
  • watch -n0.1 /proc/interrupts
• Run command with high or low CPU priority:
  • Command: nice -n<n> <cmd> (-20 <= n <= 19)
  • The nice value goes from -20 (highest priority) to 19 (lowest priority), with 0 as the default priority.
  • The nice value is inherited by child processes (meaning forking processes maintains the nice value it started with).
  • Use renice to change the value.
  • Use ionice to set the I/O scheduler and scheduler-specific priority.
• Stress test with stress-ng:
  • Install (Debian): apt install stress-ng
  • Stress CPU: stress-ng -c $(nproc) -t $((10*60)) (use all CPU threads for 10 minutes)
• Chroot into other Linux installation:
  1. (Note) Used to e.g. fix a broken install or reset a user password from a USB live ISO.
  2. Mount the root partition: mount /dev/sda2 /mnt (example)
  3. Mount e.g. the EFI partition: mount /dev/sda1 /mnt/boot/efi (example)
  4. Mount system stuff:
     1. mount -t proc proc /mnt/proc
     2. mount -t sysfs sys /mnt/sys
     3. mount -t bind /dev /mnt/dev
  5. Enter jail: chroot /mnt [/bin/bash]
     • Specifying the shell to use, e.g. /bin/bash, may be necessary.
  6. Do stuff.
  7. Exit jail: exit

Tasks

Burn Windows ISO (Ubuntu)

1. Install the graphical application woeusb from ppa:nilarimogard/webupd8.

{% include footer.md %}