System Administration
Base System
SSH
Server Side
- Config files are located at /etc/ssh/sshdconfig
A few key items to look at:
- PermitRootLogin yes|no
- PasswordAuthentication yes|no
- Client Side
- $ ssh-keygen
- Will produce a key on the user's machine
- $ ssh-copy-id
- Will copy the public key of the ssh keypair to the remote machine. Requires password login to be enabled on the remote machine.
Firewalls
Several avenues:
- ufw (a front end for iptables):
- A basic ufw set up:
- # ufw default deny
- # ufw allow ssh
- # ufw allow nginx-full
- # ufw enable
- A basic ufw set up:
- iptables: This is the old school method. Still works great, not a bad option.
- nftables (replaces iptables): A newer implementation, became standard in several distrobutions (Debian Buster and RHEL 7 circa 2017)
- firewalld (a frontend for iptables): The systemd solution
UFW
Network
Find other computers
Use nmap
to scan the network for all devices on it.
nmap -sn 192.168.8.0/24
Wifi:
- Scanning:
- iwconfig : list network interfaces
- ip link set interface up : turn on interface
- iwlist interface scan | less : scan available networks
- ip link set interface down
- Accessing:
- wifi-menu
Uncomplicated Firewall makes setting up a firewall easy. To make a profile for an application simply place a file in `/etc/ufw/applications.d` with the following format: ``` [appname] title=1-liner here description=A longer line ports=80,443,8080/tcp|10000/udp|53 ```
User Management
- Adding users
- `adduser user`
- Creates user 'user'
- Creates usergroup and home directory according to /etc/adduser.conf
- Populates the user's home directory with the contents of /etc/skel
This will create the user "user" and prompt for password and other info, including creating a standard home dir for the user.
- Granting Sudo
- usermod -aG sudo user Give the user "user" sudo privileges
- $ cp -r .ssh home/user && chown -R user:user home/user.ssh Useful for copying the public key the root is using to log in to a user they have elevated to sudo priveleges and wish to use going forward.
- Changing users
- `su - user` log in as user
- `sudo su` elevate to root
- `sudo su -` elevate to root and obtain root's environment
- `sudo su - user` change to user and the environment you'd expect for them
- `su - user -c "commands to execute"` execute commands as given user
- Changing login shell: usermod –shell /bin/shell user
Distribution Specific
Arch
- General Dependencies:
- base-devel sudo git man-db, wget
- Autologin into virtual console:
- Relies on `drop-in` files in systemd
- systemctl edit getty@tty1:
- [Service]
- ExecStart=
- ExecStart=-/usr/bin/agetty –autologin username –noclear %I $TERM
Debian
when sources have changed their version (e.g. from stable to oldstable when a new release is out) you need to confirm the change for apt: apt-get update –allow-releaseinfo-change
Init Systems
Systemd
- systemctl
- The main command to use to manipulate systemd services.
- options to use with it are
- status
- start
- stop
- enable
- disable
- journalctl
- the standard way to view the output of services managed by systemd. If things are working check here.
- -u phrase
- will filter entries in journalctl according to phrase
- Hostname
- hostnamectl
- `hostnamectl set-hostname new-hostname`
- edit `/etc/hosts` and change the entry to the new hostname.
- example systemd application.service file
- located in usr/lib/systemd/system
[Unit] Description=Describe me [Service] ExecStart=absolute/path/to/bin [Install] WantedBy=multi-user.target
Root Level
Hardware configuration
GRUB
- config: /etc/default/grub
- update config `grub2-mkconfig -o /boot/grub2/grub.cfg`
- quick n dirty rescue mode: highlight kernel, press e to edit, and change everything after the `root=<uuid>` to `single`
LVG managements
- list block devices lsblk
- list volume groups vgs
- list all logical volumes lvs
- extend logical volume lvextend -L -2G /path/to/volume
- expand the filesystem to fill newly available space resize2fs /path/to/volume
- create new physical volumes on partitions so that you can add them to a lvg pvcreate /dev/partitions
- create a new logical volume group vgcreate NAME /dev/partition /dev/partition
- create new logical volume lvcreate -L 2G VolumeGroup –name nameit
Block Devices
- lsblk | blkid (this one shows UUID)
- Linux automounts devices according to /etc/fstab (this is a critical file! backup first!)
- man fstab for how to construct the fstab file
- Back up a MBR: dd if=/dev/sda of=/tmp/sdaMBR.bak bs=512 count=1
- fdisk to partition drives
- mkfs to format them mkfs.<tab> to view all options put the type of filesystem you want after the '.'
Bad sectors fix I have a different process for this that replaced the bad superblock with one of the alternatives. FSCK can be a "lossy" process because FSCK may want to remove too much data or to remove data from a sensitive location (e.g. data directory for a data base) so there are times when I don't want to use it or it doesn't work.
You can sudo yourself silly or just become root for the process. Just remember that when you are root, Linux assumes that you know what you're doing when you issue commands. If so directed, it will speedily delivery Mr. Bullet to Mr. Foot. Like many other things, with great power comes great responsibility. That concludes my warning on running your system as root.
sudo -s fdisk -l
Figure out which device - assuming /dev/sdc1 for this example along with EXT4 as its the most common for this explanation.
fsck -N /dev/sdc1
Your device and your file system (ZFS, UFS, XFS, etc.) may vary so know what you have first. Do not assume it's EXT4. Ignoring this step can cause you problems later if it's NOT an EXT4 file system.
fsck.ext4 -v /dev/sdc1
Get your error message which says the superblock is bad. You don't want to do this if your superblock is OK.
mke2fs -n /dev/sdc1
This will output the alternate superblocks stored on your partition.
Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208
Pick an alternate superblock - keep in mind that the first one is the default and its bad so let's not use that one. You will also want to pick one from the list you get from your partition. Do not use the example. Your superblocks may be stored elsewhere.
e2fsck -b 98304 /dev/sdc1
Reboot and see if this worked. If not try the next superblock on the list. I've had to go the third or fourth one a couple of times.
e2fsck -b 163840 /dev/sdc1
Now try the command to validate the disk again. See if you get the same messabout about bad superblocks.
fsck.ext4 -v /dev/sdc1
Keep trying until you either run out of superblocks or it works. If you run out, you likely have bigger issues and I hope you have good backups. You can try running FSCK at that point.
Encrypted partitions
To open:
cryptsetup luksOpen /dev/sdb1 name
Swap
- Steps to make a swap file: dd if=/dev/zero of=/swapfile bs=32M count=128 # Use dd and /dev/zero for safe creation of ram, this is 4G mkswap /swapfile chmod 600 /swapfile # RAM needs restricted permissions for safety swapon /swapfile # all done! New swap is now active. Note this is not persistent. Add it to fstab if you wish it so.
Hard Drive Health:
- The smartctl command can give lots of data about hard drive status.
- These five attributes have been noted by large cloud providers as indicative of failure
- SMART 5 ReallocatedSectorCount
- SMART 187 ReportedUncorrectableErrorsCount
- SMART 188 ComandTimeout
- SMART 197 CurrentPendingSectorCount
- SMART 198 OfflineUncorrectable
System Monitoring
- top
- htop
- atop
- iotop
- free
- iostat
- sensors
- hddtemp
- sar
LetsEncrypt Cets
If you need to move certs from one server to another:
- Copy the certs from `/etc/letsencrypt/archive` and `/etc/letsencrypt/renewal`
- Symlink from `archive` to `live` appropriately.
Nginx
- Block to serve publichtml dir in the user directory ``` location ~ ^/(.+?)(/.*)?$ { alias /home/$1/www$2; index index.html index.htm; autoindex on; } ```
- Serve files not in the root directory for specific URI ``` location /URI/to/match { root /path/to/files } ``` This is useful for linking to static files that you don't want to host in the root directory for whatever reason.
- Test nginx config for any errors:
- `nginx -c /etc/nginx/nginx.conf -t`
Jitsi
- Multiple Domains
It is possible to make one jitsi daemon accesible from multiple domains. Two settings need to be altered.
- Make a proper configuration for the new domain in nginx. This really just means copying the one generated by the jitsi install and altering the server entries to the new domain.
- Then change the bosh entry in /etc/jitsi/meet/domain-config.js to `bosh: '//<!–# echo var="httphost" –>/<!–# echo var="subdir" default="" –>http-bind'`
Postgres
Maintinance
these commands are run in the psql shell
- `\c database` connect to database
- `vacuum (verbose|analyze|full)` verbose can be used with any option. vacuum returns the space occupied by outdated or empty table entries to the database for resuse. full locks the database and write the whole shebang to a new table in order to return the space to the OS. omitting full will run the automatic mode, which runs on a live database, freeing the space for reuse but not returning it to the OS.
- psql; \list: view a list of all databases
- pgdump -U user -F t database > backup.tar : dump the database for backup. Can be done while running.
Creation of a database
Change the user to postgres :
su - postgres
Create User for Postgres (in the shell and NOT with psql)
$ createuser testuser
Create Database (same)
$ createdb testdb
Acces the postgres Shell
psql ( enter the password for postgressql)
Provide the privileges to the postgres user
$ alter user testuser with encrypted password 'qwerty'; $ grant all privileges on database testdb to testuser;
Jitsi
Change ram limits
for jitsi-videobridge: Alter the appropriate values in /usr/share/jitsi-videobridge/jvb.sh for jicofo: add `JICOFOMAXMEMORY` variable to /etc/jitsi/jicofo/config
Quick Self Hosting
- Prerequisite packages
gnupg2 nginx-full apt-transport-https
- Add Jitsi Repo (Debian)
curl https://download.jitsi.org/jitsi-key.gpg.key | sudo sh -c 'gpg –dearmor > usr/share/keyrings/jitsi-keyring.gpg' echo 'deb [signed-by=/usr/share/keyrings/jitsi-keyring.gpg] https://download.jitsi.org stable' | sudo tee /etc/apt/sources.list.d/jitsi-stable.list > /dev/null
Certbot
Useful commands:
- certbot certificates Lists all the certs that certbot is managing on the system
- certbot delete –cert-name CERTNAME Cleans up files for a cert, does not revoke
- certbot revoke –delete-after-revoke –cert-name CERT
User Level
Fonts
Fonts are defined on a user level in `~/.config/fontconfig/fonts.conf`. You can see there what fonts the system will attempt to load when particular font families get called.
$ display ~/Comic-mono.ttf Use imagemagick tool display to view fonts
Default applications
Use xdg-mime to manipulate when not using a DE.
- To see what the filetype for a file is: xdg-mime query filetype FILE
- To see the default application for a type is: xdg-mime query default application/type
- To change the default:
- First find the .desktop file to use. Peak in usr/share/applications and grep around for the one you want.
- then run this command: xdg-mime default application.desktop application/type
Default Applications
- Set default applications via xdg utilities. DE's do this with their guis.
- Check the application type for a given filetype: xdg-mime query filetype path/to/some/file
- Set the default application: xdg-mime default application.desktop application/filetype
GPG keys
Export to another machine: gpg –export-secret-key SOMEKEYID | ssh othermachine gpg –import
Set trust level: gpg –edit-key [key-id] enter command "trust"
Make sure you have an appropriate pinentry program installed.
the GPGTTY variable might also need to be set
Dot files
Chezmoi Templates remove trailing newline from shell command: {{ (output "shell.sh") | trimAll "\n" }}
Initial Setup
Software To Install On Every Server
- mosh
- man-db
- apache/nginx
Things to do
$ cp -r .ssh home/user && chown -R user:user home/user.ssh Useful for copying the public key the root is using to log in to a user they have elevated to sudo priveleges and wish to use going forward.
https://www.digitalocean.com/community/tutorials/how-to-install-linux-nginx-mariadb-php-lemp-stack-on-debian-10 https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-debian-10 https://www.digitalocean.com/community/tutorials/how-to-install-wordpress-with-lemp-nginx-mariadb-and-php-on-debian-10
Wireguard
Option 2: use the WireGuard app
- Install WireGuard
sudo apt-get update && sudo apt-get install curl jq openresolv wireguard
For non-Debian based distributions, follow WireGuard's official installation instructions.
- Run our configuration script
curl -LO https://mullvad.net/media/files/mullvad-wg.sh && chmod +x ./mullvad-wg.sh && ./mullvad-wg.sh
If you're running WireGuard on multiple devices, generate a separate key pair for each device. You will otherwise likely run into connectivity issues.
- Turn on WireGuard
wg-quick up mullvad-se4
You may replace "se4" with any of the other regions found on our server page.
Note: Make sure that the configuration file name is no more than 15 characters long (not including ".conf"). Disconnect
wg-quick down mullvad-se4
As before, you may replace "se4" with the currently used region. Verify your connection
To verify that WireGuard is working, use our Connection check to check your IP.
Multihop with WireGuard
Multihop can be used for many different reasons, for example, increasing your privacy or improving latency/performance due to suboptimal ISP peering.
Each WireGuard server is connected to all the other WireGuard servers through WireGuard tunnels. This means you can multihop from one server to another. One way to do this is to connect to a specific port on a WireGuard server which will then connect to the other WireGuard server via the tunnel.
In the WireGuard configuration file the Endpoint port defines the exit server, and the preceding IP-address or hostname defines the entry server: "INSERVER-IP:OUTSERVER-PORT". The PublicKey is that of the exit server.
For example, let's say you want to connect to nl1 via se4 (so nl1 is the exit server). To do this, you would connect to se4-wireguard.mullvad.net:3004 and use the public key of the nl1 server. The port 3004 is the multihop port for nl1.
Choose a multihop port from our server page. Then add a new multihop configuration file by modifying an existing one. In the following example you start with an existing configuration file for nl1 so you have the public key for that. Then you change the Endpoint to connect to se4-wireguard.mullvad.net (or 193.138.218.130) and jump to nl1 via it's multihop port 3004. The configuration is saved as a new configuration file.
sudo sh -c "umask 077; sed 's/Endpoint.*/Endpoint = se4-wireguard.mullvad.net:3004/' /etc/wireguard/mullvad-nl1.conf > /etc/wireguard/wireguard-se4nl1.conf" Multihop via SOCKS5 proxies
Our SOCKS5 proxy guide includes steps for configuring your browser or other programs to multihop using our WireGuard SOCKS5 proxies. Troubleshooting
If you run into any issues while testing WireGuard, please contact us at support@mullvad.net and let us know what you experience.
Due to a Debian bug, Debian/Ubuntu users may want to install openresolv rather than Debian's broken resolvconf, in order to prevent DNS leaks. DNS leaking Ubuntu 18.04 or newer (or other systems that use systemd-resolved)
In the WireGuard configuration file replace the 'DNS = ' line with : PostUp = systemd-resolve -i %i –set-dns=193.138.218.74 –set-domain=~. FAQ How do I enable a kill switch?
Add the following lines under the [Interface] section of the WireGuard configuration files found in etc/wireguard:
PostUp = iptables -I OUTPUT ! -o %i -m mark ! –mark $(wg show %i fwmark) -m addrtype ! –dst-type LOCAL -j REJECT && ip6tables -I OUTPUT ! -o %i -m mark ! –mark $(wg show %i fwmark) -m addrtype ! –dst-type LOCAL -j REJECT PreDown = iptables -D OUTPUT ! -o %i -m mark ! –mark $(wg show %i fwmark) -m addrtype ! –dst-type LOCAL -j REJECT && ip6tables -D OUTPUT ! -o %i -m mark ! –mark $(wg show %i fwmark) -m addrtype ! –dst-type LOCAL -j REJECT
Issue man wg-quick for more information. How do I make WireGuard start automatically on boot?
Run the following command, replacing mullvad-se4 with the WireGuard server you wish to use.
systemctl enable wg-quick@mullvad-se4 How do I enable port forwarding?
Log in with your account on our website and then add the ports from your account page, you can move the ports to different pubkeys. External resources
WireGuard homepage WireGuard Whitepaper (PDF) Installation Instructions Quickstart Instructions Donate to Upstream WireGuard Development Formal Verification of WireGuard Protocol wg(8) man page wg-quick(8) man page
"WireGuard" is a registered trademark of Jason A. Donenfeld.
KDE Desktop
Fix an issue on arch linux where kde connect can't reach the phone bc of a firewall.
sudo firewall-cmd –permanent –zone=public –add-service=kdeconnect sudo firewall-cmd –reload
CUPS (Printing)
lpoptions -d Printer-to-be-default localhost:631 : The web interface to CUPS print with `lp` command