RSS Atom Add a new post titled:
Using a GitHub Gist like a git repo

A GitHub gist is backed by a regular git repository, but it's not exposed explicitly via the user interface.

For example, this "secret" gist can be cloned using this command:

git clone https://gist.github.com/fmarier/b652bad2e759675e8650f3d3ee81ab08.git test

Within this test directory, the normal git commands can be used:

touch empty
git add empty
git commit -a -m "Nothing to see here"

A gist can contain multiple files just like normal repositories.

In order to push to this repo, add the following pushurl:

git remote set-url --push origin git@gist.github.com:b652bad2e759675e8650f3d3ee81ab08.git

before pushing using the regular command:

git push

Note that the GitHub history UI will not show you the normal commit details such as commit message and signatures.

If you want to access the latest version of a file contained within this gist, simply access gate io.

Proper Multicast DNS Handling with NetworkManager and systemd-resolved

Using NetworkManager and systemd-resolved together in Debian bookworm does not work out of the box. The first sign of trouble was these constant messages in my logs:

avahi-daemon[pid]: Host name conflict, retrying with hostname-2

Then I realized that CUPS printer discovery didn't work: my network printer could not be found. Since this discovery now relies on Multicast DNS, it would make sense that both problems are related to an incompatibility between NetworkManager and Avahi.

What didn't work

The first attempt I made at fixing this was to look for known bugs in Avahi. Neither of the work-arounds I found worked:

What worked

The real problem turned out to be the fact that NetworkManager turns on full mDNS support in systemd-resolved which conflicts with the mDNS support in avahi-daemon.

You can see this in the output of resolvectl status:

Global
       Protocols: -LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp6s0)
    Current Scopes: DNS mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute -LLMNR +mDNS -DNSOverTLS
                    DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: lan

which includes +mDNS for the main network adapter.

I initially thought that I could just uninstall avahi-daemon and rely on the systemd-resolved mDNS stack, but it's not actually compatible with CUPS.

The solution was to tell NetworkManager to set mDNS to resolve-only mode in systemd-resolved by adding the following to /etc/NetworkManager/conf.d/mdns.conf:

[connection]
connection.mdns=1

leaving /etc/avahi/avahi-daemon.conf to the default Debian configuration.

Verifying the configuration

After rebooting, resolvectl status now shows the following:

Global
       Protocols: -LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (enp6s0)
    Current Scopes: DNS mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute -LLMNR mDNS=resolve -DNSOverTLS
                    DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
        DNS Domain: lan

Avahi finally sees my printer (called hp in the output below):

$ avahi-browse -at | grep Printer
+ enp6s0 IPv6 hp @ myprintserver   Secure Internet Printer local
+ enp6s0 IPv4 hp @ myprintserver   Secure Internet Printer local
+ enp6s0 IPv6 hp @ myprintserver   Internet Printer        local
+ enp6s0 IPv4 hp @ myprintserver   Internet Printer        local
+ enp6s0 IPv6 hp @ myprintserver   UNIX Printer            local
+ enp6s0 IPv4 hp @ myprintserver   UNIX Printer            local

and so does CUPS:

$ sudo lpinfo --include-schemes dnssd -v
network dnssd://myprintserver%20%40%20hp._ipp._tcp.local/cups?uuid=d46942a2-b730-11ee-b05c-a75251a34287

Firewall rules

Since printer discovery in CUPS relies on mDNS, another thing to double-check is that the correct ports are open on the firewall.

This is what I have in /etc/network/iptables.up.rules:

# Allow mDNS for local service discovery
-A INPUT -d 100.64.0.0/10 -p udp --dport 5353 -j ACCEPT
-A INPUT -d 192.168.1.0/24 -p udp --dport 5353 -j ACCEPT

and in etc/network/ip6tables.up.rules:

# Allow mDNS for local service discovery
-A INPUT -d ff02::/16 -p udp --dport 5353 -j ACCEPT
Filtering your own spam using SpamAssassin

I know that people rave about GMail's spam filtering, but it didn't work for me: I was seeing too many false positives. I personally prefer to see some false negatives (i.e. letting some spam through), but to reduce false positives as much as possible (and ideally have a way to tune this).

Here's the local SpamAssassin setup I have put together over many years. In addition to the parts I describe here, I also turn off greylisting on my email provider (KolabNow) because I don't want to have to wait for up to 10 minutes for a "2FA" email to go through.

This setup assumes that you download all of your emails to your local machine. I use fetchmail for this, though similar tools should work too.

Three tiers of emails

The main reason my setup works for me, despite my receiving hundreds of spam messages every day, is that I split incoming emails into three tiers via procmail:

  1. not spam: delivered to inbox
  2. likely spam: quarantined in a soft_spam/ folder
  3. definitely spam: silently deleted

I only ever have to review the likely spam tier for false positives, which is on the order of 10-30 spam emails a day. I never even see the the hundreds that are silently deleted due to a very high score.

This is implemented based on a threshold in my .procmailrc:

# Use spamassassin to check for spam
:0fw: .spamassassin.lock
| /usr/bin/spamassassin

# Throw away messages with a score of > 12.0
:0
* ^X-Spam-Level: \*\*\*\*\*\*\*\*\*\*\*\*
/dev/null

:0:
* ^X-Spam-Status: Yes
$HOME/Mail/soft_spam/

# Deliver all other messages
:0:
${DEFAULT}

I also use the following ~/.muttrc configuration to easily report false negatives/positives and examine my likely spam folder via a shortcut in mutt:

unignore X-Spam-Level
unignore X-Spam-Status

macro index S "c=soft_spam/\n" "Switch to soft_spam"

# Tell mutt about SpamAssassin headers so that I can sort by spam score
spam "X-Spam-Status: (Yes|No), (hits|score)=(-?[0-9]+\.[0-9])" "%3"
folder-hook =soft_spam 'push ol'
folder-hook =spam 'push ou'

# <Esc>d = de-register as non-spam, register as spam, move to spam folder.
macro index \ed "<enter-command>unset wait_key\n<pipe-entry>spamassassin -r\n<enter-command>set wait_key\n<save-message>=spam\n" "report the message as spam"

# <Esc>u = unregister as spam, register as non-spam, move to inbox folder.
macro index \eu "<enter-command>unset wait_key\n<pipe-entry>spamassassin -k\n<enter-command>set wait_key\n<save-message>=inbox\n" "correct the false positive (this is not spam)"

Custom SpamAssassin rules

In addition to the default ruleset that comes with SpamAssassin, I've also accrued a number of custom rules over the years.

The first set comes from the (now defunct) SpamAssassin Rules Emporium. The second set is the one that backs bugs.debian.org and lists.debian.org. Note this second one includes archived copies of some of the SARE rules and so I only use some of the rules in the common/ directory.

Finally, I wrote a few custom rules of my own based on specific kinds of emails I have seen slip through the cracks. I haven't written any of those in a long time and I suspect some of my rules are now obsolete. You may want to do your own testing before you copy these outright.

In addition to rules to match more spam, I've also written a ruleset to remove false positives in French emails coming from many of the above custom rules. I also wrote a rule to get a bonus to any email that comes with a patch:

describe FM_PATCH   Includes a patch
body FM_PATCH   /\bdiff -pruN\b/
score FM_PATCH  -1.0

since it's not very common in spam emails :)

SpamAssassin settings

When it comes to my system-wide SpamAssassin configuration in /etc/spamassassin/, I enable the following plugins:

loadplugin Mail::SpamAssassin::Plugin::AntiVirus
loadplugin Mail::SpamAssassin::Plugin::AskDNS
loadplugin Mail::SpamAssassin::Plugin::ASN
loadplugin Mail::SpamAssassin::Plugin::AutoLearnThreshold
loadplugin Mail::SpamAssassin::Plugin::Bayes
loadplugin Mail::SpamAssassin::Plugin::BodyEval
loadplugin Mail::SpamAssassin::Plugin::Check
loadplugin Mail::SpamAssassin::Plugin::DKIM
loadplugin Mail::SpamAssassin::Plugin::DNSEval
loadplugin Mail::SpamAssassin::Plugin::FreeMail
loadplugin Mail::SpamAssassin::Plugin::FromNameSpoof
loadplugin Mail::SpamAssassin::Plugin::HashBL
loadplugin Mail::SpamAssassin::Plugin::HeaderEval
loadplugin Mail::SpamAssassin::Plugin::HTMLEval
loadplugin Mail::SpamAssassin::Plugin::HTTPSMismatch
loadplugin Mail::SpamAssassin::Plugin::ImageInfo
loadplugin Mail::SpamAssassin::Plugin::MIMEEval
loadplugin Mail::SpamAssassin::Plugin::MIMEHeader
loadplugin Mail::SpamAssassin::Plugin::OLEVBMacro
loadplugin Mail::SpamAssassin::Plugin::PDFInfo
loadplugin Mail::SpamAssassin::Plugin::Phishing
loadplugin Mail::SpamAssassin::Plugin::Pyzor
loadplugin Mail::SpamAssassin::Plugin::Razor2
loadplugin Mail::SpamAssassin::Plugin::RelayEval
loadplugin Mail::SpamAssassin::Plugin::ReplaceTags
loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody
loadplugin Mail::SpamAssassin::Plugin::SpamCop
loadplugin Mail::SpamAssassin::Plugin::TextCat
loadplugin Mail::SpamAssassin::Plugin::TxRep
loadplugin Mail::SpamAssassin::Plugin::URIDetail
loadplugin Mail::SpamAssassin::Plugin::URIEval
loadplugin Mail::SpamAssassin::Plugin::VBounce
loadplugin Mail::SpamAssassin::Plugin::WelcomeListSubject
loadplugin Mail::SpamAssassin::Plugin::WLBLEval

Some of these require extra helper packages or Perl libraries to be installed. See the comments in the relevant *.pre files or use this command to install everything:

apt install spamassassin  pyzor razor libencode-detect-perl liblog-log4perl-perl libgeoip-dev libmail-dkim-perl libarchive-zip-perl libio-string-perl libmail-dmarc-perl fuzzyocr

My ~/.spamassassin/user_prefs file contains the following configuration:

required_hits   5
ok_locales en fr

# Bayes options
score BAYES_00 -4.0
score BAYES_40 -0.5
score BAYES_60 1.0
score BAYES_80 2.7
score BAYES_95 4.0
score BAYES_99 6.0
bayes_auto_learn 1
bayes_ignore_header X-Miltered
bayes_ignore_header X-MIME-Autoconverted
bayes_ignore_header X-Evolution
bayes_ignore_header X-Virus-Scanned
bayes_ignore_header X-Forwarded-For
bayes_ignore_header X-Forwarded-By
bayes_ignore_header X-Scanned-By
bayes_ignore_header X-Spam-Level
bayes_ignore_header X-Spam-Status

as well as manual score reductions due to false positives, and manual score increases to help push certain types of spam emails over the 12.0 definitely spam threshold.

Finally, I have the FuzzyOCR package installed since it has occasionally flagged some spam that other tools had missed. It is a little resource intensive though and so you may want to avoid this one if you are filtering spam for other people.

As always, feel free to leave a comment if you do something else that works well and that's not included in my setup. This is a work-in-progress.

Automatically rebooting for kernel updates

I use reboot-notifier on most of my servers to let me know when I need to reboot them for kernel updates since I want to decide exactly when those machines go down. On the other hand, my home backup server has very predictable usage patterns and so I decided to go one step further there and automate these necessary reboots.

To do that, I first installed reboot-notifier which puts the following script in /etc/kernel/postinst.d/reboot-notifier to detect when a new kernel was installed:

#!/bin/sh

if [ "$0" = "/etc/kernel/postinst.d/reboot-notifier" ]; then
    DPKG_MAINTSCRIPT_PACKAGE=linux-base
fi

echo "*** System restart required ***" > /var/run/reboot-required
echo "$DPKG_MAINTSCRIPT_PACKAGE" >> /var/run/reboot-required.pkgs

Note that unattended-upgrades puts a similar script in /etc/kernel/postinst.d/unattended-upgrades:

#!/bin/sh

case "$DPKG_MAINTSCRIPT_PACKAGE::$DPKG_MAINTSCRIPT_NAME" in
   linux-image-extra*::postrm)
      exit 0;;
esac

if [ -d /var/run ]; then
    touch /var/run/reboot-required
    if ! grep -q "^$DPKG_MAINTSCRIPT_PACKAGE$" /var/run/reboot-required.pkgs 2> /dev/null ; then
        echo "$DPKG_MAINTSCRIPT_PACKAGE" >> /var/run/reboot-required.pkgs
    fi
fi

and so you only need one of them to be installed since they both write to /var/run/reboot-required. It doesn't hurt to have both of them though.

Then I created the following cron job (/etc/cron.daily/reboot-local) to actually reboot the server:

#!/bin/bash

REBOOT_REQUIRED=/var/run/reboot-required

if [ -s $REBOOT_REQUIRED ] ; then
    cat "$REBOOT_REQUIRED" | /usr/bin/mail -s "Rebooting $HOSTNAME" root
    /bin/systemctl reboot
fi

With that in place, my server will send me an email and then automatically reboot itself.

This is a work in progress because I'd like to add some checks later on to make sure that no backup is in progress during that time (maybe by looking for active ssh connections?), but it works well enough for now. Feel free to leave a comment if you've got a smarter script you'd like to share.

Upgrading from Debian 11 bullseye to 12 bookworm

Over the last few months, I upgraded my Debian machines from bullseye to bookworm. The process was uneventful (besides the asterisk issue described below), but I ended up reconfiguring several things afterwards in order to modernize my upgraded machines.

Logcheck

I noticed in this release that the transition to journald is essentially complete. This means that rsyslog is no longer needed on most of my systems:

apt purge rsyslog

Once that was done, I was able to comment out the following lines in /etc/logcheck/logcheck.logfiles.d/syslog.logfiles:

#/var/log/syslog
#/var/log/auth.log

I did have to adjust some of my custom logcheck rules, particularly the ones that deal with kernel messages:

--- a/logcheck/ignore.d.server/local-kernel
+++ b/logcheck/ignore.d.server/local-kernel
@@ -1,1 +1,1 @@
-^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel: \[[0-9. ]+]\ IN=eno1 OUT= MAC=[0-9a-f:]+ SRC=[0-9a-f.:]+
+^\w{3} [ :[:digit:]]{11} [._[:alnum:]-]+ kernel: (\[[0-9. ]+]\ )?IN=eno1 OUT= MAC=[0-9a-f:]+ SRC=[0-9a-f.:]+

Then I moved local entries from /etc/logcheck/logcheck.logfiles to /etc/logcheck/logcheck.logfiles.d/local.logfiles (/var/log/syslog and /var/log/auth.log are enabled by default when needed) and removed some files that are no longer used:

rm /var/log/mail.err*
rm /var/log/mail.warn*
rm /var/log/mail.info*

Finally, I had to fix any unescaped | characters in my local rules. For example error == NULL || \*error == NULL must now be written as error == NULL \|\| \*error == NULL.

Networking

After the upgrade, I got a notice that the isc-dhcp-client is now deprecated and so I removed if from my system:

apt purge isc-dhcp-client

This however meant that I need to ensure that my network configuration software does not depend on the now-deprecated DHCP client.

On my laptop, I was already using NetworkManager for my main network interfaces and that has built-in DHCP support.

Migration to systemd-networkd

On my backup server, I took this opportunity to switch from ifupdown to systemd-networkd by removing ifupdown:

apt purge ifupdown
rm /etc/network/interfaces

putting the following in /etc/systemd/network/20-wired.network:

[Match]
Name=eno1

[Network]
DHCP=yes
MulticastDNS=yes

and then enabling/starting systemd-networkd:

systemctl enable systemd-networkd
systemctl start systemd-networkd

I also needed to install polkit:

apt install --no-install-recommends policykit-1

in order to allow systemd-networkd to set the hostname.

In order to start my firewall automatically as interfaces are brought up, I wrote a dispatcher script to apply my existing iptables rules.

Migration to predictacle network interface names

On my Linode server, I did the same as on the backup server, but I put the following in /etc/systemd/network/20-wired.network since it has a static IPv6 allocation:

[Match]
Name=enp0s4

[Network]
DHCP=yes
Address=2600:3c01::xxxx:xxxx:xxxx:939f/64
Gateway=fe80::1

and switched to predictable network interface names by deleting these two files:

  • /etc/systemd/network/50-virtio-kernel-names.link
  • /etc/systemd/network/99-default.link

and then changing eth0 to enp0s4 in:

  • /etc/network/iptables.up.rules
  • /etc/network/ip6tables.up.rules
  • /etc/rc.local (for OpenVPN)
  • /etc/logcheck/ignored.d.*/*

Then I regenerated all initramfs:

update-initramfs -u -k all

and rebooted the virtual machine.

Giving systemd-resolved control of /etc/resolv.conf

After reading this history of DNS resolution on Linux, I decided to modernize my resolv.conf setup and let systemd-resolved handle /etc/resolv.conf.

I installed the package:

apt install systemd-resolved

and then removed no-longer-needed packages:

apt purge openresolv resolvconf avahi-daemon

I also disabled support for Link-Local Multicast Name Resolution (LLMNR) after reading this person's reasoning by putting the following in /etc/systemd/resolved.conf.d/llmnr.conf:

[Resolve]
LLMNR=no

I verified that mDNS is enabled and LLMNR is disabled:

$ resolvectl mdns
Global: yes
Link 2 (enp0s25): yes
Link 3 (wlp3s0): yes
$ resolvectl llmnr
Global: no
Link 2 (enp0s25): no
Link 3 (wlp3s0): no

Note that if you want auto-discovery of local printers using CUPS, you need to keep avahi-daemon and ensure that systemd-resolved does not conflict with it.

DNS resolution problems with ifupdown

Also, if you haven't migrated to systemd-networkd yet and are still using ifupdown with a static IP address, you will likely run into DNS problems which can be fixed using the following patch to /etc/network/if-up.d/resolved:

@@ -43,11 +43,11 @@ if systemctl is-enabled systemd-resolved > /dev/null 2>&1; then
     fi
     if  [ -n "$NEW_DNS" ]; then
         cat <<EOF >"$mystatedir/ifupdown-${ADDRFAM}-$interface"
-"$DNS"="$NEW_DNS"
+$DNS="$NEW_DNS"
 EOF
         if  [ -n "$NEW_DOMAINS" ]; then
             cat <<EOF >>"$mystatedir/ifupdown-${ADDRFAM}-$interface"
-"$DOMAINS"="$NEW_DOMAINS"
+$DOMAINS="$NEW_DOMAINS"
 EOF
         fi
     fi
@@ -66,7 +66,7 @@ EOF
     # ignore errors due to nonexistent file
     md5sum "$mystatedir/isc-dhcp-v4-$interface" "$mystatedir/isc-dhcp-v6-$interface" "$mystatedir/ifupdown-inet-$interface" "$mystatedir/ifupdown-inet6-$interface" > "$newstate" 2> /dev/null || true
     if ! cmp --silent "$oldstate" "$newstate" 2>/dev/null; then
-        DNS DNS6 DOMAINS DOMAINS6 DEFAULT_ROUTE
+        unset DNS DNS6 DOMAINS DOMAINS6 DEFAULT_ROUTE
         # v4 first
         if [ -e "$mystatedir/isc-dhcp-v4-$interface" ]; then
             . "$mystatedir/isc-dhcp-v4-$interface"

and make sure you have nameservers setup in your static config, for example one of my servers' /etc/network/interfaces looks like this:

iface enp4s0 inet static
     address 192.168.1.2
     netmask 255.255.255.0
     gateway 192.168.1.1
     dns-nameservers 149.112.121.20
     dns-nameservers 149.112.122.20
     pre-up iptables-restore /etc/network/iptables.up.rules

Dynamic DNS

I replaced ddclient with inadyn since it doesn't work with no-ip.com anymore, using the configuration I described in an old blog post.

chkrootkit

I moved my customizations in /etc/chkrootkit.conf to /etc/chkrootkit/chkrootkit.conf after seeing this message in my logs:

WARNING: /etc/chkrootkit.conf is deprecated. Please put your settings in /etc/chkrootkit/chkrootkit.conf instead: /etc/chkrootkit.conf will be ignored in a future release and should be deleted.

ssh

As mentioned in Debian bug#1018106, to silence the following warnings:

sshd[6283]: pam_env(sshd:session): deprecated reading of user environment enabled

I changed the following in /etc/pam.d/sshd:

--- a/pam.d/sshd
+++ b/pam.d/sshd
@@ -44,7 +44,7 @@ session    required     pam_limits.so
 session    required     pam_env.so # [1]
 # In Debian 4.0 (etch), locale-related environment variables were moved to
 # /etc/default/locale, so read that as well.
-session    required     pam_env.so user_readenv=1 envfile=/etc/default/locale
+session    required     pam_env.so envfile=/etc/default/locale

 # SELinux needs to intervene at login time to ensure that the process starts
 # in the proper default security context.  Only sessions which are intended

I also made the following changes to /etc/ssh/sshd_config.d/local.conf based on the advice of ssh-audit 2.9.0:

-KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256
+KexAlgorithms curve25519-sha256@libssh.org,curve25519-sha256,sntrup761x25519-sha512@openssh.com,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512

Unwanted power management

I ran into a problem with one of my servers where it would suspend itself after a certain amount of time. This was due to default GDM behaviour it turns out and while I could tell gdm not to sleep on inactivity, I instead put the following in /etc/systemd/sleep.conf.d/nosuspend.conf to fully disable systemd-based suspend or hibernate:

[Sleep]
AllowSuspend=no
AllowHibernation=no
AllowSuspendThenHibernate=no
AllowHybridSleep=no

Asterisk has been removed from Debian

The only major problem I ran into while upgrading to bookworm is that I discovered that Asterisk has been removed from stable and testing. For some reason, this was not mentioned in the release notes and I have not yet found a good solution.

If you upgrade to bookworm, be warned that the bullseye packages will remain installed (and will work fine in my experience) unless you "clean them up" with apt purge '~o' accidentally and then you'll have to fetch these old debs manually.

Monitoring browser network traffic on Android using mitmproxy

Using mitmproxy to intercept your packets is a convenient way to inspect a browser's network traffic.

It's pretty straightforward to setup on a desktop computer:

  1. Install mitmproxy (apt install mitmproxy on Debian) and start it:

     mitmproxy --mode socks5 --listen-port 9000

  2. Start your browser specifying the proxy to use:

     chrome --proxy-server="socks5://localhost:9000"
 brave-browser --proxy-server="socks5://localhost:9000"

  3. Add its certificate authority to your browser.

At this point, all of the traffic from that browser should be flowing through your mitmproxy instance.

Android setup

On Android, it's a little less straightforward:

  1. Start mitmproxy on your desktop:

     mitmproxy --mode regular --listen-port 9000

  2. Open that port on your desktop firewall if needed.

  3. On your Android device, change your WiFi settings for the current access point:
  4. Proxy: Manual
  5. Proxy hostname: 192.168.1.100 (IP address of your desktop)
  6. Proxy port: 9000
  7. Turn off any VPN.
  8. Turn off WiFi.
  9. Turn WiFi back on.
  10. Open http://mitm.it in a browser to download the certificate authority file.
  11. Open the system Settings, Security and privacy, More security and privacy, Encryption & credentials, Install a certificate and finally choose CA certificate.
  12. Tap Install anyway to dismiss the warning and select the file you just downloaded.

Once you have gone through all of these steps, you should be able to monitor (on your desktop) the HTTP and HTTPS requests made inside of your Android browsers.

Note that many applications will start failing due to certificate pinning.

Enabling AppArmor on a Linode VPS in enforcement mode

Enabling AppArmor on a Debian Linode VPS is not entirely straightforward. Here's what I had to do in order to make it work.

Packages to install

The easy bit was to install a few packages:

apt install grub2 apparmor-profiles-extra apparmor-profiles apparmor

and then adding apparmor=1 security=apparmor to the kernel command line (GRUB_CMDLINE_LINUX) in /etc/default/grub.

Move away from using Linode's kernels

As mentioned in this blog post, I found out that these parameters are ignored by the Linode kernels.

I had to:

  1. login to the Linode Manager (i.e. https://cloud.linode.com/linodes/<linode ID>/configurations),
  2. click the node relevant node,
  3. click "Edit" next to the configuration profile, and
  4. change the kernel to "GRUB 2".

Fix grub

Next I found out that grub doesn't actually install itself properly because it can't be installed directly on the virtual drives provided by Linode (KVM). Manually running this hack worked for me:

grub-install --grub-setup=/bin/true /dev/null

Unbound + Let's Encrypt fix

Finally, my local Unbound installation stopped working because it couldn't access the Let's Encrypt certificates anymore.

The solution to this was pretty straightforward. All I needed to do was to add the following to /etc/apparmor.d/local/usr.sbin.unbound:

/etc/letsencrypt/archive/** r,
/etc/letsencrypt/live/** r,
Things I do after uploading a new package to Debian

There are a couple of things I tend to do after packaging a piece of software for Debian, filing an Intent To Package bug and uploading the package. This is both a checklist for me and (hopefully) a way to inspire other maintainers to go beyond the basic package maintainer duties as documented in the Debian Developer's Reference.

If I've missed anything, please leave an comment or send me an email!

Salsa for collaborative development

To foster collaboration and allow others to contribute to the packaging, I upload my package to a new subproject on Salsa. By doing this, I enable other Debian contributors to make improvements and propose changes via merge requests.

I also like to upload the project logo in the settings page (i.e. https://salsa.debian.org/debian/packagename/edit) since that will show up on some dashboards like the Package overview.

Launchpad for interacting with downstream Ubuntu users

While Debian is my primary focus, I also want to keep an eye on how my package is doing on derivative distributions like Ubuntu. To do this, I subscribe to bugs related to my package on Launchpad. Ubuntu bugs are rarely Ubuntu-specific and so I will often fix them in Debian.

I also set myself as the answer contact on Launchpad Answers since these questions are often the sign of a Debian or a lack of documentation.

I don't generally bother to fix bugs on Ubuntu directly though since I've not had much luck with packages in universe lately. I'd rather not spend much time preparing a package that's not going to end up being released to users as part of a Stable Release Update. On the other hand, I have succesfully requested simple Debian syncs when an important update was uploaded after the Debian Import Freeze.

Screenshots and tags

I take screenshots of my package and upload them on https://screenshots.debian.net to help users understand what my package offers and how it looks. I believe that these screenshots end up in software "stores" type of applications.

Similarly, I add tags to my package using https://debtags.debian.org. I'm not entirely sure where these tags are used, but they are visible from apt show packagename.

Monitoring Upstream Releases

Staying up-to-date with upstream releases is one of the most important duties of a software packager. There are a lot of different ways that upstream software authors publicize their new releases. Here are some of the things I do to monitor these releases:

  • I have a cronjob which run uscan once a day to check for new upstream releases using the information specified in my debian/watch files:

      0 12 * * 1-5   francois  test -e /home/francois/devel/deb && HTTPS_PROXY= https_proxy= uscan --report /home/francois/devel/deb || true

  • I subscribe to the upstream project's releases RSS feed, if available. For example, I subscribe to the GitHub tags feed for git-secrets and Launchpad announcements for email-reminder.

  • If the upstream project maintains an announcement mailing list, I subscribe to it (e.g. rkhunter-announce or tor release announcements).

When nothing else is available, I write a cronjob that downloads the upstream changelog once a day and commits it to a local git repo:

#!/bin/bash
pushd /home/francois/devel/zlib-changelog > /dev/null
wget --quiet -O ChangeLog.txt https://zlib.net/ChangeLog.txt || exit 1
git diff
git commit -a -m "Updated changelog" > /dev/null
popd > /dev/null

This sends me a diff by email when a new release is added (and no emails otherwise).

Using iptables with systemd-networkd

I used to rely on ifupdown to bring up my iptables firewall automatically using a config like this in /etc/network/interfaces:

allow-hotplug eno1
iface eno1 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules

iface eno1 inet6 dhcp
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

but I wanted to modernize my network configuration and make use of systemd-networkd after upgrading one of my servers to Debian bookworm.

Since I already wrote an iptables dispatcher script for NetworkManager, I decided to follow the same approach for systemd-networkd.

I started by installing networkd-dispatcher:

apt install networkd-dispatcher moreutils

and then adding a script for the routable state in /etc/networkd-dispatcher/routable.d/iptables:

#!/bin/sh

LOGFILE=/var/log/iptables.log

if [ "$IFACE" = lo ]; then
    echo "$0: ignoring $IFACE for \`$STATE'" | ts >> $LOGFILE
    exit 0
fi

case "$STATE" in
    routable)
        echo "$0: restoring iptables rules for $IFACE" | ts >> $LOGFILE
        /sbin/iptables-restore /etc/network/iptables.up.rules 2>&1 | ts >> $LOGFILE
        /sbin/ip6tables-restore /etc/network/ip6tables.up.rules 2>&1 | ts >> $LOGFILE
        ;;
    *)
        echo "$0: nothing to do with $IFACE for \`$STATE'" | ts >> $LOGFILE
        ;;
esac

before finally making that script executable (otherwise it won't run):

chmod a+x /etc/NetworkManager/dispatcher.d/pre-up.d/iptables

With this in place, I can put my iptables rules in the usual place (/etc/network/iptables.up.rules and /etc/network/ip6tables.up.rules) and use the handy iptables-apply and ip6tables-apply commands to test any changes to my firewall rules.

Looking at /var/log/iptables.log confirms that it is being called correctly for each network interface as they are started.

Finally, create a new /etc/logrotate.d/iptables-local file to ensure that the log file does not grow unbounded:

/var/log/iptables.log {
        monthly
        rotate 1
        nocreate
        nomail
        noolddir
        notifempty
        missingok
}
Upgrading from Ubuntu 20.04 focal to 22.04 jammy

A few weeks ago, I upgraded a few machines from Ubuntu 20.04 (focal) to 22.04 (jammy). Here are the things that needed fixing after the upgrade.

Network problems

Firstly, I had to fix the resolution of .local domains the same way as I did when I upgraded a different machine from 18.04 (bionic) to 20.04 (focal).

ssh agent problems

Then, I found that ssh-add no longer worked and instead returned this error:

Could not open connection to your authentication agent

While this appears to be a known issue, the work-around suggested in the i3 forum didn't work for me. What did work was the solution described in this blog post:

  1. Add this to my ~/.bash_profile:

    eval $(systemctl --user show-environment | grep SSH_AUTH_SOCK)
export SSH_AUTH_SOCK

  2. Add this to my startup script:

    /usr/bin/systemctl --user start ssh-agent.service

I'm not sure why ED25519 keys don't work in gnome-keyring since that bug was supposedly fixed a while back, but starting gnome-keyring-ssh.service instead of ssh-agent.service didn't work for me.

Packages

When it comes to specific packages, I removed this obsolete package:

  • popularity-contest

I also installed these two new packages:

As always, I put any packages I backport from Debian unstable into my PPA. So far with jammy, I only had to update tiger to silence some bogus warnings.