Category: sysop

lnxsense, a system monitoring tool for Linux

PhillipLeave a comment

Ever since I got my AMD Athlon XP 2500+, I’ve been into overclocking. While my overclocking activities were limited at the time (as a student I couldn’t risk burning up my CPU or motherboard), I made sure that ever since, none of my desktops ran at stock speeds. Even my trusty Intel 2500K that I’m writing this blog on still hums along at 4,4Ghz all core.

Overclocking has always been a Windows thing though, and for good reason; in 2009 the Linux market share was only 0,6%, while Windows dominated the market with a 95% market share. With such a dominating OS, motherboards manufacturers focused fully on (usually terrible) software which allowed you to overclock and monitor your system without leaving Windows. The overclocking community didn’t stop there either, tools like 8rdavcore (apparently ported from Linux), setfsb, MemSet, CPU-Tweaker and many more made it possible to overclock and tweak your system to the max. Combined with a lot of monitoring software like HWInfo, Aida64, SpeedFan, CPU-z and benchmarks like 3Dmark, Sisoft Sandra, Cinebench, and it was clear: overclocking belonged to Windows.

Fast forward to 2025, and things have changed; Linux has a market share of 3% while Windows has dropped to 66%. OCCT is now also available on Linux, GreenWithEnvy makes it easier to overclock NVIDIA gpu’s and benchmarks like y-cruncher, 7-zip and Geekbench run fine on Linux. But when it comes to graphical monitoring applications, we only have Psensor or xsensors. Both work fine but it can still be better.

A screenshot showing xsensors and psensor side by side
Xsensors and PSensor side by side

This is where I want to change a couple of things and after this years release of Java 25 and its Foreign Function and Memory API, I can finally work in a language I love while using C libraries like libsensors, libcpuid, the NVIDIA management API and many more.

After returning from Devoxx I decided to create a Linux alternative to Open Hardware Monitor, HWMonitor and HWInfo and that’s how lnxsense was born. It’s a still in early alpha stages and what it can show depends heavily on what the underlying libraries can return (e.g. NVIDIA’s nvml doesn’t even have an option to get the hotspot temperature or actual fan RPM). Even so, I’m already really happy with what it can do.

lnxsense showing different metrics like cpu usage, power draw, GPU frequency.

In it’s very early stage it supports (when running the back-end server as root)

  • CPU Frequencies (as reported by the Linux kernel)
  • CPU Utilization
  • Memory Utilization
  • Core temperatures
  • Intel requested VCore (the VID)
  • Intel Core multipliers
  • Intel Throttling reasons
  • Intel RAPL Power Management information like PP0, PP1 and Platform power limits and usage
  • NVIDIA Clocks, Utilization, Temperature and Fan speed (in % because why would nvml expose the actual fan speed), P-state and current PCIe speed
  • SMART and NVMe log
  • Blockdevice IOPS and read/write speed
  • Remote monitoring using sockets

If you want to try it out, you can download a release version from Codeberg. Just be sure to read the INSTALL.md, it’s still in early development, so it’s not a one-click experience and definitely not production-ready.

// 2025/12/15: I decided to rename the project from HWJinfo to lnxsense, it just makes more sense, doesn’t it ?

Ansible: VARIABLE IS NOT DEFINED!

PhillipLeave a comment

So, I let my certificates expire (again) and thus I had to re-run all my Ansible playbooks to roll out my new self-signed certificates on all my severs and the reality was that a lot of my playbooks didn’t run or didn’t survive the galaxy update I ran a couple of weeks before this happened.

The weirdest thing of all was that the Postgres role that I use failed on an assert for a variable that 100% exists and which has worked before.

TASK [robertdebock.postgres : assert | Test postgres_hba_entries] *************************************************************************************************************************************************************************************************************
fatal: [postgresql-01]: FAILED! => changed=false 
  assertion: postgres_hba_entries is defined
  evaluated_to: false
  msg: Assertion failed

Running an ansible.builtin.debug in a pre-task did confirm that the variable “did not exist”

  pre_tasks:
    - name: Debug
      ansible.builtin.debug:
        var: postgres_hba_entries

TASK [Debug] ******************************************************************************************************************************************************************************************************************************************************************
ok: [postgresql-01] => 
  postgres_hba_entries: VARIABLE IS NOT DEFINED!

Even with the verbosity set to 6 there was no sign of anything being wrong. While debugging other variables, I noticed the same behavior when trying to output the value of postgres_listen_addresses: "{{ hostvars[inventory_hostname]['ansible_default_ipv4']['address'] }}" while gathering facts was disabled.

As it turns out, if you’re using an unknown variable to create another variable, then it will simply not exist, even if you’re using it in a map like postgres_hba_entries. So in the below example, the non existing DOES_NOT_EXIST variable will result in the complete map missing from the environment.

postgres_hba_entries:
  - type: local
    database: all
    user: all
    method: peer
  - type: host
    database: all
    user: all
    address: 127.0.0.1/32
    method: ident
  - type: hostssl
    address: all
    database: "{{ DOES_NOT_EXIST }}"
    method: md5
    user: all

This is on some ways pretty good, because you don’t want to roll out only half of your config without knowing it. On the other hand it’s pretty annoying there’s absolutely no feedback about what is going on (even though I can come up with many reasons why it is so).

Using a read-only SMB share in a root-less Immich setup with SELinux enabled

PhillipLeave a comment

I recently wanted to switch from Google Photos to Immich and while doing so I stumbled across some difficulties while adding the photo’s on my NAS as an external library. In the past 20+ years I organized my library by hand without relying on any tools, so I did not want Immich to make any changes to my photo library, hence I mounted the Samba share as read-only.

//nas.internal/photo /mnt/photo cifs credentials=/root/samba.cred,ro,nodev,noexec,nosuid,gid=0,dir_mode=0777,file_mode=0444 0 0

If I try to add a folder from this share as an external library I get the following error: “Lacking read permissions for folder”

Disabling SELinux would fix the issue, but even if the instance is not publicly available, it’s still a bad idea to disable any security measures. So we need to tell SELinux it’s fine for the container to access the share. Usually this is done by appending :z to the volume:

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.ml.yml
    #   service: vaapi # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - /var/immich/media/:/usr/src/app/upload:z
      - /etc/localtime:/etc/localtime:ro
      - /mnt/photo:/usr/src/app/external:z
    env_file:
      - immich.env
    ports:
      - '2283:2283'
    depends_on:
      - database
    restart: always
    healthcheck:
      disable: false

But simply adding “:z” in the Docker compose file won’t work for two reasons:

  1. The user does not have any root privileges to change the SELinux context
  2. The filesystem is mounted read-only and changing the context is a write operation

Luckily, we can mount the SMB share with an SELinux content which will allow the container to access the files:

//nas.internal/photo /mnt/photo cifs credentials=/root/samba.cred,ro,nodev,noexec,nosuid,gid=0,dir_mode=0777,file_mode=0444,context="system_u:object_r:container_file_t:s0" 0 0

To apply the changes we need to unmount/remount the share

# stop the immich containers first
podman compose down
# Remount (mount -o remount won't work, remount can't change permissions)
sudo umount /mnt/photo
sudo mount /mnt/photo

If we now restart Immich and add the Samba share we can see that it can access the files:

The mysterious traffic on my containers

PhillipLeave a comment

While perusing my dashboards in Grafana I noticed something odd; repetitive traffic on all of my Proxmox container instances, most of them shouldn’t even be seeing any traffic at all:

Screenshot of the LXC traffic in on 3 different containers showing almost exactly the same spikes.

Trying to find the source of these spikes, I started to shutdown container by container until the point where I had to shutdown my Home Assistant VM, hoping that it wouldn’t be the cause of the suspicious traffic. It was.

Suspect myself

Okay, so something in Home Assistant was sending data to all my containers and the prime suspect was the nmap tracker that I use to track whether my computers are on so that I can toggle WLED strip behind my desk. Disabling the integration confirmed my suspicion since the spikes disappeared again. Weirdly enough, the nmap tracker was not configured to scan the range of my Proxmox LXC containers, it only had to scan two /24 subnets of my /16 subnet.

With the integration disabled I tried running the same nmap command from my desktop to make sure that it’s not some weird Proxmox metrics bug since I am the one sending out suspicious traffic from within a Proxmox VM. A couple of seconds later, a new spike arose, telling me that it’s probably not a bug.

After going through the nmap documentation and trying out different settings, the only thing I could change was how big the spikes were, but they wouldn’t go away no matter what option I chose.

In the first green box, the spikes no longer appear after disabling the nmap tracker. The second green box was a scan from my local machine, the third green box was a much bigger, more aggressive scan from my local machine.

Nothing is gratuitous

Being curious as ever, there was only one more thing I could do: run pcap on a fresh LXC container, capture the data and check it with Wireshark (though after a night pondering about it I already had a n idea about that cause).

After opening the pcap file in Wireshark it was clear as day that trying to reach a bunch of IP’s that don’t exist on my network result in a lot of ARP boardcasts. As for what ARP and Gratuitous ARP is, I’ll let AI do the talking here since it’s much more capable of doing so than I am.

ARP (Address Resolution Protocol)

  • Purpose: ARP is used to find the hardware address (MAC address) of a device when you only know its network address (IP address).
  • How it Works: When a device wants to send data to another device on the same local network, it needs the recipient’s MAC address. The sender broadcasts an ARP request asking, “Who has this IP address?” The device with that IP address responds with its MAC address.
  • Example: Think of ARP like asking for someone’s phone number (MAC address) when you only know their name (IP address).

Gratuitous ARP

  • Purpose: Gratuitous ARP is used to update other devices on the network about a change in the sender’s MAC address or to announce its presence.
  • How it Works: A device sends an ARP request or reply, but instead of asking for another device’s MAC address, it announces its own IP and MAC address. This can be useful in scenarios like:
    • IP Conflict Detection: To check if another device is using the same IP address.
    • Updating Network Devices: To inform other devices about a change in its MAC address, which can happen if a network interface is replaced or if a virtual machine moves to a different host.
  • Example: Imagine you move to a new house (change your MAC address) but keep the same phone number (IP address). You send a message to your friends (other devices) saying, “Hey, I still have the same phone number, but I’m at a new address now.”

In summary, ARP helps devices find each other on a local network, while Gratuitous ARP is used to announce changes or updates to the network.

https://chat.mistral.ai/chat

Armed with some new knowledge, I decided to reduce the range of the nmap tracker to two /26 subnets, since this is where my computers live and quite frankly, I don’t need nmap to track my IoT devices anyway. While the gains are marginal, I do see some advantages here:

  • Less (broadcast) traffic over my entire network (including wireless)
  • LXC containers have less CPU time to spend on handling ARP traffic
  • A lot of nmap entities can be deleted from Home Assistant
    • Reduces the number of state_changed events
    • Reduces the database load

Manjaro: Black Screen when booting

PhillipLeave a comment

Yesterday I wanted to do a system upgrade on my Manjaro box because there were about 250+ packages that needed an update. There were quite a couple of conflicts that prevented me from upgrading so I manually disabled the packages that were conflicting until pacman was happy and I left my system to upgrade while I went to watch The Office.

An hour later I went to check the progress and I was greeted with a “screen locker is broken” error message all over the screen with instructions on how I could fix it. After logging in on another TTY I got an error that flatpak was missing some libraries and this error would block the whole terminal. A hard reset later I got nothing but a black screen. No services scrolling by, no splash screen, no kernel panic and not even the sound of my spinning rust stopping. My Manjaro installation was dead.

To fix this I downloaded the latest Manjaro ISO and booted from it. Once I got into the live environment it was fairly easy to chroot into my dead system

manjaro-chroot -a

The first thing people suggested was to update the mirrors and re-run a system update

pacman-mirrors -f
pacman -Syyu

Unfortunately, pacman was also broken and gave me the following error message

pacman: error while loading shared libraries: libicuuc.so.76: cannot open shared object file: No such file or directory

This is unfortunate, because without pacman I can’t re-install the missing library. So I headed to the package repository and downloaded the package from the mirror. After unpacking the archive in the live environment’s Downloads folder I moved everything to my Manjaro installation (your target location my vary depending on how manjaro-chroot mounted your volumes).

cp -r ~/Downloads/icu-76.1-1-x86_64.pkg/usr /mnt/usr

After copying the files, pacman worked, but I couldn’t install the missing icu package because the files were already there. Telling pacman to overwrite the files anyways solved the issue.

pacman -Syu --overwrite "/usr/*" --overwite "/var/*"

Unfortunately, after rebooting my system, I still got nothing but a black screen. So once more I went to the live environment and chroot’ed into my system. At this point I started guessing that there was something wrong with my nvidia drivers and after looking through the installed packages I quickly discovered that the nvidia drivers weren’t even installed anymore. So I reinstalled them (depending on your GPU, you might need another version):

pacman -Sy nvidia-470xx-dkms

After ignoring the packagekit error, I rebooted the system and ‘lo and behold, it was alive again.

So, what did we learn this weekend ?

  • Don’t cherry pick when doing system updates.
  • Make sure flatpak isn’t installed
  • Nvidia drivers can really kill your system (if I remove my GPU to boot from the iGPU, I actually get the same black screen)