Linux on RESEARCHUT

Immutable Debian

rrs@researchut.com (Ritesh Raj Sarraf) — Sat, 20 Dec 2025 00:00:00 +0000

Immutable Atomic Linux Distirbutions

Of late, I’ve been hearing a lot of (good) things about Immutable Linux Distributions, from friends, colleagues and mentors. It has been something on my plate for some time, to explore. But given the nature of the subject, it has been delayed for a while. Reasons are simple; I can only really judge this product if I use it for some time; and it has to be on my primary daily driver machine.

Personal life, this year, has been quite challenging as well. Thus it got pushed to until now.

Chrome OS

I’ve realized that I’ve been quite late to a lot of Linux parties. Containers, Docker, Kubernetes, Golang, Rust, Immutable Linux and many many more.

Late to the extent that I’ve had a Chrome Book lying at home for many months but never got to tinker with it at all.

Having used it for just around 2 weeks now, I can see what a great product Google built with it. In short, this is exactly how a Linux desktop integration should be. The GUI integration is just top notch. There’s consistency across all applications rendered on the Chrome OS

The integration of [X]Wayland and friends is equally good. Maybe Google should consider opensourcing all those components. IIRC, exo, sommelier, xwayland, ash and many more.

I was equally happy to see their Linux Development Environment offering on supported hardware. While tightly integrated, it still allows power users to tinker things around. I was quite impressed to see nested containers in crostini. Job well done.

All of this explains why there’s much buzz about Immutable Atomic Linux Distributions these days.

Then, there’s the Android integration, which is just awesome in case you care of it. Both libndk and libhoudini are well integrated and nicely usable.

Immutable Linux Distributions

This holiday season I wanted to find and spend some time catching up on stuff I had been prolonging.

I chose to explore this subject while trying to remain in familiar Debian land. So my first look was to see if there was any product derived out of the Debian base.

That brought me to Vanilla OS Orchid. This is a fresh out of oven project, recently switched to being based on Debian Sid. Previous iteration used Ubuntu as the base.

Vanilla OS turned out to be quite good an experience. The stock offering is created well enough to serve the general audience. And the framework is such wonderfully structured that seasoned users can tinker around with it, without much fuss.

Vanilla OS uses an A/B Partition model for how system updates are rolled. At any point, when a new OTA update is pushed, it gets applied to the inactive A/B partition. And it gets activated at next boot. If things break, user has the option to switch to the previous state. Just the usual set of expectations one would have with an immutable distribution.

What they’ve done beautifully is:

Integration Device Mapper LVM for A/B Partition
Linux Container OCI images to provison/flash A/B Paritions
Developed abroot utility for A/B Partition management
APX (Distrobox) integration for container workflows, with multiple Linux flavors
No sudo. Everything done via pkexec

But the most awesome thing I liked in Vanilla OS is custom images. This allows power users to easily tinker with the developer workflow and generate new images, tailored for their specific use cases. All of this done levraging the GitHub/GitLab CI/CD workflows, which I think is just plain awesome. Given that payload is of the OCI format, the CICD workflow just generates new OCI images and publishes to a registry. And then the same is pulled to the client as an OTA.

Hats off to this small team/community for doing such nice integration work, ultimately producing a superb Immutable Atomic Linux Distribution based on the Debian base.

Immutable Linux

My primary work machine has grown over the years, being on the rolling Debian Testing/Unstable channel. And I don’t much feel the itch ever to format my (primary) machine so quick, no matter how great the counter offer is.

So that got me wondering how to have some of bling of the immutable world that I’ve tasted (Thanks Chrome OS and Vanilla OS). With a fair idea of what they offer in features, I drew a line to what I’d want on my primary machine.

read-only rootfs
read-only /etc/

This also kinda hardens my systems to an extent that I can’t accidentally cause catastrophic damage to it.

The feature I’m letting go of is the A/B Partition (rpm-ostree for Fedora land). While a good feature, having to integrate it into my current machine is going to be very very challenging.

I actually feel that, the core assumption the Immutable Distros make, that all hardware is going to Just Work, is flawed. While Linux has substantially improved over the past years, there’s still a hit/miss when introducing very recent hardware.

Immutable Linux is targeted for the novice user, who won’t accidentally mess with the system. But what would the novice user do in case they have issues with their recently purchased hardware, that they are attempting to run (Immutable) Linux on.

Ritesh’s Immutable Debian

With the premise set, on to sailing in immutable land.

There’s another ground breaking innovation that has been happening; which I think everyone is aware of. And may be using it as well, direct or indirect.

Artificial Intelligence

While I’ve only been a user for a couple of months as I draft this post, I’m now very much impressed with all this innovation. Being at the consumer end has me appreciating it for what it has offered thus far. And I haven’t even scratched the surface. I’m making attempts at developing understanding of Machine Learning and Artificial Intelligence but there’s a looonnngg way to go still.

What I’m appreciating the most is the availability of the AI Technology. It has helped me be more efficient. And thus I get to use the gain (time) with family.

To wrap, what I tailored my primary OS to, wouldn’t have been possible without assistance from AI.

With that, I disclaim that the rest of this article is primarily drafted by my AI Companion. This is going to serve me as a reference for future, when I forget about how all of this was structured.

🏗️ System Architecture: Immutable Debian (Btrfs + MergerFS)

This system is a custom-hardened Immutable Workstation based on Debian Testing/Unstable. It utilizes native Btrfs properties and surgical VFS mounting to isolate the Operating System from persistent data.

1. Storage Strategy: Subvolume Isolation

The system resides on a LUKS-encrypted NVMe partition, using a flattened subvolume layout to separate the “Gold Master” OS from volatile and persistent data.

Mount Point	Subvolume Path	State	Purpose
`/`	`/ROOTVOL`	RO	The core OS image.
`/etc`	`/ROOTVOL/etc`	RO	System configuration (Snapshot-capable).
`/home/rrs`	`/ROOTVOL/home/rrs`	RW	User data and Kitty terminal configs.
`/var/lib`	`/ROOTVOL/var/lib`	RW	Docker, Apt state, and system DBs.
`/var/spool`	`/ROOTVOL/var/spool`	RW	Mail queues and service state.
`/swap`	`/ROOTVOL/swap`	RW	Isolated path for No_COW Swapfile.
`/disk-tmp`	`/ROOTVOL/disk-tmp`	RW	MergerFS overflow tier.

1.1 /etc/fstab

❯ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
# --- ROOT & BOOT ---
/dev/mapper/nvme0n1p3_crypt / btrfs autodefrag,compress=zstd,discard=async,noatime,defaults,ro 0 0
/dev/nvme0n1p2 /boot ext4 defaults 0 2
/dev/nvme0n1p1 /boot/efi vfat umask=0077 0 1
# --- SWAP ---
# Mount the "Portal" to the swap subvolume using UUID (Robust)
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /swap btrfs subvol=/ROOTVOL/swap,defaults,noatime 0 0
# Activate the swap file by path (Correct for files)
/swap/swapfile none swap defaults 0 0
# --- DATA / MEDIA ---
UUID=439e297a-96a5-4f81-8b3a-24559839539d /media/rrs/TOSHIBA btrfs noauto,compress=zstd,space_cache=v2,subvolid=5,subvol=/,user
# --- MERGERFS ---
# --- DISK-TMP (MergerFS Overflow Tier) ---
# Ensure this ID matches your actual disk-tmp subvolume
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /disk-tmp btrfs subvolid=417,discard=async,defaults,noatime,compress=zstd 0 0
tmpfs /ram-tmp tmpfs defaults 0 0
/ram-tmp:/disk-tmp /tmp fuse.mergerfs x-systemd.requires=/ram-tmp,x-systemd.requires=/disk-tmp,defaults,allow_other,use_ino,nonempty,minfreespace=1G,category.create=all,moveonenospc=true 0 0
# --- IMMUTABILITY PERSISTENCE LAYERS ---
# We explicitly mount these subvolumes so they remain Writable later.
# UUID is the same as your /var/lib entry (your main Btrfs volume).
# 1. /var/lib (Docker, Apt state) - ID 50659
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /var/lib btrfs subvolid=50659,discard=async,defaults,noatime,compress=zstd 0 0
# 2. /home/rrs (User Data) - ID 13032
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /home/rrs btrfs subvolid=13032,discard=async,defaults,noatime,compress=zstd 0 0
# 3. /etc (System Config) - ID 13030
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /etc btrfs subvolid=13030,discard=async,defaults,noatime,compress=zstd,ro 0 0
# 4. /var/log (Logs) - ID 406
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /var/log btrfs subvolid=406,discard=async,defaults,noatime,compress=zstd 0 0
# 5. /var/cache (Apt Cache) - ID 409
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /var/cache btrfs subvolid=409,discard=async,defaults,noatime,compress=zstd 0 0
# 6. /var/tmp (Temp files) - ID 401
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /var/tmp btrfs subvolid=401,discard=async,defaults,noatime,compress=zstd 0 0
# /var/spool
UUID=4473b40b-bb46-43d6-b69c-ef17bfcac41c /var/spool btrfs subvolid=50689,discard=async,defaults,noatime,compress=zstd 0 0

2. Tiered Memory Model (/tmp)

To balance performance and capacity, /tmp is managed via MergerFS:

Tier 1 (RAM): tmpfs mounted at /ram-tmp.
Tier 2 (Disk): Btrfs subvolume mounted at /disk-tmp.
Logic: Files are written to RAM first. If RAM falls below 1GB available, files spill over to the Btrfs disk tier.

3. Hibernation & Swap Logic

Size: 33 GiB (Configured for Suspend-to-Disk with 24GB RAM).
Attribute: The /swap subvolume is marked No_COW (+C).
Kernel Integration:
- resume=UUID=... (Points to the unlocked LUKS container).
- resume_offset=... (Physical extent mapping for Btrfs).

3.1 systemd sleep/Hibernation

❯ cat /etc/systemd/sleep.conf.d/sleep.conf
[Sleep]
HibernateDelaySec=12min

and

❯ cat /etc/systemd/logind.conf.d/logind.conf
[Login]
HandleLidSwitch=suspend-then-hibernate
HandlePowerKey=suspend-then-hibernate
HandleSuspendKey=suspend-then-hibernate
SleepOperation==suspend-then-hibernate

4. Immutability & Safety Mechanisms

The system state is governed by two key components:

A. The Control Script (`immutectl`)

Handles the state transition by flipping Btrfs properties and VFS mount flags in the correct order.

sudo immutectl unlock: Sets ro=false and remounts rw.
sudo immutectl lock: Sets ro=true and remounts ro.

❯ cat /usr/local/bin/immutectl
#!/bin/bash
# Ensure script is run as root
if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root (sudo)."
exit 1
fi
ACTION=$1
case $ACTION in
unlock)
echo "🔓 Unlocking / and /etc for maintenance..."
# 1. First, tell the Kernel to allow writes to the mount point
mount -o remount,rw /
mount -o remount,rw /etc
# 2. Now that the VFS is RW, Btrfs will allow you to change the property
btrfs property set / ro false
btrfs property set /etc ro false
echo "Status: System is now READ-WRITE."
;;
lock)
echo "🔒 Locking / and /etc (Immutable Mode)..."
sync
btrfs property set / ro true
btrfs property set /etc ro true
# We still attempt remount, but we ignore failure since Property is the Hard Lock
mount -o remount,ro / 2>/dev/null
mount -o remount,ro /etc 2>/dev/null
echo "Status: System is now READ-ONLY (Btrfs Property Set)."
;;
status)
echo "--- System Immutability Status ---"
for dir in "/" "/etc"; do
# Get VFS state
VFS_STATE=$(grep " $dir " /proc/mounts | awk '{print $4}' | cut -d, -f1)
# Get Btrfs Property state
BTRFS_PROP=$(btrfs property get "$dir" ro | cut -d= -f2)
# Determine overall health
if [[ "$BTRFS_PROP" == "true" ]]; then
FINAL_STATUS="LOCKED (RO)"
else
FINAL_STATUS="UNLOCKED (RW)"
fi
echo "Path: $dir"
echo " - VFS Layer (Mount): $VFS_STATE"
echo " - Btrfs Property: ro=$BTRFS_PROP"
echo " - Effective State: $FINAL_STATUS"
# Check for mismatch (The "Busy" scenario)
if [[ "$VFS_STATE" == "rw" && "$BTRFS_PROP" == "true" ]]; then
echo " ⚠️ NOTICE: VFS is RW but Btrfs is RO. System is effectively Immutable."
fi
echo ""
done
;;
*)
echo "Usage: $0 {lock|unlock|status}"
exit 1
;;
esac

B. The Smart Seal (`immutability-seal.service`)

A systemd one-shot service that ensures the system is locked on boot.

Fail-safe: The service checks /proc/cmdline for the standalone word rw. If found (via GRUB manual override), the seal is aborted to allow emergency maintenance.

❯ cat /etc/systemd/system/immutability-seal.service
[Unit]
Description=Ensure Btrfs Immutable Properties are set on Boot (unless rw requested)
DefaultDependencies=no
After=systemd-remount-fs.service
Before=local-fs.target
# Don't run in emergency/rescue modes
#ConditionPathExists=!/run/systemd/seats/seat0
[Service]
Type=oneshot
# The robust check: exit if 'rw' exists as a standalone word
ExecStartPre=/bin/sh -c '! grep -qE "\brw\b" /proc/cmdline'
ExecStartPre=mount -o remount,rw /
ExecStart=/usr/bin/btrfs property set / ro true
ExecStart=/usr/bin/btrfs property set /etc ro true
ExecStartPost=mount -o remount,ro /
RemainAfterExit=yes
[Install]
WantedBy=local-fs.target

5. Monitoring & Maintenance

Nagging: A systemd user-timer runs immutability-nag every 15 minutes to notify the desktop session if the system is currently in an “Unlocked” state.
Verification: Use sudo immutectl status to verify that both the VFS Layer and Btrfs Properties are in sync.

5.1 Nagging

❯ cat ~/bin/immutability-nag
#!/bin/bash
# Check Btrfs property
BTRFS_STATUS=$(btrfs property get / ro | cut -d= -f2)
if [[ "$BTRFS_STATUS" == "false" ]]; then
# Use notify-send (Standard, fast, non-intrusive)
notify-send -u critical -i security-low \
"🔓 System Unlocked" \
"Root is currently WRITABLE. Run 'immutectl lock' when finished."
fi

and

❯ usystemctl cat immutability-nag.service
# /home/rrs/.config/systemd/user/immutability-nag.service
[Unit]
Description=Check Btrfs immutability and notify user
# Ensure it doesn't run before the graphical session is ready
After=graphical-session.target
[Service]
Type=oneshot
ExecStart=%h/bin/immutability-nag
# Standard environment for notify-send to find the DBus session
Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/%U/bus
[Install]
WantedBy=default.target
   ~   20:35:15
❯ usystemctl cat immutability-nag.timer
# /home/rrs/.config/systemd/user/immutability-nag.timer
[Unit]
Description=Check immutability every 15 mins
[Timer]
OnStartupSec=5min
OnUnitActiveSec=15min
[Install]
WantedBy=timers.target

And the resultant nag in action.

Immutable Debian Nag

5.2 Verification

❯ sudo immutectl status
[sudo] password for rrs:
--- System Immutability Status ---
Path: /
- VFS Layer (Mount): rw
- Btrfs Property: ro=false
- Effective State: UNLOCKED (RW)
Path: /etc
- VFS Layer (Mount): rw
- Btrfs Property: ro=false
- Effective State: UNLOCKED (RW)
   ~   21:14:08
❯ sudo immutectl lock
🔒 Locking / and /etc (Immutable Mode)...
Status: System is now READ-ONLY (Btrfs Property Set).
   ~   21:14:15
❯ sudo immutectl status
--- System Immutability Status ---
Path: /
- VFS Layer (Mount): rw
- Btrfs Property: ro=true
- Effective State: LOCKED (RO)
⚠️ NOTICE: VFS is RW but Btrfs is RO. System is effectively Immutable.
Path: /etc
- VFS Layer (Mount): rw
- Btrfs Property: ro=true
- Effective State: LOCKED (RO)
⚠️ NOTICE: VFS is RW but Btrfs is RO. System is effectively Immutable.

Date Configured: December 2025
Philosophy: The OS is a diagnostic tool. If an application fails to write to a locked path, the application is the variable, not the system.

Wrap

Overall, I’m very very happy with, the result of a day of working together with AI. I wouldn’t have gotten things done so quick in such time if it wasn’t around. Such great is this age of AI.

KDE PowerDevil Systemd Inhibit

rrs@researchut.com (Ritesh Raj Sarraf) — Tue, 28 Oct 2025 00:00:00 +0000

With KDE 6.5.0, PowerDevil has ~~broken~~ forced its own set of suspend/hibernate inhibitors onto logind. And to my knowledge, there’s no way to disable them.

As a user, I’d prefer to set lid action as Do nothing and really expect KDE/PowerDevil to do nothing in that regard. But with KDE 6.5.0 PowerDevil forces those inhibitors whatsoever.

❯ systemd-inhibit --list
WHO UID USER PID COMM WHAT WHY >
ModemManager 0 root 3541 ModemManager sleep ModemManager needs to reset devices >
NetworkManager 0 root 3453 NetworkManager sleep NetworkManager needs to turn off networks >
UPower 0 root 4342 upowerd sleep Pause device polling >
PowerDevil 1000 rrs 82735 org_kde_powerde handle-power-key:handle-suspend-key:handle-hibernate-key:handle-lid-switch KDE handles power events >
Screen Locker 1000 rrs 4844 kwin_wayland sleep Ensuring that the screen gets locked before going to sleep >

5 inhibitors listed.

This essentially prohibits logind to act on the lid actions. And instead forces the user to depend on nothing else and other than PowerDevil. This assumes the wishful thought that PowerDevil is Solid.

I’d love to continue using my suspend workflow via systemd’s suspend-then-hibernate target as it has been working reliably for years. And it also allows me to customize the behavior as I see fit.

Of course, I do have the option to trigger systemd suspend-then-hibernate manually, every time, before closing the lid. But computers and automation has spoilt things.

The quick workaround/fix is to delegate it to ACPI, on platforms that support it. Thankfully all of x86 to my knowledge.

So, in ACPI actions I’ve a new config set to:

❯ cat /etc/acpi/actions/lm_lid.sh
#! /bin/sh

grep close /proc/acpi/button/lid/LID/state && systemctl suspend-then-hibernate

And with that I can be back to reliably (and carelessly) suspend-then-hibernate my laptop.

apt-offline 1.8.6

rrs@researchut.com (Ritesh Raj Sarraf) — Wed, 26 Feb 2025 18:56:04 +0530

apt-offline 1.8.6

apt-offline version 1.8.6 was released almost 3 weeks ago on 08/February/2025

This release includes many bug fixes from community users.

Error out if we cannot initialize the APT lock. Thanks to Matthew Maslak
check for checksum and handle appropriately (#217) Thanks to Dan Whitman (Github:kyp44)
Honor the –allow-unauthenticated option. Thanks to João A (Github: Jonybat)
Retry when server reports 429 Too Many Requests occurs. Thanks to Zoltan Kelemen (Github: misterzed88)
Also support file:/// url types. Thanks to c4bhuf@github
Honor user specified extra gpg keyrings

Changelog

apt-offline (1.8.6-1) unstable; urgency=medium

 * Error out if we cannot initialize the APT lock.
 Thanks to Matthew Maslak
 * check for checksum and handle appropriately (#217)
 Thanks to Dan Whitman (Github:kyp44)
 * Honor the --allow-unauthenticated option.
 Thanks to João A (Github: Jonybat)
 * Retry when server reports 429 Too Many Requests occurs.
 Thanks to Zoltan Kelemen (Github: misterzed88)
 * Also support file:/// url types.
 Thanks to c4bhuf@github
 * Honor user specified extra gpg keyrings

 -- Ritesh Raj Sarraf <rrs@debian.org> Sat, 08 Feb 2025 20:46:24 +0530

Resources

Tarball and Zip archive for apt-offline are available here
Packages should be available in Debian.
Development for apt-offline is currently hosted here

Systemd Service Hang

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 22 Apr 2022 00:00:00 +0000

Finally, TIL, what can all be the reason for systemd services to hang indefinitely. The internet is flooded with numerous reports on this topic but no clear answers. So no more uselessly marked workarounds like: systemctl daemon-reload and systemctl-daemon-reexec for this scenario.

The scene would be something along the lines of:

rrs 6467 0.0 0.0 23088 15852 pts/1 Ss 12:53 0:00 | | \_ /bin/bash
rrs 11512 0.0 0.0 14876 4608 pts/1 S+ 13:18 0:00 | | | \_ systemctl restart snapper-timeline.timer
rrs 11513 0.0 0.0 14984 3076 pts/1 S+ 13:18 0:00 | | | \_ /bin/systemd-tty-ask-password-agent --watch
rrs 11514 0.0 0.0 234756 6752 pts/1 Sl+ 13:18 0:00 | | | \_ /usr/bin/pkttyagent --notify-fd 5 --fallback

The snapper-timeline service is important to me and it not running for months is a complete failure. Disappointingly, commands like systemctl --failed do not report of this oddity. The overall system status is reported to be fine, which is completely incorrect.

Thankfully, a kind soul’s comment gave the hint. The problem is that you could be having certain services in Activating status, which thus blocks all other services; quietly. So much for the unnecessary fun.

Looking further, in my case, it was:

rrs@priyasi:~$ systemctl list-jobs
JOB UNIT TYPE STATE
81 timers.target start waiting
85 man-db.timer start waiting
88 fstrim.timer start waiting
3832 snapper-timeline.service start waiting
83 snapper-timeline.timer start waiting
39 systemd-time-wait-sync.service start running
87 logrotate.timer start waiting
84 debspawn-clear-caches.timer start waiting
89 plocate-updatedb.timer start waiting
91 dpkg-db-backup.timer start waiting
93 e2scrub_all.timer start waiting
40 time-sync.target start waiting
86 apt-listbugs.timer start waiting

13 jobs listed.
13:12 ♒ ॐ ♅ ♄ ⛢ ☺ 😄

That was it. I knew the systemd-timesyncd service, in the past, had given me enough headaches. And so was it this time, just quietly doing it all again.

rrs@priyasi:~$ systemctl status systemd-time-wait-sync.service
● systemd-time-wait-sync.service - Wait Until Kernel Time Synchronized
 Loaded: loaded (/lib/systemd/system/systemd-time-wait-sync.service; enabled; vendor preset>
 Active: activating (start) since Fri 2022-04-22 13:14:25 IST; 1min 38s ago
 Docs: man:systemd-time-wait-sync.service(8)
 Main PID: 11090 (systemd-time-wa)
 Tasks: 1 (limit: 37051)
 Memory: 836.0K
 CPU: 7ms
 CGroup: /system.slice/systemd-time-wait-sync.service
 └─11090 /lib/systemd/systemd-time-wait-sync

Apr 22 13:14:25 priyasi systemd[1]: Starting Wait Until Kernel Time Synchronized...
Apr 22 13:14:25 priyasi systemd-time-wait-sync[11090]: adjtime state 5 status 40 time Fri 2022->
13:16 ♒ ॐ ♅ ♄ ⛢ ☹ 😟=> 3

Dear LazyWeb, anybody knows of why the systemd-time-wait-sync service would hang indefinitely? I’ve had identical setups on many machines, in the same network, where others don’t exhibit this problem.

rrs@priyasi:~$ systemctl cat systemd-time-wait-sync.service

...snipped...

[Service]
Type=oneshot
ExecStart=/lib/systemd/systemd-time-wait-sync
TimeoutStartSec=infinity
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

The TimeoutStartSec=infinity is definitely an attribute that shouldn’t be shipped in any system services. There are use cases for it but that should be left for local admins to explicitly decide. Hanging for infinity is not a desired behavior for a system service.

In figuring all this out, today I learnt the handy systemctl list-jobs command, which will give the list of active running/blocked/waiting jobs.

Update: 2024-08-15

This week I finally found the cause of the issue. I have a bunch of bridge interfaces defined on my machine. And all of them, most of their time, would be DOWN


@ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: withnet: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether XXXXXXXXXXXXXXXXX brd ff:ff:ff:ff:ff:ff
3: nonet: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether XXXXXXXXXXXXXXXXX brd ff:ff:ff:ff:ff:ff
4: tap0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether XXXXXXXXXXXXXXXXX brd ff:ff:ff:ff:ff:ff
6: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether XXXXXXXXXXXXXXXXX brd ff:ff:ff:ff:ff:ff
inet 10.42.1.66/24 brd 10.42.1.255 scope global dynamic noprefixroute wlan0
valid_lft 5325sec preferred_lft 5325sec
inet6 fe80::9bc0:e362:7c9d:be7c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
7: email-laptop: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet XXXXXXXXXXXXXXXXX scope global noprefixroute email-laptop
valid_lft forever preferred_lft forever
inet6 XXXXXXXXXXXXXXXXX scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 XXXXXXXXXXXXXXXXXXXXXXXX/64 scope link noprefixroute
valid_lft forever preferred_lft forever
⛢ 13:55:21 rrs@priyasi ~/pu/researchut-hugo researchut|+1…2

And as it stands, that is the cause of the problem.

systemd-timesyncd would invoke an attempt to sync time for all the interfaces defined under systemd-networkd. And given the setup, that is bound to ~~fail~~ timeout; which is why I had added the override against:

TimeoutStartSec=infinity

So, apparently, for every interface in networkd, it’d attempt the ntp sync. It reports along the lines of:

Aug 15 13:24:06 priyasi systemd-timesyncd[1987]: Network configuration changed, trying to establish connection.

So I learnt this week, of the attribute to handle this:

@ cat sysbr0.network
[Match]
Name=withnet
[Network]
DHCPServer=yes
IPv4Forwarding=yes
IPv6Forwarding=yes
IPMasquerade=both
Address=192.168.1.1/24
LLMNR=yes
[Link]
RequiredForOnline=no
#[DHCPServer]
#DNS=192.168.1.1

With that, NTP time synchronization is back to what it should be. Precision synchronization of the system time.

ॐ 14:11:35 rrs@priyasi /etc/systemd/network
@ timedatectl show
LocalRTC=no
CanNTP=yes
NTP=yes
NTPSynchronized=yes
TimeUSec=Thu 2024-08-15 14:12:45 IST
RTCTimeUSec=Thu 2024-08-15 14:12:45 IST

Btrfs Subvol Fix

rrs@researchut.com (Ritesh Raj Sarraf) — Wed, 20 Apr 2022 22:41:53 +0530

There surely is need for better tooling on the BTRFS File System side.

While migrating my setup from one machine to another, this is one issue I came to be aware of, only today, when my backup tool (btrbk) complained about it. Following the pointers, I see the below snippet in btrfs-subvolume manual page.

 A snapshot that was created by send/receive will be read-only, with different last change generation, read-only and with set received_uuid which identifies the subvolume on the
 filesystem that produced the stream. The usecase relies on matching data on both sides. Changing the subvolume to read-write after it has been received requires to reset the
 received_uuid. As this is a notable change and could potentially break the incremental send use case, performing it by btrfs property set requires force if that is really desired by
 user.

 Note
 The safety checks have been implemented in 5.14.2, any subvolumes previously received (with a valid received_uuid) and read-write status may exist and could still lead to
 problems with send/receive. You can use btrfs subvolume show to identify them. Flipping the flags to read-only and back to read-write will reset the received_uuid manually. There
 may exist a convenience tool in the future.

Fixing the Received UUID: flag meant running the below:

rrs@priyasi:.../spool$ sudo btrfs sub show /
WARNING: the subvolume is read-write and has received_uuid set,
 don't use it for incremental send. Please see section
 'SUBVOLUME FLAGS' in manual page btrfs-subvolume for
 further information.
ROOTVOL
 Name: ROOTVOL
 UUID: 122b0de1-e6f2-6845-aba0-6bf766c16526
 Parent UUID: -
 Received UUID: 34772967-c709-5146-bf20-898f7dbc2c1f
 Creation time: 2021-12-02 19:59:29 +0530
 Subvolume ID: 256
 Generation: 138473
 Gen at creation: 7
 Parent ID: 5
 Top level ID: 5
 Flags: -
 Send transid: 35245
 Send time: 2021-12-02 19:59:29 +0530
 Receive transid: 34
 Receive time: 2021-12-02 20:13:11 +0530
 Snapshot(s):
 ROOTVOL/.snapshots/1/snapshot
 ROOTVOL/.snapshots/2/snapshot
22:40 ♒ ॐ ♅ ♄ ⛢ ☺ 😄


rrs@priyasi:.../spool$ sudo btrfs property set / ro true
WARNING: read-write subvolume with received_uuid, this is bad
22:40 ♒ ॐ ♅ ♄ ⛢ ☺ 😄



rrs@priyasi:.../spool$ sudo btrfs property set -f / ro false
22:40 ♒ ॐ ♅ ♄ ⛢ ☺ 😄



rrs@priyasi:.../spool$ sudo btrfs sub show /
ROOTVOL
 Name: ROOTVOL
 UUID: 122b0de1-e6f2-6845-aba0-6bf766c16526
 Parent UUID: -
 Received UUID: -
 Creation time: 2021-12-02 19:59:29 +0530
 Subvolume ID: 256
 Generation: 138473
 Gen at creation: 7
 Parent ID: 5
 Top level ID: 5
 Flags: -
 Send transid: 0
 Send time: 2021-12-02 19:59:29 +0530
 Receive transid: 138480
 Receive time: 2022-04-20 22:40:43 +0530
 Snapshot(s):
 ROOTVOL/.snapshots/1/snapshot
 ROOTVOL/.snapshots/2/snapshot
22:40 ♒ ॐ ♅ ♄ ⛢ ☺ 😄

Hoping there won’t be surprises in the coming months. 🤞

apt-offline 1.8.4

rrs@researchut.com (Ritesh Raj Sarraf) — Sat, 12 Feb 2022 20:47:04 +0530

apt-offline 1.8.4

apt-offline version 1.8.4 has been released.

This release includes many bug fixes but the important ones are:

Better GPG signature handling
Support for verifying InRelease files

Changelog

apt-offline (1.8.4-1) unstable; urgency=medium

 [ Debian Janitor ]
 * Update standards version to 4.5.0, no changes needed.

 [ Paul Wise ]
 * Clarify file type in unknown file message
 * Fix typos
 * Remove trailing whitespace
 * Update LICENSE file to match official GNU version
 * Complain when there are no valid keyrings instead of missing keyrings
 * Make all syncrhronised files world readable
 * Fix usage of indefinite articles
 * Only show the APT Offline GUI once in the menu
 * Update out of date URLs
 * Fix date and whitespace issues in the manual page
 * Replace stereotyping with an appropriate word
 * Switch more Python shebangs to Python 3
 * Correct usage of the /tmp/ directory
 * Fix YAML files
 * Fix usage of the log API
 * Make the copying of changelog lines less brittle
 * Do not split keyring paths on whitespace

 [ Ritesh Raj Sarraf ]
 * Drop the redundant import of the apt module.
 Thanks to github/dandelionred
 * Fix deprecation of get_bugs() in debianbts
 * Drop the unused IgnoredBugTypes
 * Set encoding for files when opening
 * Better error logging when apt fails
 * Don't mandate a default option
 * Demote metadata errors to verbose
 * Also log an error message for every failed .deb url
 * Check hard for the url type
 * Check for ascii armored signature files.
 Thanks to David Klnischkies
 * Add MIME type for InRelease files
 * Drop patch 0001-Drop-the-redundant-import-of-the-apt-module.patch.
 Now part of the 1.8.4 release
 * Prepare release 1.8.3
 * Prepare release 1.8.4
 * debian packaging
 + Bump debhelper compatibility to 13
 + Update install files

 [ Dean Anderson ]
 * [#143] Added support for verifying InRelease files

 -- Ritesh Raj Sarraf <rrs@debian.org> Sat, 12 Feb 2022 18:52:58 +0530

Resources

Tarball and Zip archive for apt-offline are available here
Packages should be available in Debian.
Development for apt-offline is currently hosted here

ThinkPad AMD Debian

rrs@researchut.com (Ritesh Raj Sarraf) — Tue, 11 Jan 2022 19:37:24 +0530

After a hiatus of 6 years, it was nice to be back with the ThinkPad. This blog post briefly touches upon my impressions with the current generation ThinkPad T14 Gen2 AMD variant.

ThinkPad T14 Gen2 AMD

Lenovo

It took 8 weeks to get my hands on the machine. Given the pandemic, restrictions and uncertainities, not sure if I should call it an ontime delivery. This was a CTO - Customise-to-order; so was nice to get rid of things I really didn’t care/use much. On the other side, it also meant I could save on some power. It also came comparatively cheaper overall.

No fingerprint reader
No Touch screen

There’s still parts where Lenovo could improve. Or less frustate a customer. I don’t understand why a company would provide a full customization option on their portal, while at the same time, not provide an explicit option to choose the make/model of the hardware one wants. Lenovo deliberately chooses to not show/specify which WiFi adapter one could choose. So, as I suspected, I ended up with a MEDIATEK Corp. Device 7961 wifi adapter.

AMD

For the first time in my computing life, I’m now using AMD at the core. I was pretty frustrated with annoying Intel Graphics bugs, so decided to take the plunge and give AMD/ATI a shot, knowing that the radeon driver does have decent support. So far, on the graphics side of things, I’m glad that things look bright. The stock in-kernel radeon driver has been working perfect for my needs and I haven’t had to tinker even once so far, in my 30 days of use.

On the overall system performance, I have not done any benchmarks nor do I want to do. But wholly, the system performance is smooth.

Power/Thermal

This is where things need more improvement on the AMD side. This AMD laptop terribly draws a lot of power in suspend mode. And it isn’t just this machine, but also the previous T14 Gen1 which has similar problems. I’m not sure if this is a generic ThinkPad problem, or an AMD specific problem. But coming from the Dell XPS 13 9370 Intel, this does draw a lot lot more power. So much, that I chose to use hibernation instead.

Similarly, on the thermal side, this machine doesn’t cool down well as compared the the Dell XPS Intel one. On an idle machine, its temperature are comparatively higher. Looking at powertop reports, it does show to consume an average of 10 watts power even while idle.

I’m hoping these are Linux ingeration issues and that Lenovo/AMD will improve things in the coming months. But given the user feedback on the ThinkPad T14 Gen1 thread, it may just be wishful thinking.

Linux

The overall hardware support has been surprisingly decent. The MediaTek WiFi driver had some glitches but with Linux 5.15+, things have considerably improved. And I hope the trend will continue with forthcoming Linux releases. My previous device driver experience with MediaTek wasn’t good but I took the plunge, considering that in the worst scenario I’d have the option to swap the card.

There’s a lot of marketing about Linux + Intel. But I took a jibe with Linux + AMD. There are glitches but nothing so far that has been a dealbreaker. If anything, I wish Lenovo/AMD would seriously work on the power/thermal issues.

Migration

Other than what’s mentioned above, I haven’t had any serious issues. I may have had some rare occassional hangs but they’ve been so infrequent that I haven’t spent time to investigate those.

Upon receiving the machine, my biggest requirement was how to switch my current workstation from Dell XPS to Lenovo ThinkPad. I’ve been using btrfs for some time now. And over the years, built my own practise on how to structure it. Things like, provisioning [sub]volumes, based on use cases is one thing I see. Like keeping separate subvols for: cache/temporary data, copy-on-write data , swap etc. I wish these things could be simplified; either on the btrfs tooling side or some different tool on top of it.

Below is filtered list of subvols created over years, that were worthy of moving to the new machine.

rrs@priyasi:~$ cat btrfs-volume-layout
ID 550 gen 19166 top level 5 path home/foo/.cache
ID 552 gen 1522688 top level 5 path home/rrs
ID 553 gen 1522688 top level 552 path home/rrs/.cache
ID 555 gen 1426323 top level 552 path home/rrs/rrs-home/Libvirt-Images
ID 618 gen 1522672 top level 5 path var/spool/news
ID 634 gen 1522670 top level 5 path var/tmp
ID 635 gen 1522688 top level 5 path var/log
ID 639 gen 1522226 top level 5 path var/cache
ID 992 gen 1522670 top level 5 path disk-tmp
ID 1018 gen 1522688 top level 552 path home/rrs/NoBackup
ID 1196 gen 1522671 top level 5 path etc
ID 23721 gen 775692 top level 5 path swap
18:54 ♒ ॐ ♅ ♄ ⛢ ☺ 😄

btrfs send/receive

This did come in handy but I sorely missed some feature. Maybe they aren’t there, or are there and I didn’t look close enough. Over the years, different attributes were set to different subvols. Over time I forget what feature was added where. But from a migration point of view, it’d be nice to say, “Take this volume and take it with all its attributes”. I didn’t find that functionality in send/receive.

There’s get/set-property which I noticed later but by then it was late. So some sort of tooling, ideally something like btrfs migrate or somesuch would be nicer.

In the file system world, we already have nice tools to take care of similar scenarios. Like with rsync, I can request it to carry all file attributes.

Also, iirc, send/receive works only on ro volumes. So there’s more work one needs to do in:

create ro vol
send
receive
don’t forget to set rw property
And then somehow find out other properties set on each individual subvols and [re]apply the same on the destination

I wish this all be condensed into a sub-command.

For my own sake, for this migration, the steps used were:

user@debian:~$ for volume in `sudo btrfs sub list /media/user/TOSHIBA/Migrate/ | cut -d ' ' -f9 | grep -v ROOTVOL | grep -v etc | grep -v btrbk`; do echo $volume; sud
o btrfs send /media/user/TOSHIBA/$volume | sudo btrfs receive /media/user/BTRFSROOT/ ; done
Migrate/snapshot_disk-tmp
At subvol /media/user/TOSHIBA/Migrate/snapshot_disk-tmp
At subvol snapshot_disk-tmp
Migrate/snapshot-home_foo_.cache
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_foo_.cache
At subvol snapshot-home_foo_.cache
Migrate/snapshot-home_rrs
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs
At subvol snapshot-home_rrs
Migrate/snapshot-home_rrs_.cache
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs_.cache
At subvol snapshot-home_rrs_.cache
ERROR: crc32 mismatch in command
Migrate/snapshot-home_rrs_rrs-home_Libvirt-Images
At subvol /media/user/TOSHIBA/Migrate/snapshot-home_rrs_rrs-home_Libvirt-Images
At subvol snapshot-home_rrs_rrs-home_Libvirt-Images
ERROR: crc32 mismatch in command
Migrate/snapshot-var_spool_news
At subvol /media/user/TOSHIBA/Migrate/snapshot-var_spool_news
At subvol snapshot-var_spool_news
Migrate/snapshot-var_lib_machines
At subvol /media/user/TOSHIBA/Migrate/snapshot-var_lib_machines
At subvol snapshot-var_lib_machines
Migrate/snapshot-var_lib_machines_DebianSidTemplate
..... snipped .....

And then, follow-up with:

user@debian:~$ for volume in `sudo btrfs sub list /media/user/BTRFSROOT/ | cut -d ' ' -f9`; do echo $volume; sudo btrfs property set -ts /media/user/BTRFSROOT/$volume ro false; done
ROOTVOL
ERROR: Could not open: No such file or directory
etc
snapshot_disk-tmp
snapshot-home_foo_.cache
snapshot-home_rrs
snapshot-var_spool_news
snapshot-var_lib_machines
snapshot-var_lib_machines_DebianSidTemplate
snapshot-var_lib_machines_DebSidArmhf
snapshot-var_lib_machines_DebianJessieTemplate
snapshot-var_tmp
snapshot-var_log
snapshot-var_cache
snapshot-disk-tmp

And then finally, renaming everything to match proper:

user@debian:/media/user/BTRFSROOT$ for x in snapshot*; do vol=$(echo $x | cut -d '-' -f2 | sed -e "s|_|/|g"); echo $x $vol; sudo mv $x $vol; done
snapshot-var_lib_machines var/lib/machines
snapshot-var_lib_machines_Apertisv2020ospackTargetARMHF var/lib/machines/Apertisv2020ospackTargetARMHF
snapshot-var_lib_machines_Apertisv2021ospackTargetARM64 var/lib/machines/Apertisv2021ospackTargetARM64
snapshot-var_lib_machines_Apertisv2022dev3ospackTargetARMHF var/lib/machines/Apertisv2022dev3ospackTargetARMHF
snapshot-var_lib_machines_BusterArm64 var/lib/machines/BusterArm64
snapshot-var_lib_machines_DebianBusterTemplate var/lib/machines/DebianBusterTemplate
snapshot-var_lib_machines_DebianJessieTemplate var/lib/machines/DebianJessieTemplate
snapshot-var_lib_machines_DebianSidTemplate var/lib/machines/DebianSidTemplate
snapshot-var_lib_machines_DebianSidTemplate_var_lib_portables var/lib/machines/DebianSidTemplate/var/lib/portables
snapshot-var_lib_machines_DebSidArm64 var/lib/machines/DebSidArm64
snapshot-var_lib_machines_DebSidArmhf var/lib/machines/DebSidArmhf
snapshot-var_lib_machines_DebSidMips var/lib/machines/DebSidMips
snapshot-var_lib_machines_JenkinsApertis var/lib/machines/JenkinsApertis
snapshot-var_lib_machines_v2019 var/lib/machines/v2019
snapshot-var_lib_machines_v2019LinuxSupport var/lib/machines/v2019LinuxSupport
snapshot-var_lib_machines_v2020 var/lib/machines/v2020
snapshot-var_lib_machines_v2021dev3Slim var/lib/machines/v2021dev3Slim
snapshot-var_lib_machines_v2021dev3SlimTarget var/lib/machines/v2021dev3SlimTarget
snapshot-var_lib_machines_v2022dev2OspackMinimal var/lib/machines/v2022dev2OspackMinimal
snapshot-var_lib_portables var/lib/portables
snapshot-var_log var/log
snapshot-var_spool_news var/spool/news
snapshot-var_tmp var/tmp

snapper

Entirely independent of this, but indirectly related. I use snapper as my snapshotting tool. It worked perfect on my previous machine. While everything got migrated, the only thing that fell apart was snapper. It just wouldn’t start/run proper. Funny thing is that I just removed the snapper configs and reinitialized with the exact same config again, and voila snapper was happy.

Conclusion

That was pretty much it. With the above and then also migrating /boot and then just chroot to install the boot loader. At some time, I’d like to explore other boot options but given that that is such a non-essential task, it is low on the list.

The good part was that I booted into my new machine with my exact workstation setup as it was. All the way to the user cache and the desktop session. So it was nice on that part.

But I surely think there’s room for a better migration experience here. If not directly as btrfs migrate, then maybe as an independent tool. The problem is that such a tool is going to be used once in years, so I didn’t find the motivation to write one. But this surely would be a good use case for the distribution vendors.

Setting Up a Secure Webapp

rrs@researchut.com (Ritesh Raj Sarraf) — Sat, 22 May 2021 15:44:48 +0530

As a person who prefers full access to data in the simplest format, while at the same time having it useful with latest technologies, my quest for trying things out is an ongoing activity.

Earlier, I blogged about my needs of collating news feeds in a simple format, readily accessible offline, while still being useful and aligned with the modern paradigm.

In today’s age, the other common aspect of our life, is digitization of moments. With the advent of great technology and affordable economics, the world now has access to great devices to capture moments in digital form. Most people, these days, are equipped with smart devices, like mobile phones, that come with pretty good image capturing devices. Our lives, our societies, how we interact; a lot of it is now built around the assumption of smart devices and digital services.

A lot of good things have happened of it. We are now able to send messages to people, securely, in a matter of seconds. We are now able to capture moments, which otherwise we’d often miss; all thanks to devices like smart mobile phones that most of all carry almost all along with us.

In this regard, we are quite indebted to the technology revolution that has improved life styles all over. Of course, like every coin has a flip side, thus too, too much of obsession with digital life, also has its drawbacks. A simple reminder to self should always be about what a human life is designed for and try to live by its rules.

Moments

Given the tools general availability, we are all more used to capturing moments. Once upon a time, taking a photo meant going to a photo studio.

Then came a generation where we’d have personal devices with roll films, to which we captured the moments. And then get it processed from a photo developing service provider. And that is when we’d know which all moments came out correct and which ones were spoilt. In such cases, there’s always the wish I could have that moment again.

Thankfully, now, in our current generation, we have the liberty to take pictures and validate them almost immediately. We also have the flexibility to take multiple shots of the moment and do the filtering later. There’s also many intelligent and invasive services, all mostly provided free, to help you organize these moments; at a small cost of keeping an eye on your life activity.

In a summary, mankind, now, generates lots and lots of data. So much lot that now even mammoths like Google are forced to make a call whether it is more profitable to sniff user activity while providing them a service for free, or ask people to pay otherwise too.

Privacy

Many of us are wary of the amount of personal data we generate, which is our asset; and how the big tech giants want a piece of it in the name of free services. And in such quest, free aside, there’s always a lookout for privacy savvy tools that can help us draw a bold line in between Public and Private

Google Photos

Google Photos has been a great tool. The way it organizes, siphons and present your data is simply amazing. No good would have been the photos taken, if they weren’t easily searchable, organized, annotated, and presented.

But Google Photos is proprietary and invasive. The amount of information they have access to should be a concern to all people. For services that invade so much, the world needs Free Services.

PhotoPrism

I only came across this project around a month ago, while the project has been around since 2019. So far, in my exploration, this is one of the best Free Software tools to manage and organize my digital photos. The other favorite tool that I regularly depend on is Digikam and it still is a gem.

In a gist, PhotoPrism aims to be the equivalent of Google Photos, for the privacy savvy people.

As of date, it has a decent list of features available. And for some of the missing ones, I’ve come up with a fairly okay workflow with other tools, which is one of the reasons of this blog post.

PhotoPrism is a web app, written in the Go Programming Language. Its layout and workflow as a Photo Management application is similar to Google Photos. It is very performant compared to other applications. Since it is a Progressive Web App, access through a Laptop, Tablet or Mobile is almost the same. It uses Google TensorFlow for some of its features and thus you feel like using Google Photos in some regard.

Some shortcomings and workarounds

Facial Recognition: To date, facial recognition is a planned feature. But this was easy to tackle given that Digikam has pretty solid facial recognition. So I use Digikam to detect, recognize and annotate faces. And then PhotoPrism is happy enough to use those annotations and present relevant data.
HTTP Web App: The upstream project has done a good job of making use of Docker container technology in presenting this web app solution. A software solution, the equivalent of a Google Photos, does need a heavyweight config. In free software, it is all about reusing available tools. PhotoPrism makes use of tools like MySQL, Vue.js, TensorFlow, GoLang, GoLibs and strives to provide a single package solution, all thanks to Docker containers.

In its current offering, PhotoPrism is run as a HTTP Web App. I wanted to have an added layer of security on top of it. And thus run an nginx reverse proxy on top of it. Along with it, I run the proxy service on HTTPS, thus making all traffic from clients to the proxy encrypted. I also wanted an added layer of HTTP AUTH on top, so I explored some options and finally settled down with http-auth-digest

Also, in the current implementation, PhotoPrism doesn’t have a strong notion of normal and private photos in its data organization. I wanted normal photos available under a standard auth realm and private photos under a different realm. And along with a different realm, I also wanted some added security directives for it. So far, it looks like I’ve put together a decent solution with the help of nginx.

First of all, since the port from the host is forwarded to the docker instance, that needed to be controlled. Instead of the default of listening on all interfaces, I changed it to loopback only. Because my primary and only interface is going to be the nginx reverse proxy
Setup nginx with a self-signed certificate to have all communication encrypted.

Setup nginx as a reverse proxy to talk to PhotoPrism.

server {

 listen 80;
 listen [::]:80;
 server_name lenovo;
 return 302 https://$host$request_uri;
 rewrite ^ https://$http_host$request_uri? permanent; # force redirect http to https

 server_tokens off;
}

server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;
ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
server_tokens off;
server_name lenovo;
client_max_body_size 500M;
 auth_digest_expires 900s;
 auth_digest_evasion_time 5s;
 auth_digest_replays 500;

location /private {
 auth_digest 'abc';
 auth_digest_user_file /etc/nginx/htdigest;
 auth_digest_expires 900s;
 auth_digest_evasion_time 5s;
 auth_digest_replays 500;

 proxy_pass http://localhost:2342/private;
}
location /discover {
 auth_digest 'abc';
 auth_digest_user_file /etc/nginx/htdigest;
 auth_digest_expires 900s;
 auth_digest_evasion_time 5s;
 auth_digest_replays 500;

 proxy_pass http://localhost:2342/discover;
}
location / {
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $host;
 proxy_buffering off;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 proxy_pass http://localhost:2342;

 auth_basic "Priv";
 auth_basic_user_file /etc/nginx/htpasswd;
}
}

nginx packaging: I was so happy to see the simplicity of the nginx packaging in Debian. Becuase http-auth-digest is not upstream, it needs to be pulled in separately and compiled. I was happy to see how simple the packaging structure of nginx in Debian is. It was just a matter of putting in the module in the right modules location, which as structured such in the packaging falls under the debian/ packaging sub-folder, that any future upgrades will be quite easy to manage.
http-auth-digest: I’d love to see http-auth-digest module be part of the upstream package. While I’m a web n00b, I felt this module perfect for my use case. From what I’ve understood, set up and tested so far, this module fills in all my requirements; which is more of like a session management.

With the combined setup in place, https://host is authenticated with a different set of credentials. On the other hand, https://host/discover and /private get covered with a different set of credentials and policies. While, this will continue to be an ongoing effort and audit of the services that I build, so far now, I feel this is in decent shape that I can use it as my daily driver.

The end result is:

nginx auth prompt

PhotoPrism Photo List

PhotoPrism Menu

Updates

Things can really get frustrating at times. These days, it is Google that contributes to it. https://bugs.chromium.org/p/chromium/issues/detail?id=544244
- It is is now fair to say that Google is big enough to rewrite all the standards. Or at least break them. Or best, their arrogance of it. This bug report is a fine example of what other ways it could have been dealt.

Wayland KDE X11

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 26 Feb 2021 17:08:26 +0530

KDE Impressions

These days, I often hear a lot about Wayland. And how much of effort is being put into it; not just by the Embedded world but also the usual Desktop systems, namely KDE and GNOME.

In recent past, I switched back to KDE and have been (very) happy about the switch. Even though the KDE 4 (and initial KDE 5) debacle had burnt many, coming back to a usable KDE desktop is always a delight. It makes me feel home with the elegance, while at the same time the flexibility, it provides. It feels so nice to draft this blog article from Kwrite + VI Input Mode

Thanks to the great work of the Debian KDE Team, but Norbert Preining in particular, who has helped bring very up-to-date KDE packages into Debian. Right now, I’m on a Plamsa 5.21.1 desktop, which is recent by all standards.

Wayland

Almost all the places in the Linux world these days are busy with integrating Wayland as the primary display service. Not sure what the current status on the GNOME side is but I definitely keep trying KDE + Wayland with every release.

I keep trying with every release because it still is not prime for daily use. And it makes me get back to X11, no matter how dated some may call. Fact is, X11 still shines to me as an end-user.

Glitches with Wayland still are (Based on this week’s test on Plasma 5.21.1):

Horrible performance compared to X11
Very crashy, especially when hotplugging secondary display. Plasma would just crash. X11 is very resilient to such things, part of the reason I can think is the age of the codebase.
Many many applications still need to be fixed for Wayland. Or Wayland needs to accomodate them in some way. XWayland does not really feel like the answer.

And while KDE keeps insisting users to switch to Wayland, as that’s where all the new enhancements and fixes are put in, someone like me still needs to stick to X11 for the time being. So to get my shiny new LG 27" 4K Monitor (3840x2160 60.00*+) to work without too much glitch, I had to live with an alias:

$ alias | grep xrandr
alias rrs_xrandr_lg='xrandr --output DP-1 --mode 3840x2160 --scale .75x.75'
18:31 ♒ ॐ ♅ ♄ ⛢ ☺ 😄

Plasma 5.21

On the brighter side, the Plasma 5.21.1 release brings some nice enhancements in other areas.

I’m now able to make use of tighter integration with systemd/cgroups, with better organization and management of processes overall.
The new Plasma theme, Breeze Twilight, is a good blend of Light + Dark.

I also appreciate the work put in by Michail Vourlakos. The KDE project is lucky to have a developer/designer like him. His vision and work into the KDE desktop is well beyond a writing by me.

$ usystemctl status plasma-plasmashell.service
● plasma-plasmashell.service - KDE Plasma Workspace
 Loaded: loaded (/usr/lib/systemd/user/plasma-plasmashell.service; enabled; vendor preset: enabled)
 Active: active (running) since Fri 2021-02-26 18:34:23 IST; 13s ago
 Main PID: 501806 (plasmashell)
 Tasks: 21 (limit: 18821)
 Memory: 759.8M
 CPU: 13.706s
 CGroup: /user.slice/user-1000.slice/user@1000.service/session.slice/plasma-plasmashell.service
 └─501806 /usr/bin/plasmashell --no-respawn

Feb 26 18:35:00 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:21 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:49 priyasi plasmashell[501806]: qml: recreating buttons
Feb 26 18:35:57 priyasi plasmashell[501806]: qml: recreating buttons
18:36 ♒ ॐ ♅ ♄ ⛢ ☺ 😄

OBS - Open Build Service

I should also thank the OpenSUSE folks for the OBS work. It has enabled the close equivalent (or better, in my experience) of PPAs for Debian. And that is what has enabled developers like Norbert to easily and quickly be able to deliver the entire KDE suite.

OBS - Some detail

Christian asked for some more details on the OBS side of things, of my view. I’m updating this article with it because the comment system may not always be reliable and I hate losing content.

Having been using OBS myself, and also others in the Debian community who are making use of it, I surely think we as project should consider making use of OBS.

Given that OBS is Free Software, it is a perfect fit for Debian. Gitlab is another example of what we’ve made available in Debian.

OBS is divided into multiple parts

OBS Server
OBS DoD service
OBS Publisher
OBS Workers
OBS Warden
OBS Rep Server

For every Debian release I care about, I add an OBS project per release. So I have OBS projects for: Sid, Bullseye, Buster, Jessie.

Now, say you have a package, foo . You prep your package and enable all the releases that you want to build the package for. So the same package gets built, in separate clean environments, for every release I mentioned as an example above. You don’t have to manually trigger the build for every release/every architcture. You should add the release (as projects) in OBS, set their supported architectures, and then add those enabled release projets as bits to your package.

Every build involves:

Creating a new chroot for each build
Building the package

Builds can be scattered across multiple hosts, known as workers in OBS terminology. Your workers are independent machine entities, supporting different architectures. The machines can be Bare-Metal ones, VMs, even containers. So this allows for very nice scale-in and scale-out. There may be auto-scaling too but that is something worth investigating.

Think of things like cross architecture builds. Let’s assume the cloud vendors decide to donate resources to the Debian project. We could enable OBS worker instances on the respective clouds (different architectures) and plug them into the master OBS instance that Debian hosts. Fully distributed. Similarly, big hardware vendors willing to donate compute resources could house them in their premises and Debian could just easily establish a connection to them. All of this just a TCP connection away.

So when I look at the features of OBS, from the point of view of Debian, I like it more. Extensibility won’t be an issue. Supporting a new Debian release would just be a matter of bootstrapping the Debian release as a project in OBS, and then all done. The single effort of setting of the target release project is a one time job, and then all can leverage it.

The PPA was a long craved feature missing in Debian, in my opinion. OBS allows to not just fulfil that gap but also extend it in a very easy way.

Andrew Lee had put in a nice video presentation about the same @ Debconf 20

Your browser does not support the video tag.

LUKS Headless Laptop

rrs@researchut.com (Ritesh Raj Sarraf) — Thu, 20 Aug 2020 19:29:14 +0530

As we grow old, so do our computing machines. And just like we don’t decommission ourselves, so should be the case of the machines. They should be semi-retired, delegating major tasks to newer machines while they can still serve some less demaning work: File Servers, UPNP Servers et cetera.

It is common on a Debian installer based derivative, and otherwise too, to use block encryption on Linux. With machines from this decade, I think we’ve always had CPU extension for encryption.

So, as would be the usual case, all my laptops are block encrypted. But as they reach the phase of their life to retire and serving as a headless boss, it becomes cumbersome to keep feeding it a password and all the logistics involved to feed it. As such, I wanted to get rid of feeding it the password.

Then, there’s also the case of bad/faulty hardware, many of which mostly can temporarily fix their functionality when reset, which usually is to reboot the machine. I still recollect words of my Linux Guru - Dhiren Raj Bhandari - that many of the unexplainable errors can be resolved by just rebooting the machine. This was more than 20 years ago in the prime era of Microsoft Windows OS and the context back then was quite different, but yes, some bits of that saying still apply today.

So I wanted my laptop, which had LUKS set up for 2 disks, to go password-less now. I stumbled across a slightly dated article where the author achieved similar results with keyscript. So the thing was doable.

To my delight, Debian cryptsetup has the best setup and documentation in place to do it with just adding keyfiles

rrs@lenovo:~$ dd if=/dev/random of=sda7.key bs=1 count=512
512+0 records in
512+0 records out
512 bytes copied, 0.00540209 s, 94.8 kB/s
19:19 ♒♒♒ ☺ 😄

rrs@lenovo:~$ dd if=/dev/random of=sdb1.key bs=1 count=512
512+0 records in
512+0 records out
512 bytes copied, 0.00536747 s, 95.4 kB/s
19:20 ♒♒♒ ☺ 😄

rrs@lenovo:~$ sudo cryptsetup luksAddKey /dev/sda7 sda7.key
[sudo] password for rrs:
Enter any existing passphrase:
No key available with this passphrase.
19:20 ♒♒♒ ☹ 😟=> 2

rrs@lenovo:~$ sudo cryptsetup luksAddKey /dev/sda7 sda7.key
Enter any existing passphrase:
19:20 ♒♒♒ ☺ 😄

rrs@lenovo:~$ sudo cryptsetup luksAddKey /dev/sdb1 sdb1.key
Enter any existing passphrase:
19:21 ♒♒♒ ☺ 😄

and the nice integration in crypttab to ensure your keys propagate to initramfs

rrs@lenovo:~$ cat /etc/cryptsetup-initramfs/conf-hook
#
# Configuration file for the cryptroot initramfs hook.
#

#
# KEYFILE_PATTERN: ...
#
# The value of this variable is interpreted as a shell pattern.
# Matching key files from the crypttab(5) are included in the initramfs
# image. The associated devices can then be unlocked without manual
# intervention. (For instance if /etc/crypttab lists two key files
# /etc/keys/{root,swap}.key, you can set KEYFILE_PATTERN="/etc/keys/*.key"
# to add them to the initrd.)
#
# If KEYFILE_PATTERN if null or unset (default) then no key file is
# copied to the initramfs image.
#
# Note that the glob(7) is not expanded for crypttab(5) entries with a
# 'keyscript=' option. In that case, the field is not treated as a file
# name but given as argument to the keyscript.
#
# WARNING: If the initramfs image is to include private key material,
# you'll want to create it with a restrictive umask in order to keep
# non-privileged users at bay. For instance, set UMASK=0077 in
# /etc/initramfs-tools/initramfs.conf
#

KEYFILE_PATTERN="/etc/luks/sd*.key"
19:44 ♒♒♒ ☺ 😄

The whole thing took me around 20-25 minutes, including drafting this post. From Retired Head and Password Prompt to Headless and Password-less. The beauty of Debian and FOSS

Laptop Mode Tools 1.74

rrs@researchut.com (Ritesh Raj Sarraf) — Sat, 18 Jul 2020 21:15:04 +0530

Laptop Mode Tools 1.74

Laptop Mode Tools version 1.74 has been released. This release includes important bug fixes, some defaults settings updated to current driver support in Linux and support for devices with nouveau based nVIDIA cards.

A filtered list of changes is mentioned below. For the full log, please refer to the git repository

1.74 - Sat Jul 18 19:10:40 IST 2020

* With 4.15+ kernels, Linux Intel SATA has a better link power
saving policy, med_power_with_dipm, which should be the recommended
one to use
* Disable defaults for syslog logging
* Initialize LM_VERBOSE with default to disabled
* Merge pull request #157 from rickysarraf/nouveau
* Add power saving module for nouveau cards
* Disable ethernet module by default
* Add board-specific folder and documentation
* Add execute bit on module radeon-dpm
* Drop unlock because there is no lock acquired

Resources

Source tarball, Feodra/SUSE RPM Packages available at project’s release page
Debian packages will be available soon in Unstable.
Homepage: https://github.com/rickysarraf/laptop-mode-tools/wiki
Mailing List: https://groups.google.com/d/forum/laptop-mode-tools

What is Laptop Mode Tools

Description: Tools for Power Savings based on battery/AC status
Laptop mode is a Linux kernel feature that allows your laptop to save
considerable power, by allowing the hard drive to spin down for longer
periods of time. This package contains the userland scripts that are
needed to enable laptop mode.
.
It includes support for automatically enabling laptop mode when the
computer is working on batteries. It also supports various other power
management features, such as starting and stopping daemons depending on
power mode, automatically hibernating if battery levels are too low, and
adjusting terminal blanking and X11 screen blanking
.
laptop-mode-tools uses the Linux kernel's Laptop Mode feature and thus
is also used on Desktops and Servers to conserve power

Kodi PS3 BD Remote

rrs@researchut.com (Ritesh Raj Sarraf) — Tue, 16 Jun 2020 11:08:13 +0530

Setting up a Sony PS3 Blu-Ray Disc Remote Controller with Kodi

TLDR; Since most of the articles on the internet were either obsolete or broken, I’ve chosen to write these notes down in the form of a blog post so that it helps me now and in future, and hopefully others too.

Raspberry Pi

All this time, I have been using the Raspberry Pi for my HTPC needs. The first RPi I acquired was in 2014 and I have been very very happy with the amount of support in the community and quality of the HTPC offering it has. I also appreciate the RPi’s form factor and the power consumption limits. And then, to add more sugar to it, it uses a derivative of Debian, Raspbian, which was very familiar and feel good to me.

Raspberry Pi Issues

So primarily, I use my RPi with Kodi. There are a bunch of other (daemon) services but the primary use case is HTPC only. RPi + Kodi has a very very annoying issue wherein it loses its audio pitch during video playback. The loss is so bad that the audio is barely audible. The workaround is to seek the video playback either way and then it comes back to its actual audio level, just to fade again in a while.

My suspicion was that it may be a problem with Kodi. Or at least, Kodi would have a workaround in software. But unfortunately, I wasted a lot of time in dealing with my suspicion with no fruitful result.

This started becoming a PITA over time. And it seems the issue is with the hardware itself because after I moved my setup to a regular laptop, the audio loss is gone.

Laptop with Kodi

Since I had my old Lenovo Yoga 2 13 lying on all the time, it made sense to make some more use of it, using as the HTPC. This machine comes with a Micro-HDMI Out port, so it felt ideal for my High Definition video rendering needs.

It comes stock with just Intel HD Video with good driver support in Linux, so it was quite quick and easy getting Kodi charged up and running on it. And as I mentioned above, the sound issues are not seen on this setup.

Some added benefits are that I get to run stock Debian on this machine. And I must say a big THANK YOU to the Debian Multimedia Maintainers, who’ve done a pretty good job maintaining Kodi under Debian.

HDMI CEC

Only after I decommissioned my RPi, I came to notice how convenient the HDMI CEC functionality is. Turns out no standard laptops ship CEC functionality onto them. Even the case of my laptop, which has a Micro HDMI Out port, but still no CEC capabilities. As far as I know, the RPi came with the Pulse-Eight CEC module, so obvious first thought was to opt for a compatible external module of the same; but it comes with a nice price tag, me not willing to spend.

WiFi Remotes

Kodi has very well implemented network interface for almost all its features. One could take the Yatse or Music Pump Kodi Remote Android applications that work very very well with Kodi.

But wifi can be flaky some times. Especially, my experience with the Realtek network devices hasn’t been very good. The driver support in Linux is okay but there are many firmware bugs to deal with. In my case, the machine will lose wifi signal/network every once in a while. And it turns out, for this machine, with this network device type, I’m not the only one running into such problems.

And to add to that, this is an UltraBook, which means it doesn’t have an Ethernet port. So I’ve had not much choice other than to live and ~~suffer~~ deal with it.

The WiFi chip also provides the Bluetooth module, which so far I had not used much. From my /etc/modprobe.d/blacklist-memstick.conf, all relevant BT modules were added to the blacklist, all this time.

rrs@lenovo:~$ cat /etc/modprobe.d/blacklist-memstick.conf
blacklist memstick
blacklist rtsx_usb_ms
# And bluetooth too
#blacklist btusb
#blacklist btrtl
#blacklist btbcm
#blacklist btintel
#blacklist bluetooth
21:21 ♒♒♒ ☺ 😄

Also to keep in mind is that the driver for my card gives a very misleading kernel message, which is one of the many reasons for this blog post, so that I don’t forget it a couple of months later. The missing firmware error message is okay to ignore, as per this upstream comment.

Jun 14 17:17:08 lenovo kernel: usbcore: registered new interface driver btusb
Jun 14 17:17:08 lenovo systemd[1]: Mounted /boot/efi.
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: examining hci_ver=06 hci_rev=000b lmp_ver=06 lmp_subver=8723
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: rom_version status=0 version=1
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8723b_fw.bin
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: firmware: direct-loading firmware rtl_bt/rtl8723b_fw.bin
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8723b_config.bin
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: firmware: failed to load rtl_bt/rtl8723b_config.bin (-2)
Jun 14 17:17:08 lenovo kernel: firmware_class: See https://wiki.debian.org/Firmware for information about missing firmware
Jun 14 17:17:08 lenovo kernel: bluetooth hci0: Direct firmware load for rtl_bt/rtl8723b_config.bin failed with error -2
Jun 14 17:17:08 lenovo kernel: Bluetooth: hci0: RTL: cfg_sz -2, total sz 22496

This device’s network + bt are on the same chip.

01:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8723BE PCIe Wireless Network Adapter

And then, when the btusb module is initialed (along with the misleading driver message), you’ll get the following in your USB device listing

Bus 002 Device 005: ID 0bda:b728 Realtek Semiconductor Corp. Bluetooth Radio

Sony PlayStation 3 BD Remote

Almost 10 years ago, I bought the PS3 and many of its accessories. The remote has just been rotting in the shelf. It had rusted so bad that it is better described with these pics.

Rusted inside

Rusted inside and cover

Rusted spring

The rust was so much that the battery holding spring gave up.

A little bit scrubbing and cleaning has gotten it working. I hope it lasts for some time before I find time to open it up and give it a full clean-up.

Pairing the BD Remote to laptop

Honestly, with the condition of the hardware and software on both ends, I did not have much hopes of getting this to work. And in all the years on my computer usage, I hardly recollect much days when I’ve made use of BT. Probably, because the full BT stack wasn’t that well integrated in Linux, earlier. And I mostly used to disable them in hardware and software to save on battery.

All yielded results from the internet talked about tools/scripts that were either not working, pointing to broken links etc.

These days, bluez comes with a nice utility, bluetoothctl. It was a nice experience using it.

First, start your bluetooth service and ensure that the device talks well with the kernel

rrs@lenovo:~$ systemctl status bluetooth
● bluetooth.service - Bluetooth service
Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2020-06-15 12:54:58 IST; 3s ago
Docs: man:bluetoothd(8)
Main PID: 310197 (bluetoothd)
Status: "Running"
Tasks: 1 (limit: 9424)
Memory: 1.3M
CGroup: /system.slice/bluetooth.service
└─310197 /usr/lib/bluetooth/bluetoothd
Jun 15 12:54:58 lenovo systemd[1]: Starting Bluetooth service...
Jun 15 12:54:58 lenovo bluetoothd[310197]: Bluetooth daemon 5.50
Jun 15 12:54:58 lenovo systemd[1]: Started Bluetooth service.
Jun 15 12:54:58 lenovo bluetoothd[310197]: Starting SDP server
Jun 15 12:54:58 lenovo bluetoothd[310197]: Bluetooth management interface 1.15 initialized
Jun 15 12:54:58 lenovo bluetoothd[310197]: Sap driver initialization failed.
Jun 15 12:54:58 lenovo bluetoothd[310197]: sap-server: Operation not permitted (1)
12:55 ♒♒♒ ☺ 😄

Next, then is to discover and connect to your device

rrs@lenovo:~$ bluetoothctl
Agent registered
[bluetooth]# devices
Device E6:3A:32:A4:31:8F MI Band 2
Device D4:B8:FF:43:AB:47 MI RC
Device 00:1E:3D:10:29:0F BD Remote Control
[CHG] Device 00:1E:3D:10:29:0F Connected: yes
[BD Remote Control]# info 00:1E:3D:10:29:0F
Device 00:1E:3D:10:29:0F (public)
Name: BD Remote Control
Alias: BD Remote Control
Class: 0x0000250c
Paired: no
Trusted: yes
Blocked: no
Connected: yes
LegacyPairing: no
UUID: Human Interface Device... (00001124-0000-1000-8000-00805f9b34fb)
UUID: PnP Information (00001200-0000-1000-8000-00805f9b34fb)
Modalias: usb:v054Cp0306d0100
[bluetooth]#

In case of the Sony BD Remote, there’s no need to pair. In fact, trying to pair fails. It prompts for the PIN code, but neither 0000 or 1234 are accepted.

So, the working steps so far are to Trust the device and then Connect the device.

For the sake of future use, I also populated /etc/bluetooth/input.conf based on suggestions on the internet. Note: The advertised keymappings in this config file do not work. Note: I’m only using it for the power saving measures in instructing the BT connection to sleep after 3 minutes.

rrs@priyasi:/tmp$ cat input.conf
# Configuration file for the input service
# This section contains options which are not specific to any
# particular interface
[General]
# Set idle timeout (in minutes) before the connection will
# be disconnect (defaults to 0 for no timeout)
IdleTimeout=3
# Enable HID protocol handling in userspace input profile
# Defaults to false (HIDP handled in HIDP kernel module)
#UserspaceHID=true
# Limit HID connections to bonded devices
# The HID Profile does not specify that devices must be bonded, however some
# platforms may want to make sure that input connections only come from bonded
# device connections. Several older mice have been known for not supporting
# pairing/encryption.
# Defaults to false to maximize device compatibility.
#ClassicBondedOnly=true
# LE upgrade security
# Enables upgrades of security automatically if required.
# Defaults to true to maximize device compatibility.
#LEAutoSecurity=true
#
#[00:1E:3D:10:29:0F]
[2c:33:7a:8e:d6:30]
[PS3 Remote Map]
# When the 'OverlayBuiltin' option is TRUE (the default), the keymap uses
# the built-in keymap as a starting point. When FALSE, an empty keymap is
# the starting point.
#OverlayBuiltin = TRUE
#buttoncode = keypress # Button label = action with default key mappings
#OverlayBuiltin = FALSE
0x16 = KEY_ESC # EJECT = exit
0x64 = KEY_MINUS # AUDIO = cycle audio tracks
0x65 = KEY_W # ANGLE = cycle zoom mode
0x63 = KEY_T # SUBTITLE = toggle subtitles
0x0f = KEY_DELETE # CLEAR = delete key
0x28 = KEY_F8 # /TIME = toggle through sleep
0x00 = KEY_1 # NUM-1
0x01 = KEY_2 # NUM-2
0x02 = KEY_3 # NUM-3
0x03 = KEY_4 # NUM-4
0x04 = KEY_5 # NUM-5
0x05 = KEY_6 # NUM-6
0x06 = KEY_7 # NUM-7
0x07 = KEY_8 # NUM-8
0x08 = KEY_9 # NUM-9
0x09 = KEY_0 # NUM-0
0x81 = KEY_F2 # RED = red
0x82 = KEY_F3 # GREEN = green
0x80 = KEY_F4 # BLUE = blue
0x83 = KEY_F5 # YELLOW = yellow
0x70 = KEY_I # DISPLAY = show information
0x1a = KEY_S # TOP MENU = show guide
0x40 = KEY_M # POP UP/MENU = menu
0x0e = KEY_ESC # RETURN = back/escape/cancel
0x5c = KEY_R # TRIANGLE/OPTIONS = cycle through recording options
0x5d = KEY_ESC # CIRCLE/BACK = back/escape/cancel
0x5f = KEY_A # SQUARE/VIEW = Adjust Playback timestretch
0x5e = KEY_ENTER # CROSS = select
0x54 = KEY_UP # UP = Up/Skip forward 10 minutes
0x56 = KEY_DOWN # DOWN = Down/Skip back 10 minutes
0x57 = KEY_LEFT # LEFT = Left/Skip back 5 seconds
0x55 = KEY_RIGHT # RIGHT = Right/Skip forward 30 seconds
0x0b = KEY_ENTER # ENTER = select
0x5a = KEY_F10 # L1 = volume down
0x58 = KEY_J # L2 = decrease the play speed
0x51 = KEY_HOME # L3 = commercial skip previous
0x5b = KEY_F11 # R1 = volume up
0x59 = KEY_U # R2 = increase the play speed
0x52 = KEY_END # R3 = commercial skip next
0x43 = KEY_F9 # PS button = mute
0x50 = KEY_M # SELECT = menu (as per PS convention)
0x53 = KEY_ENTER # START = select / Enter (matches terminology in mythwelcome)
0x30 = KEY_PAGEUP # PREV = jump back (default 10 minutes)
0x76 = KEY_J # INSTANT BACK (newer RCs only) = decrease the play speed
0x75 = KEY_U # INSTANT FORWARD (newer RCs only) = increase the play speed
0x31 = KEY_PAGEDOWN # NEXT = jump forward (default 10 minutes)
0x33 = KEY_COMMA # SCAN BACK = decrease scan forward speed / play
0x32 = KEY_P # PLAY = play/pause
0x34 = KEY_DOT # SCAN FORWARD decrease scan backard speed / increase playback speed; 3x, 5, 10, 20, 30, 60, 120, 180
0x60 = KEY_LEFT # FRAMEBACK = Left/Skip back 5 seconds/rewind one frame
0x39 = KEY_P # PAUSE = play/pause
0x38 = KEY_P # STOP = play/pause
0x61 = KEY_RIGHT # FRAMEFORWARD = Right/Skip forward 30 seconds/advance one frame
0xff = KEY_MAX
21:48 ♒ ॐ ♅ ⛢ ☺ 😄

I have not spent much time finding out why not all the key presses work. Especially, given that most places on the internet mention these mappings. For me, some of the key scan codes aren’t even reported. For keys like L1, L2, L3, R1, R2, R3, Next_Item, Prev_Item, they generate no codes in the kernel.

If anyone has suggestions, ideas or fixes, I’d appreciate if you can drop a comment or email me privately.

But given my limited use to get a simple remote ready, to be usable with Kodi, I was apt with only some of the keys working.

Mapping the keys in Kodi

With the limited number of keys detected, mapping those keys to what Kodi could use was the next step. Kodi has a very nice and easy to use module, Keymap Editor. It is very simple to use and map detected keys to functionalities you want. With it, I was able to get a functioning remote to use with my Kodi HTPC setup.

Update: Wed Jun 17 11:38:20 2020

One annoying problem that breaks the overall experience is the following bug on the driver side, that results in connections not being established instantly.

Once the device goes into sleep mode, in random attempts, waking up and re-establishing a BT connection can be multi-poll affair. This can last from a couple of seconds to well over minute.

Random suggestions on the internet mention disabling the autosuspend functionality for the device in the driver with btusb.enable_autosuspend=n, but that did not help in this case.

Given that this device is enumberated over the USB Bus, it probably needs this feature applied to the whole USB tree of the device’s chain. Something to investigate over the weekend.

Jun 16 20:41:23 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 7
Jun 16 20:41:43 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 8
Jun 16 20:41:59 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 9
Jun 16 20:42:18 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:10/0005:054C:030>
Jun 16 20:42:18 lenovo kernel: sony 0005:054C:0306.0006: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30
Jun 16 20:51:59 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:11/0005:054C:030>
Jun 16 20:51:59 lenovo kernel: sony 0005:054C:0306.0007: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Supervising 3 threads of 1 processes of 1 users.
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Successfully made thread 32747 of process 1646 owned by '1000' RT at priority 5.
Jun 16 21:05:55 lenovo rtkit-daemon[1723]: Supervising 4 threads of 1 processes of 1 users.
Jun 16 21:05:56 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 12
Jun 16 21:06:12 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 1
Jun 16 21:06:34 lenovo kernel: Bluetooth: hci0: ACL packet for unknown connection handle 2
Jun 16 21:06:59 lenovo kernel: input: BD Remote Control as /devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0/hci0:3/0005:054C:0306>
Jun 16 21:06:59 lenovo kernel: sony 0005:054C:0306.0008: input,hidraw1: BLUETOOTH HID v1.00 Gamepad [BD Remote Control] on 2c:33:7a:8e:d6:30

Others

There’s a package, kodi-eventclients-ps3, which can be used to talk to the BD Remote. Unfortunately, it isn’t up-to-date. When trying to make use of it, I ran into a couple of problems.

First, the easy one is:

rrs@lenovo:~/ps3pair$ kodi-ps3remote localhost 9777
usr/share/pixmaps/kodi//bluetooth.png
Traceback (most recent call last):
File "/usr/bin/kodi-ps3remote", line 220, in <module>
File "/usr/bin/kodi-ps3remote", line 208, in main
xbmc.connect(host, port)
packet = PacketHELO(self.name, self.icon_type, self.icon_file)
File "/usr/lib/python3/dist-packages/kodi/xbmcclient.py", line 285, in __init__
with open(icon_file, 'rb') as f:
11:16 ♒♒♒ ☹ 😟=> 1

This one was simple as it was just a broken path.

The second issue with the tool is a leftover from python2 to python3 conversion.

rrs@lenovo:/etc/bluetooth$ kodi-ps3remote localhost
/usr/share/pixmaps/kodi//bluetooth.png
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
Redmi (E8:5A:8B:73:57:44) in range
Living Room TV (E4:DB:6D:24:23:E9) in range
Could not find BD Remote Control. Trying again...
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
Living Room TV (E4:DB:6D:24:23:E9) in range
Redmi (E8:5A:8B:73:57:44) in range
Could not find BD Remote Control. Trying again...
Searching for BD Remote Control
(Hold Start + Enter on remote to make it discoverable)
BD Remote Control (00:1E:3D:10:29:0F) in range
Found BD Remote Control with address 00:1E:3D:10:29:0F
Attempting to pair with remote
Remote Paired.
Traceback (most recent call last):
File "/usr/bin/kodi-ps3remote", line 221, in <module>
main()
File "/usr/bin/kodi-ps3remote", line 212, in main
if process_keys(remote, xbmc):
File "/usr/bin/kodi-ps3remote", line 164, in process_keys
keycode = data.encode("hex")[10:12]
AttributeError: 'bytes' object has no attribute 'encode'
11:24 ♒♒♒ ☹ 😟=> 1

Fixing that too did not give me the desired result on using the BD Remote in the way I want. So eventually, I gave up and used Kodi’s Keymap Editor instead.

Next in line, when I can manage to get some free time, is to improve the Kodi Video Scraper to have a fallback mode. Currently, for files where it cannot determine the content, it reject the file resulting in those files not showing up in your collection at all. A better approach would have been to have a fallback mode, that when the scraper cannot determine the content, it should fallback to using the filename scraper

Apt Offline 1.8.2

rrs@researchut.com (Ritesh Raj Sarraf) — Sun, 09 Feb 2020 19:52:51 +0530

apt-offline 1.8.2

I am pleased to announce the release of apt-offline version 1.8.2

This release has many bug fixes along with a long standing issue of signature validation

2017 - The year of realization and change

Back in 2017, the bug was reported that apt-offline did not validate apt meta Packages files. apt-offline was only doing a signature validation for the Release files but did no validation of the apt meta Packages files, which had their checksums listed in the Release files. This validation was completely missing in apt-offline and gave the user the wrong impression that validation was in place.

I had hoped to fix this issue soon when it was reported, to have it part of the next Debian Stable release. But that never happened. On the contrary, I think 2 stable releases happened in between. And now it is 2020.

2017 was a year to spend a large chunk of my time on real life issues, for good. I realized that it is important to always give precedence to personal life, fix issues, set realistic priorities, spend time on realizing the happenings around, get life rolling smooth and then come back to work. This helps sustain in the longer run. Otherwise, with no self, everything can fall apart catastrophically.

From that phase, I learned many things. I now have much more respect for people who really have been successful at committing a large amount of their time on a volunteer project like Debian. Having myself gone through the time crunch phase, I can only imagine how many of the fellow DDs manage their time, sustainably, over the years. There are many folks I have seen active for more than a decade and they still rock.

1.8.2 release

Because the apt meta validation was a major issue, I have decided to run through the workflow and explain how apt-offline reacts to invalid tampered data. Below are konsole captures, with snipped output, where not very relevant.

rrs@priyasi:/var/tmp/Debian-Build/Result$ sudo apt-offline set /tmp/set.uris
Gathering details needed for 'update' operation
Gathering details needed for 'upgrade' operation
16:33 ♒ ॐ ☺ 😄

One item to pay attention to, in this step, is some of the errors that are reported. Not all repository admins enable all the apt meta data available on their mirrors. This is commonly seen for localization related files. Similarly, not all compression types are available on all the repository servers. Some may only have .xz based meta files hosted while others may have .gz ones. So, for apt-offline, which has to bridge the gap of the offline <=> online setup, there is more work.

For compression types, apt-offline cycles through the known list of types. Only if, after cycling through all the known compression types, if the return is still a 404, then we error out.

Similarly, for localization related meta, we do the same cycling. But in addition to that, there is the possibility that the repository admin may not have enabled the localization data to be served at all. In that case, apt-offline ultimately will report and error.

And that is what is shown below. Because I see them not breaking the functionality, I treat them as non-fatal errors.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94


rrs@priyasi:/var/tmp/Debian-Build/Result$ apt-offline get /tmp/set-trimmed.uris --bundle /tmp/set.zip --threads 5

Fetching APT Data

WARNING: If you are on a slow connection, it is good to
WARNING: limit the number of threads to a low number like 2.
WARNING: Else higher number of threads executed could cause
WARNING: network congestion and timeouts.

Downloading http://deb.debian.org/debian/dists/testing/Release.gpg
Downloading http://deb.debian.org/debian/dists/testing/Release
Downloading http://deb.debian.org/debian/dists/testing/InRelease
Downloading http://deb.debian.org/debian/dists/unstable/Release.gpg
Downloading http://deb.debian.org/debian/dists/unstable/Release
http://deb.debian.org/debian/dists/unstable/Release.gpg done
Downloading http://deb.debian.org/debian/dists/unstable/InRelease
http://deb.debian.org/debian/dists/testing/Release.gpg done
Downloading http://deb.debian.org/debian/dists/experimental/Release.gpg
http://deb.debian.org/debian/dists/unstable/Release done
Downloading http://deb.debian.org/debian/dists/experimental/Release
http://deb.debian.org/debian/dists/testing/InRelease done
Downloading http://deb.debian.org/debian/dists/experimental/InRelease
http://deb.debian.org/debian/dists/testing/Release done
Downloading http://deb.debian.org/debian/dists/testing/main/source/Sources.xz
http://deb.debian.org/debian/dists/unstable/InRelease done
Downloading http://deb.debian.org/debian/dists/testing/non-free/source/Sources.xz
http://deb.debian.org/debian/dists/experimental/Release.gpg done
Downloading http://deb.debian.org/debian/dists/testing/contrib/source/Sources.xz
http://deb.debian.org/debian/dists/experimental/InRelease done
Downloading http://deb.debian.org/debian/dists/testing/main/binary-amd64/Packages.xz
http://deb.debian.org/debian/dists/experimental/Release done
Downloading http://deb.debian.org/debian/dists/testing/main/binary-i386/Packages.xz
http://deb.debian.org/debian/dists/testing/contrib/source/Sources.xz done
Downloading http://deb.debian.org/debian/dists/testing/main/binary-all/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/source/Sources.xz done
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_IN.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en.xz
http://deb.debian.org/debian/dists/testing/main/binary-all/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_US.xz
http://deb.debian.org/debian/dists/testing/main/source/Sources.xz done
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-amd64.xz
http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en.bz2 done
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-i386.xz
Downloading http://deb.debian.org/debian/dists/testing/main/Contents-all.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/main/Contents-all.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-amd64/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-i386/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-i386/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/non-free/binary-all/Packages.xz
http://deb.debian.org/debian/dists/testing/non-free/binary-all/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_IN.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en.xz
http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en.bz2 done
Downloading http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_US.xz
http://deb.debian.org/debian/dists/testing/main/binary-i386/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-amd64.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-i386.xz
http://deb.debian.org/debian/dists/testing/non-free/Contents-i386.gz done
Downloading http://deb.debian.org/debian/dists/testing/non-free/Contents-all.xz
http://deb.debian.org/debian/dists/testing/non-free/Contents-amd64.gz done
http://deb.debian.org/debian/dists/testing/main/binary-amd64/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.xz
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-i386/Packages.xz
http://deb.debian.org/debian/dists/testing/contrib/binary-amd64/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/contrib/binary-all/Packages.xz
http://deb.debian.org/debian/dists/testing/contrib/binary-i386/Packages.xz done
http://deb.debian.org/debian/dists/testing/contrib/binary-all/Packages.xz done
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_IN.xz
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/non-free/Contents-all.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_US.xz
http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en.bz2 done
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-amd64.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_IN.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-i386.xz
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/i18n/Translation-en_US.lzma
Downloading http://deb.debian.org/debian/dists/testing/contrib/Contents-all.xz
http://deb.debian.org/debian/dists/testing/contrib/Contents-amd64.gz done
http://deb.debian.org/debian/dists/testing/contrib/Contents-i386.gz done
ERROR: Giving up on URL http://deb.debian.org/debian/dists/testing/contrib/Contents-all.lzma
http://deb.debian.org/debian/dists/testing/main/Contents-i386.gz done
http://deb.debian.org/debian/dists/testing/main/Contents-amd64.gz done
 81 / 81 items: [##############################] 100.0% of 101 MiB
Downloaded data to /tmp/set.zip
ERROR: Some items failed to download. Downloaded data may be incomplete
ERROR: Please run in verbose mode to see details about failed items


16:38 ♒ ॐ ☹ 😟=> 100 

First, lets unpack the archive file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56


rrs@priyasi:/var/tmp/Debian-Build/Result$ cd /tmp/
16:39 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp$ mkdir set-folder
16:39 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp$ cd set-folder/
16:39 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp/set-folder$ unzip ../set.zip
Archive: ../set.zip
 inflating: deb.debian.org_debian_dists_unstable_Release.gpg
 inflating: deb.debian.org_debian_dists_testing_Release.gpg
 inflating: deb.debian.org_debian_dists_unstable_Release
 inflating: deb.debian.org_debian_dists_testing_InRelease
 inflating: deb.debian.org_debian_dists_testing_Release
 inflating: deb.debian.org_debian_dists_unstable_InRelease
 inflating: deb.debian.org_debian_dists_experimental_Release.gpg
 inflating: deb.debian.org_debian_dists_experimental_InRelease
 inflating: deb.debian.org_debian_dists_experimental_Release
 inflating: deb.debian.org_debian_dists_testing_contrib_source_Sources.xz
 inflating: deb.debian.org_debian_dists_testing_non-free_source_Sources.xz
 inflating: deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_main_source_Sources.xz
 inflating: deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2
 inflating: deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2
 inflating: deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz
 inflating: deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz
 inflating: deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz
 inflating: deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2
 inflating: deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz
 inflating: deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz
 inflating: deb.debian.org_debian_dists_testing_main_Contents-i386.gz
 inflating: deb.debian.org_debian_dists_testing_main_Contents-amd64.gz
16:39 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp/set-folder$ ls
deb.debian.org_debian_dists_experimental_InRelease deb.debian.org_debian_dists_testing_main_Contents-i386.gz
deb.debian.org_debian_dists_experimental_Release deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2
deb.debian.org_debian_dists_experimental_Release.gpg deb.debian.org_debian_dists_testing_main_source_Sources.xz
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz
deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2 deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2
deb.debian.org_debian_dists_testing_contrib_source_Sources.xz deb.debian.org_debian_dists_testing_non-free_source_Sources.xz
deb.debian.org_debian_dists_testing_InRelease deb.debian.org_debian_dists_testing_Release
deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz deb.debian.org_debian_dists_testing_Release.gpg
deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz deb.debian.org_debian_dists_unstable_InRelease
deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz deb.debian.org_debian_dists_unstable_Release
deb.debian.org_debian_dists_testing_main_Contents-amd64.gz deb.debian.org_debian_dists_unstable_Release.gpg
16:39 ♒ ॐ ☺ 😄 

1
2


rrs@priyasi:/tmp/set-folder$ echo 112312312321 >> deb.debian.org_debian_dists_testing_non-free_source_Sources.xz
16:40 ♒ ॐ ☺ 😄 

So in this step, we tell apt-offline to install the downloaded files. This will also include the tampered file. The output you see below is standard and reports everything to have succeeded.

But note that the tampered file is not in the list of synced files. That file is just simply missing from the list.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


rrs@priyasi:/tmp/set-folder$ sudo apt-offline install .
Proceeding with installation
gpgv: Signature made Friday 07 February 2020 01:55:24 PM IST
gpgv: using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:55:43 PM IST
gpgv: using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:44 PM IST
gpgv: using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:45 PM IST
gpgv: using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:58 PM IST
gpgv: using RSA key 16E90B3FDF65EDE3AA7F323C04EE7237B7D453EC
gpgv: Good signature from "Debian Archive Automatic Signing Key (9/stretch) <ftpmaster@debian.org>"
gpgv: Signature made Friday 07 February 2020 01:56:58 PM IST
gpgv: using RSA key 0146DC6D4A0B2914BDED34DB648ACFD622F3D138
gpgv: Good signature from "Debian Archive Automatic Signing Key (10/buster) <ftpmaster@debian.org>"
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_contrib_i18n_Translation-en.bz2 synced.
deb.debian.org_debian_dists_testing_contrib_source_Sources.xz synced.
deb.debian.org_debian_dists_testing_main_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_main_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_main_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_main_i18n_Translation-en.bz2 synced.
deb.debian.org_debian_dists_testing_main_source_Sources.xz synced.
deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz synced.
deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz synced.
deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz synced.
deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2 synced.
16:41 ♒ ॐ ☺ 😄 

So, in the above example, apt-offline discarded the tampered file and the final exit of the command was a success. Now, let’s run the same command with the ‘–verbose’ switch. Below is the output.

Notice the highlighted line below, where it reports that the file is tampered and does not match the checksum

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45


rrs@priyasi:/tmp/set-folder$ sudo apt-offline install . --verbose
VERBOSE: Namespace(allow_unauthenticated=False, func=<function installer at 0x7f6a6c7c54d0>, install='.', install_simulate=False, install_src_path=None, skip_bug_reports=False, skip_changelog=False, strict_deb_check=False, verbose=True)
VERBOSE: No changelog available
Proceeding with installation
VERBOSE: {}
VERBOSE: Great!!! No bugs found for all the packages that were downloaded.

VERBOSE: APT Signature verification path is: ['/etc/apt/trusted.gpg.d/', '/etc/apt/trusted.gpg']
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-security-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-jessie-stable.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-security-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-stretch-stable.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-buster-automatic.gpg to the apt-offline keyring
VERBOSE: Adding /etc/apt/trusted.gpg.d/debian-archive-buster-security-automatic.gpg to the apt-offline keyring

.....snipped.....

VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_Contents-amd64.gz Integrity with checksum 024957d30be2acbb9e66c9802f825115d32437420300a2b28ab60ae4ecb76fcf matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_Contents-i386.gz Integrity with checksum 5266d2f3ea41c4e988e71b4bbe58dd1178a23ce1ed50908c73a0cb39201136e3 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-all_Packages.xz Integrity with checksum 9f0f3aa5560452d45f82c5121ea844c68e641c8fbb56ef69d570c641b6cce662 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-amd64_Packages.xz Integrity with checksum 811f7752a13dfcbd748478dda267fb810c52fc14769d2d5c7871c75e35350d66 matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_binary-i386_Packages.xz Integrity with checksum 7df3512b5da7258613774921023d68c71858d89fddafd694e2dfd19cef54314b matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_i18n_Translation-en.bz2 Integrity with checksum 1bf3cd0cff6fadf1a74280912c3229374344cd6c347d2f533b001843d84b236d matches
VERBOSE: localFile ./deb.debian.org_debian_dists_testing_non-free_source_Sources.xz integrity doesn't match to checksum a94589ab3c204bb4d710d72ea21abac8007b14e5c5dacbe43be07c51ba5f0a0a
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-amd64
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-amd64 file synced to APT.
deb.debian.org_debian_dists_testing_contrib_Contents-amd64.gz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-i386
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_Contents-i386 file synced to APT.
deb.debian.org_debian_dists_testing_contrib_Contents-i386.gz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-all_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-all_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-all_Packages.xz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-amd64_Packages.xz synced.
VERBOSE: Synchronized file to /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages
VERBOSE: /var/lib/apt/lists/deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages file synced to APT.
deb.debian.org_debian_dists_testing_contrib_binary-i386_Packages.xz synced.

.....snipped.....

16:42 ♒ ॐ ☺ 😄

This is pretty much the validation required and done by apt-offline for apt meta Packages files.

Please do file bug reports if you think the overall exit status of apt-offline under such scenarios should be different than what it is currently.

For the tampered meta Packages files:

should the visual representation be different ?
Should an error be printed ?
What about the exit status ?

Similarly, for the ‘get’ operation:

Should we do something different for non-existing localization files on the repository server ?
Is there any different way to go through the supported list of compression types for meta files ?

apt-offline allows a user to install a new package and all its dependencies easily on the offline machine. The below workflow will demonstrate the same and will also go through the tampering of the .deb files and see how apt-offline/apt deals with it.

In below example, a user wants to install the gnome-todo package on the offline machine, which has a couple of dependencies.

rrs@priyasi:/tmp/set-folder$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
 gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
 gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
16:47 ♒ ॐ ☹ 😟=> 1

The below command generates a (signature) file, which will include all details about requested package and its dependencies.

1
2
3


rrs@priyasi:/tmp/set-folder$ sudo apt-offline set /tmp/gnome-todo.uris --install-packages gnome-todo
Gathering installation details for package ['gnome-todo']
16:48 ♒ ॐ ☺ 😄 

Below is the usual step to be performed on the online machine with the generated gnome-todo.uris signature file.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32


rrs@priyasi:/tmp/set-folder$ apt-offline get /tmp/gnome-todo.uris --download-dir /tmp/gnome-todo --bug-reports --threads 3

Fetching APT Data

WARNING: If you are on a slow connection, it is good to
WARNING: limit the number of threads to a low number like 2.
WARNING: Else higher number of threads executed could cause
WARNING: network congestion and timeouts.

Downloading libpeas-common - 187 KiB
Downloading libpeas-1.0-0 - 196 KiB
Downloading gnome-todo-common - 228 KiB
libpeas-common done
Fetching bug report for libpeas-common
libpeas-1.0-0 done
Fetching bug report for libpeas-1.0-0
gnome-todo-common done
Fetching bug report for gnome-todo-common
Fetched bug report for libpeas-common
Downloading libgnome-todo - 6 KiB
libgnome-todo done
Fetching bug report for libgnome-todo
Fetched bug report for gnome-todo-common
Downloading gnome-todo - 146 KiB
gnome-todo done
Fetching bug report for gnome-todo
Fetched bug report for libpeas-1.0-0
Fetched bug report for libgnome-todo
Fetched bug report for gnome-todo
 5 / 5 items: [##############################] 100.0% of 765 KiB
Downloaded data to /tmp/gnome-todo
16:49 ♒ ॐ ☺ 😄 

Note: The fact is that apt-offline will not do any checksum validation for the .deb files. The validation is completely delegated to apt.

rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install -h
usage: apt-offline install [-h] [--verbose] [--simulate]
 [--install-src-path INSTALL_SRC_PATH]
 [--skip-bug-reports] [--skip-changelog]
 [--allow-unauthenticated] [--strict-deb-check]
 apt-offline-download.zip | apt-offline-download/

positional arguments:
 apt-offline-download.zip | apt-offline-download/
 Install apt-offline data, a bundle file or a directory

optional arguments:
 -h, --help show this help message and exit
 --verbose Enable verbose messages
 --simulate Just simulate. Very helpful when debugging
 --install-src-path INSTALL_SRC_PATH
 Install src packages to specified path.
 --skip-bug-reports Skip the bug report check
 --skip-changelog Skip display of changelog
 --allow-unauthenticated
 Ignore apt gpg signatures mismatch
 --strict-deb-check Perform strict checksum validaton for downloaded .deb
 files
16:50 ♒ ॐ ☺ 😄

and from the manpage:

 --strict-deb-check
 With this option enabled, apt-offline delegate's .deb package checksum validation to apt. While the .debs are already avail‐
 able, they are stored in the temporary apt cache, where apt validates its checksum, before considering it for further pro‐
 cessing. Note: This does have the caveat that apt may need network availability even though it doesn't download anything
 over the network. But it does invoke the download routines and realizes that the payload is already available. It then fur‐
 ther proceeds with checksum validation

 The default behavior is to not do strict checksum validation for .deb files. Instead, apt-offline copies the .deb files to
 apt's download location. apt still does size validation of the available .deb files and discards them in case there is a mis‐
 match.

Before we proceed with the example of checksum verification for .deb files, lets do a pristine run of the downloaded files, without any tampering to them.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install .
Proceeding with installation


Following are the list of bugs present.
822525 gnome-todo : gnome-todo: Memory leak while loading local and remote lists
853114 gnome-todo : no longer loads caldav lists
883961 libgnome-todo : libgnome-todo: Not actually a library
829470 libpeas-1.0-0 : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next: (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:51 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
 gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
 gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 0 B/784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.
16:51 ♒ ॐ ☹ 😟=> 1 

In the above example, everything is clean and all requirements to apt are satisfied.

Here’s one more exaple, where we invoke the non-default --strict-deb-check option.

Everything remains the same, but apt gives a prompt saying that it needs to download the payload from the web. The reality is that if you just proceed with yes, nothing gets downloaded.

Note: It is not possible to explain that with a still presentation and I’m lazy to make a motion object of it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42


rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install . --strict-deb-check
Proceeding with installation


Following are the list of bugs present.
822525 gnome-todo : gnome-todo: Memory leak while loading local and remote lists
853114 gnome-todo : no longer loads caldav lists
883961 libgnome-todo : libgnome-todo: Not actually a library
829470 libpeas-1.0-0 : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next: (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:52 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
 gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
 gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian testing/main amd64 libpeas-common all 1.22.0-5 [192 kB]
Get:2 http://deb.debian.org/debian testing/main amd64 libpeas-1.0-0 amd64 1.22.0-5 [201 kB]
Get:3 http://deb.debian.org/debian testing/main amd64 gnome-todo-common all 3.28.1-5 [234 kB]
Get:4 http://deb.debian.org/debian testing/main amd64 libgnome-todo amd64 3.28.1-5 [6,260 B]
Get:5 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
.....snipped.....
16:53 ♒ ॐ ☺ 😄 

To sum it up, this one is an odd case because though nothing for the debs is downloaded, BUT, the network needs to be active for this co-routine to run. If, say, the network is unavailable, apt complains. I haven’t checked, but apt does invoke some network code.

But no payload is downloaded. apt just validates and realizes that all the to-be-downloaded data, is intact and available.

rrs@priyasi:/tmp/gnome-todo$ echo fasdfadsfasdfasdfasd >> gnome-todo_3.28.1-5_amd64.deb
16:54 ♒ ॐ ☺ 😄

rrs@priyasi:/tmp/gnome-todo$ sudo apt clean
16:54 ♒ ॐ ☺ 😄

So we tampered one of the .deb files, gnome-todo_3.28.1-5_amd64.deb. And ask apt-offline to run its ‘install’ operation along with the new --strict-deb-check option.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install . --strict-deb-check
Proceeding with installation


Following are the list of bugs present.
822525 gnome-todo : gnome-todo: Memory leak while loading local and remote lists
853114 gnome-todo : no longer loads caldav lists
883961 libgnome-todo : libgnome-todo: Not actually a library
829470 libpeas-1.0-0 : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next: (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:54 ♒ ॐ ☺ 😄

rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
 gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
 gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian testing/main amd64 libpeas-common all 1.22.0-5 [192 kB]
Get:2 http://deb.debian.org/debian testing/main amd64 libpeas-1.0-0 amd64 1.22.0-5 [201 kB]
Get:3 http://deb.debian.org/debian testing/main amd64 gnome-todo-common all 3.28.1-5 [234 kB]
Get:4 http://deb.debian.org/debian testing/main amd64 libgnome-todo amd64 3.28.1-5 [6,260 B]
Get:5 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Fetched 150 kB in 1s (141 kB/s)
Retrieving bug reports... Done
Parsing Found/Fixed information... Done

16:55 ♒ ॐ ☹ 😟=> 100 

Pay attention to the downloaded data which is only 150 KiB, for the gnome-todo package, which was tampered. Even though apt stated that it needs to download 784 KiB of data, it actually downloaded 150 KiB only. All data was already downloaded by apt-offline but we had tampered one of the files, which resulted in it being re-downloaded.

Now, lets do one more run with the default behavior of apt-offline, i.e. without the --strict-deb-check option. This will result in apt (internally) detecting the tampering and prompting the user that the (tampered) file needs to be downloaded

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41


rrs@priyasi:/tmp/gnome-todo$ sudo apt-offline install .
Proceeding with installation


Following are the list of bugs present.
822525 gnome-todo : gnome-todo: Memory leak while loading local and remote lists
853114 gnome-todo : no longer loads caldav lists
883961 libgnome-todo : libgnome-todo: Not actually a library
829470 libpeas-1.0-0 : libpeas: Python Plugin Broken
(Y) Yes. Proceed with installation
(N) No, Abort.
(R) Redisplay the list of bugs.
(Bug Number) Display the bug report from the Offline Bug Reports.
(?) Display this help message.
What would you like to do next: (y, N, ?)y
gnome-todo_3.28.1-5_amd64.deb file synced.
libgnome-todo_3.28.1-5_amd64.deb file synced.
gnome-todo-common_3.28.1-5_all.deb file synced.
libpeas-1.0-0_1.22.0-5_amd64.deb file synced.
libpeas-common_1.22.0-5_all.deb file synced.
16:56 ♒ ॐ ☺ 😄
rrs@priyasi:/tmp/gnome-todo$ sudo apt^C
16:56 ♒ ॐ ☹ 😟=> 130
rrs@priyasi:/tmp/gnome-todo$ sudo apt install gnome-todo
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
 gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
The following NEW packages will be installed:
 gnome-todo gnome-todo-common libgnome-todo libpeas-1.0-0 libpeas-common
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 150 kB/784 kB of archives.
After this operation, 2,337 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://deb.debian.org/debian testing/main amd64 gnome-todo amd64 3.28.1-5 [150 kB]
Fetched 150 kB in 0s (448 kB/s)
Retrieving bug reports... Done
Parsing Found/Fixed information... Done
.....snipped......
16:57 ♒ ॐ ☺ 😄 

Notice the highlighted line, which gives a less confusing, realistic summary of what needs to be done. In this case, apt is prompting the user that 150 KiB of data needs to be downloaded, which indeed is the case.

Laptop Mode Tools 1.73

rrs@researchut.com (Ritesh Raj Sarraf) — Sat, 11 Jan 2020 15:14:22 +0530

Laptop Mode Tools 1.73

I am pleased to announce the release of Laptop Mode Tools version 1.73

This release includes many bug fixes. For user convenience, 2 command options have been added.

rrs@priyasi:~$ laptop_mode -h
****************************
Following user commands are understood
status : Display a Laptop Mode Tools power savings status
power-stats : Display the power statistics on the machine
power-events : Trap power related events on the machine
help : Display this help message (--help, -h)
version : Display program version (--version, -v)
****************************
15:22 ♒ ॐ ☺ 😄
rrs@priyasi:~$ sudo laptop_mode status
[sudo] password for rrs:
Mounts:
/dev/mapper/nvme0n1p4_crypt on / type btrfs (rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)
/dev/nvme0n1p3 on /boot type ext4 (rw,relatime)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/fuse on /run/user/1000/doc type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Drive power status:
Cannot read /dev/[hs]d[abcdefgh], permission denied - /usr/sbin/laptop_mode needs to be run as root
(NOTE: drive settings affected by Laptop Mode cannot be retrieved.)
Readahead states:
/dev/mapper/nvme0n1p4_crypt: 128 kB
/dev/nvme0n1p3: 128 kB
/dev/nvme0n1p1: 128 kB
Laptop Mode Tools is allowed to run: /var/run/laptop-mode-tools/enabled exists.
/proc/sys/vm/laptop_mode:
0
/proc/sys/vm/dirty_ratio:
40
/proc/sys/fs/xfs/xfssyncd_centisecs:
3000
/proc/sys/vm/dirty_background_ratio:
10
/proc/sys/vm/dirty_expire_centisecs:
3000
/proc/sys/vm/dirty_writeback_centisecs:
500
......SNIPPED......
/sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq:
400000
/sys/devices/system/cpu/cpu5/cpufreq/cpuinfo_max_freq:
2001000
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor:
schedutil
/sys/devices/system/cpu/cpu7/cpufreq/scaling_governor:
schedutil
/proc/acpi/button/lid/LID0/state:
state: open
/sys/class/power_supply/AC/online:
1
/sys/class/power_supply/BAT0/status:
Charging
15:22 ♒ ॐ ☺ 😄
rrs@priyasi:~$ laptop_mode power-stats
Power Supply details for /sys/class/power_supply/AC
P: /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
L: 0
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
E: POWER_SUPPLY_NAME=AC
E: POWER_SUPPLY_ONLINE=1
E: SUBSYSTEM=power_supply
Power Supply details for /sys/class/power_supply/BAT0
P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
L: 0
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
E: POWER_SUPPLY_NAME=BAT0
E: POWER_SUPPLY_STATUS=Charging
E: POWER_SUPPLY_PRESENT=1
E: POWER_SUPPLY_TECHNOLOGY=Li-poly
E: POWER_SUPPLY_CYCLE_COUNT=0
E: POWER_SUPPLY_VOLTAGE_MIN_DESIGN=7600000
E: POWER_SUPPLY_VOLTAGE_NOW=8760000
E: POWER_SUPPLY_CURRENT_NOW=545000
E: POWER_SUPPLY_CHARGE_FULL_DESIGN=6842000
E: POWER_SUPPLY_CHARGE_FULL=6592000
E: POWER_SUPPLY_CHARGE_NOW=6526000
E: POWER_SUPPLY_CAPACITY=98
E: POWER_SUPPLY_CAPACITY_LEVEL=Normal
E: POWER_SUPPLY_MODEL_NAME=DELL G8VCF6C
E: POWER_SUPPLY_MANUFACTURER=SMP
E: POWER_SUPPLY_SERIAL_NUMBER=1549
E: SUBSYSTEM=power_supply
15:23 ♒ ॐ ☺ 😄
rrs@priyasi:~$ laptop_mode power-events
Running Laptop Mode Tools in event tracing mode. Press ^C to interrupt
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent
KERNEL[140321.536870] change /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=AC
POWER_SUPPLY_ONLINE=0
SEQNUM=5908
KERNEL[140321.569526] change /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=BAT0
POWER_SUPPLY_STATUS=Discharging
POWER_SUPPLY_PRESENT=1
POWER_SUPPLY_TECHNOLOGY=Li-poly
POWER_SUPPLY_CYCLE_COUNT=0
POWER_SUPPLY_VOLTAGE_MIN_DESIGN=7600000
POWER_SUPPLY_VOLTAGE_NOW=8761000
POWER_SUPPLY_CHARGE_FULL_DESIGN=6842000
POWER_SUPPLY_CHARGE_FULL=6592000
POWER_SUPPLY_CHARGE_NOW=6526000
POWER_SUPPLY_CAPACITY=98
POWER_SUPPLY_CAPACITY_LEVEL=Normal
POWER_SUPPLY_MODEL_NAME=DELL G8VCF6C
POWER_SUPPLY_MANUFACTURER=SMP
POWER_SUPPLY_SERIAL_NUMBER=1549
SEQNUM=5909
UDEV [140321.577770] change /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=AC
POWER_SUPPLY_ONLINE=0
SEQNUM=5908
USEC_INITIALIZED=140321550931
UDEV [140321.582123] change /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=BAT0
POWER_SUPPLY_STATUS=Discharging
POWER_SUPPLY_PRESENT=1
POWER_SUPPLY_TECHNOLOGY=Li-poly
POWER_SUPPLY_CYCLE_COUNT=0
POWER_SUPPLY_VOLTAGE_MIN_DESIGN=7600000
POWER_SUPPLY_VOLTAGE_NOW=8761000
POWER_SUPPLY_CHARGE_FULL_DESIGN=6842000
POWER_SUPPLY_CHARGE_FULL=6592000
POWER_SUPPLY_CHARGE_NOW=6526000
POWER_SUPPLY_CAPACITY=98
POWER_SUPPLY_CAPACITY_LEVEL=Normal
POWER_SUPPLY_MODEL_NAME=DELL G8VCF6C
POWER_SUPPLY_MANUFACTURER=SMP
POWER_SUPPLY_SERIAL_NUMBER=1549
SEQNUM=5909
USEC_INITIALIZED=140321580812
KERNEL[140324.857185] change /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=AC
POWER_SUPPLY_ONLINE=1
SEQNUM=5912
UDEV [140324.916156] change /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=AC
POWER_SUPPLY_ONLINE=1
SEQNUM=5912
USEC_INITIALIZED=140324887055
KERNEL[140324.917955] change /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=BAT0
POWER_SUPPLY_STATUS=Unknown
POWER_SUPPLY_PRESENT=1
POWER_SUPPLY_TECHNOLOGY=Li-poly
POWER_SUPPLY_CYCLE_COUNT=0
POWER_SUPPLY_VOLTAGE_MIN_DESIGN=7600000
POWER_SUPPLY_VOLTAGE_NOW=8622000
POWER_SUPPLY_CHARGE_FULL_DESIGN=6842000
POWER_SUPPLY_CHARGE_FULL=6592000
POWER_SUPPLY_CHARGE_NOW=6526000
POWER_SUPPLY_CAPACITY=98
POWER_SUPPLY_CAPACITY_LEVEL=Normal
POWER_SUPPLY_MODEL_NAME=DELL G8VCF6C
POWER_SUPPLY_MANUFACTURER=SMP
POWER_SUPPLY_SERIAL_NUMBER=1549
SEQNUM=5913
UDEV [140324.922916] change /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)
ACTION=change
DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
SUBSYSTEM=power_supply
POWER_SUPPLY_NAME=BAT0
POWER_SUPPLY_STATUS=Unknown
POWER_SUPPLY_PRESENT=1
POWER_SUPPLY_TECHNOLOGY=Li-poly
POWER_SUPPLY_CYCLE_COUNT=0
POWER_SUPPLY_VOLTAGE_MIN_DESIGN=7600000
POWER_SUPPLY_VOLTAGE_NOW=8622000
POWER_SUPPLY_CHARGE_FULL_DESIGN=6842000
POWER_SUPPLY_CHARGE_FULL=6592000
POWER_SUPPLY_CHARGE_NOW=6526000
POWER_SUPPLY_CAPACITY=98
POWER_SUPPLY_CAPACITY_LEVEL=Normal
POWER_SUPPLY_MODEL_NAME=DELL G8VCF6C
POWER_SUPPLY_MANUFACTURER=SMP
POWER_SUPPLY_SERIAL_NUMBER=1549
SEQNUM=5913
USEC_INITIALIZED=140324922572
^C
15:24 ♒ ॐ ☹ 😟=> 130

A filtered list of changes is mentioned below. For the full log, please refer to the git repository.

1.73 - Sat Jan 11 14:52:11 IST 2020

* Respect black/white lists when disabling autosuspend
* Add newer power supply names
* Fix crash due external battery of mouse
* Honor configuration setting for battery level polling
* cpufreq: intel_pstate should use performance governors
* runtime-pm: Speed up by avoiding fork in echo_to_file
* runtime-pm: Inline echo_to_file_do
* runtime-pm: Fix echo_to_file* indentation
* runtime-pm: Speed up by avoiding fork in listed_by_{id,type}
* runtime-pm: Simplify vendor/product match
* add help and verison user commands
* Add a power-stats status command
* Separate power sysfs attributes and add sysfs status attribute
* Add device type 'sd' to default blacklist
* Fix rpm spec file for new installable files

Source tarball, Feodra/SUSE RPM Packages available at: https://github.com/rickysarraf/laptop-mode-tools/releases

Debian packages will be available soon in Unstable.

Homepage: https://github.com/rickysarraf/laptop-mode-tools/wiki

Mailing List: https://groups.google.com/d/forum/laptop-mode-tools

What is Laptop Mode Tools

Description: Tools for Power Savings based on battery/AC status
Laptop mode is a Linux kernel feature that allows your laptop to save
considerable power, by allowing the hard drive to spin down for longer
periods of time. This package contains the userland scripts that are
needed to enable laptop mode.
.
It includes support for automatically enabling laptop mode when the
computer is working on batteries. It also supports various other power
management features, such as starting and stopping daemons depending on
power mode, automatically hibernating if battery levels are too low, and
adjusting terminal blanking and X11 screen blanking
.
laptop-mode-tools uses the Linux kernel's Laptop Mode feature and thus
is also used on Desktops and Servers to conserve power

User Mode Linux 5.2

rrs@researchut.com (Ritesh Raj Sarraf) — Thu, 17 Oct 2019 19:34:00 +0530

User Mode Linux 5.2

User Mode Linux version 5.2 has been uploaded to Debian Unstable and will soon be available on the supported architectures. This upload took more time than usual as I ran into a build time failure caused by newer PCAP library.

Thanks to active upstream developers, this got sorted out quick. In the longer run, we may have a much better fix for it.

What is User Mode Linux a.k.a uml

It is one of the initial virtualization technologies that Linux provided, which still works and is supported/maintained. It is about running a Linux kernel as a single unprivileged user process.

There was a similar project, coLinux, that had a port of Linux kernel to Microsoft Windows, that I used to recommend to people, that were more familiar to Microsoft Windows environment. Conceptually, it was very similar to UML. With coLinux, XMing and PulseAudio, it was a delight to see GNU/Linux based applications to run efficiently on Microsoft Windows.

That was years ago. Unfortunately, the last time I checked on coLinux, it did not seem to be maintained any more. User Mode Linux too hasn’t had a very large user/developer base but, despite, it being part of the Linux kernel has kept it fairly well maintained.

The good news is that for the last 2 years or so, uml has had active maintainers/developers for this project. So I expect to see some parity in terms of features and performance numbers eventually. Things like virtio could definitely bring in a lot of performance improvements to UML. And much wider user acceptance ultimately.

To quote the Debian package description:

Package: user-mode-linux
Version: 5.2-1um-1
Built-Using: linux (= 5.2.17-1)
Status: install ok installed
Priority: extra
Section: kernel
Maintainer: User Mode Linux Maintainers <team+uml@tracker.debian.org>
Installed-Size: 43.7 MB
Depends: libc6 (>= 2.28)
Recommends: uml-utilities (>= 20040406-1)
Suggests: x-terminal-emulator, rootstrap, user-mode-linux-doc, slirp, vde2
Homepage: http://user-mode-linux.sourceforge.net/
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: User-mode Linux (kernel)
User-mode Linux (UML) is a port of the Linux kernel to its own system
call interface. It provides a kind of virtual machine, which runs
Linux as a user process under another Linux kernel. This is useful
for kernel development, sandboxes, jails, experimentation, and many
other things.
.
This package contains the kernel itself, as an executable program,
and the associated kernel modules.

User Mode Linux networking

I have been maintaining User Mode Linux for Debian for a couple of years now but one area where I still waste a lot of time at times, is networking.

Today, we have these major virt technologies: Virtualizaiton, Containers and UML. For simplicity, I am keeping UML as a separate type.

For my needs, I prefer keeping a single bridge on my laptop, to which all different types of technologies can talk through, to access the network. By keeping everything through the bridge, I get to focus on just one entry/exit point for firewall, monitoring, trouble shooting etc.

As of today, easily:

VirtualBox can talk to my local bridge
Libvirt/Qemu can talk to my local bridge
Docker can too
systemd-nspawn can also

The only challenge comes in with UML where in I have had to setup a separate tun interface and make UML talk through it using Tuntap Daemon Mode. And on the host, I do NAT to get it talk to the internet.

Ideally, I would like to simply tell UML that this is my bridge device and that it should associate itself to it for networking. I looked at vde and found a wrapper vdeq. Something like this for UML could simplify things a lot.

UML also has an =vde mode wherein it can attach itself to. But from the looks of it, it is similar to what we already provide through uml-utilities in tuntap daemon mode

I am curious if other User Mode Linux users have simplified ways to get networking set up. And ideally, if it can be setup through Debian’s networking ifup scripts or through systemd-networkd. If so, I would really appreciate if you could drop me a note over email until I find a simple and clean way to get comments setup ready for my blog

Bpfcc New Release

rrs@researchut.com (Ritesh Raj Sarraf) — Mon, 14 Oct 2019 14:54:33 +0530

BPF Compiler Collection 0.11.0

bpfcc version 0.11.0 has been uploaded to Debian Unstable and should be accessible in the repositories by now. After the 0.8.0 release, this has been the next one uploaded to Debian.

Multiple source respositories

This release brought in dependencies to another set of sources from the libbpf project. In the upstream repo, this is still a topic of discussion on how to release tools where one depends on another, in unison. Right now, libbpf is configured as a git submodule in the bcc repository. So anyone using the upstream git repoistory should be able to build it.

Multiple source archive for a Debian package

So I had read in the past about Multiple source tarballs for a single package in Debian but never tried it because I wasn’t maintaining anything in Debian which was such. With bpfcc it was now a good opportunity to try it out. First, I came across this post from Raphaël Hertzog which gives a good explanation of what all has been done. This article was very clear and concise on the topic

Git Buildpackage

gbp is my tool of choice for packaging in Debian. So I did a quick look to check how gbp would take care of it. And everything was in place and Just Worked

rrs@priyasi:~/rrs-home/Community/Packaging/bpfcc (master)$ gbp buildpackage --git-component=libbpf
gbp:info: Creating /home/rrs/NoBackup/Community/Packaging/bpfcc_0.11.0.orig.tar.gz
gbp:info: Creating /home/rrs/NoBackup/Community/Packaging/bpfcc_0.11.0.orig-libbpf.tar.gz
gbp:info: Performing the build
dpkg-checkbuilddeps: error: Unmet build dependencies: arping clang-format cmake iperf libclang-dev libedit-dev libelf-dev libzip-dev llvm-dev libluajit-5.1-dev luajit python3-pyroute2
W: Unmet build-dependency in source
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying fix-install-path.patch
dh clean --buildsystem=cmake --with python3 --no-parallel
dh_auto_clean -O--buildsystem=cmake -O--no-parallel
dh_autoreconf_clean -O--buildsystem=cmake -O--no-parallel
dh_clean -O--buildsystem=cmake -O--no-parallel
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building bpfcc using existing ./bpfcc_0.11.0.orig-libbpf.tar.gz
dpkg-source: info: building bpfcc using existing ./bpfcc_0.11.0.orig.tar.gz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: warning: ignoring deletion of directory src/cc/libbpf
dpkg-source: info: building bpfcc in bpfcc_0.11.0-1.debian.tar.xz
dpkg-source: info: building bpfcc in bpfcc_0.11.0-1.dsc
I: Generating source changes file for original dsc
dpkg-genchanges: info: including full source code in upload
dpkg-source: info: unapplying fix-install-path.patch
ERROR: ld.so: object 'libeatmydata.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
W: cgroups are not available on the host, not using them.
I: pbuilder: network access will be disabled during build
I: Current time: Sun Oct 13 19:53:57 IST 2019
I: pbuilder-time-stamp: 1570976637
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/sid-amd64-base.tgz]
I: copying local configuration
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
I: redirecting /dev/ptmx to /dev/pts/ptmx
I: Mounting /var/cache/apt/archives/
I: policy-rc.d already exists
W: Could not create compatibility symlink because /tmp/buildd exists and it is not a directory
I: using eatmydata during job
I: Using pkgname logfile
I: Current time: Sun Oct 13 19:54:04 IST 2019
I: pbuilder-time-stamp: 1570976644
I: Setting up ccache
I: Copying source file
I: copying [../bpfcc_0.11.0-1.dsc]
I: copying [../bpfcc_0.11.0.orig-libbpf.tar.gz]
I: copying [../bpfcc_0.11.0.orig.tar.gz]
I: copying [../bpfcc_0.11.0-1.debian.tar.xz]
I: Extracting source
dpkg-source: warning: extracting unsigned source package (bpfcc_0.11.0-1.dsc)
dpkg-source: info: extracting bpfcc in bpfcc-0.11.0
dpkg-source: info: unpacking bpfcc_0.11.0.orig.tar.gz
dpkg-source: info: unpacking bpfcc_0.11.0.orig-libbpf.tar.gz
dpkg-source: info: unpacking bpfcc_0.11.0-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying fix-install-path.patch
I: Not using root during the build.

Cross Architecture Linux Containers

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 19 Jul 2019 21:34:02 +0530

Linux and ARM

With more ARM based devices in the market, and with them getting more powerful every day, it is more common to see more of ARM images for your favorite Linux distribution. Of them, Debian has become the default choice for individuals and companies to base their work on. It must have to do with Debian’s long history of trying to support many more architectures than the rest of the distributions. Debian also tends to have a much wider user/developer mindshare even though it does not have a direct backing from any of the big Linux distribution companies.

Some of my work involves doing packaging and integration work which reflects on all architectures and image types; ARM included. So having the respective environment readily available is really important to get work done quicker.

I still recollect back in 2004, when I was much newer to Linux Enterprise while working at a big Computer Hardware Company, I had heard about the Itanium 64 architecture. Back then, trying out anything other than x86 would mean you need access to physical hardware. Or be a DD and have shell access the Debian Machines.

With Linux Virtualization, a lot seems to have improved over time.

Linux Virtualization

With a decently powered processor with virtualization support, you can emulate a lot of architectures that Linux supports.

Linux has many virtualization options but the main dominant ones are KVM/Qemu, Xen and VirtualBox. Qemu is the most feature-rich virtualization choice on Linux with a wide range of architectures that it can emulate.

In case of ARM, things are still a bit tricky as hardware definition is tightly coupled. Emulating device type on a virtualized Qemu is not straightforward as x86 architecture. Thankfully, in libvirt, you can be provided with a generic machine type called virt. For basic cross architecture tasks, this should be enough.

virt board profile under libvirt

But, while, virtualization is a nice progression, it is not always an optimal one. For one, it needs good device virtualization support, which can be tricky in the ARM context. Second, it can be (very) slow at times.

And unless you are doing low-level hardware specific work, you can look for an alternative in Linux Containers

Linux Containers

So this is nothing new now. Lots and lots of buzz around containers already. There’s many different implementations across platforms supporting similar concept. From good old chroot (with limited functionality), jails (on BSD), to well marketed products like: Docker, LXC, systemd-nspawn. There’s also some like firejail targeting specific use cases.

As long as you do not have tight dependency on the hardware or a dependency on the specific parts of the Linux kernel (like once I explored the possibility of running open-iscsi in a containerized environment instead), containers are a quick way to get an equal environment. Especially, things like Process, Namespace and Network separation are awesome helping me concentrate on the work rather than putting the focus on Host <=> Guest issues.

Given how fast work can be accomplished with containers, I have been wanting to explore the possibility of building container images for ARM and other architectures that I care about.

The good thing is that architecture virtualization is offered through Qemu and the same tool also provides similar architecture emulation. So features and fixes have a higher chance of parity as they are being served from the same tool.

systemd-nspawn

I haven’t explored all the container implementations that Linux has to offer. Initially, I used LXC for a while. But these days, for my work it is docker, while my personal preference lies with systemd-nspawn.

Part of the reason is simply because I have grown more familiar with systemd given it is the house keeper for my operating system now. And also, so far, I like most of what systemd offers.

Getting Cross Architecture Containers under Debian GNU/Linux

Use qemu-user-static for emulation
Generate cross architecture chroot images with qemu-debootstrap
Import those as sub-volumes under systemd-nspawn
Set a template for your containers and other misc stuff

Glitches

Not everything works perfect but most of it does work.

Here’s my container list. Subvolumed containers do help de-duplicate and save some space. They are also very very quick when creating a clone for the container

rrs@priyasi:~$ machinectl list-images
NAME TYPE RO USAGE CREATED MODIFIED
2019 subvolume no n/a Mon 2019-06-10 09:18:26 IST n/a
SDK1812 subvolume no n/a Mon 2018-10-15 12:45:39 IST n/a
BusterArm64 subvolume no n/a Mon 2019-06-03 14:49:40 IST n/a
DebSidArm64 subvolume no n/a Mon 2019-06-03 14:56:42 IST n/a
DebSidArmhf subvolume no n/a Mon 2018-07-23 21:18:42 IST n/a
DebSidMips subvolume no n/a Sat 2019-06-01 08:31:34 IST n/a
DebianJessieTemplate subvolume no n/a Mon 2018-07-23 21:18:54 IST n/a
DebianSidTemplate subvolume no n/a Mon 2018-07-23 21:18:05 IST n/a
aptVerifySigsDebSid subvolume no n/a Mon 2018-07-23 21:19:04 IST n/a
jenkins-builder subvolume no n/a Tue 2018-11-27 20:11:42 IST n/a
jenkins-builder-new subvolume no n/a Tue 2019-04-16 10:13:43 IST n/a
opensuse subvolume no n/a Mon 2018-07-23 21:18:34 IST n/a
12 images listed.
21:08 ♒♒♒ ☺ 😄

Problems with networking on foreign architectures

So this must mostly have to do with Qemu’s emulation. When making networking work, I did see many reports and fixes upstream about networking and other subsystems having issues with emulation. Luckily for the architectures listed above, I have been able to make use of them with some workarounds.

Here’s a running Debian Sid ARM64 container image under systemd-nsapwn with qemu emulation

DebSidArm64(fba63648a4c4451ebf56eb758463b37d)
Since: Fri 2019-07-19 21:13:51 IST; 1min 16s ago
Leader: 12061 (systemd)
Service: systemd-nspawn; class container
Root: /var/lib/machines/DebSidArm64
Iface: sysbr0
OS: Debian GNU/Linux 10 (buster)
UID Shift: 445841408
Unit: systemd-nspawn@DebSidArm64.service
├─payload
│ ├─init.scope
│ │ └─12061 /usr/bin/qemu-aarch64-static /usr/lib/systemd/systemd
│ └─system.slice
│ ├─console-getty.service
│ │ └─12245 /usr/bin/qemu-aarch64-static /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 vt220
│ ├─cron.service
│ │ └─12200 /usr/bin/qemu-aarch64-static /usr/sbin/cron -f
│ ├─dbus.service
│ │ └─12197 /usr/bin/qemu-aarch64-static /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
│ ├─rsyslog.service
│ │ └─12202 /usr/bin/qemu-aarch64-static /usr/sbin/rsyslogd -n -iNONE
│ ├─systemd-journald.service
│ │ └─12123 /usr/bin/qemu-aarch64-static /lib/systemd/systemd-journald
│ └─systemd-logind.service
│ └─12203 /usr/bin/qemu-aarch64-static /lib/systemd/systemd-logind
└─supervisor
└─12059 /usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest --network-bridge=sysbr0 --bind /var/tmp/Debian-Build/containers/ --bind-ro /var/tmp/:/var/tmp/vartmp -U --settings=override --machine=DebSidArm64
Jul 19 21:13:53 priyasi systemd-nspawn[12059]: [ OK ] Reached target Multi-User System.
Jul 19 21:13:53 priyasi systemd-nspawn[12059]: [ OK ] Reached target Graphical Interface.
Jul 19 21:13:53 priyasi systemd-nspawn[12059]: Starting Update UTMP about System Runlevel Changes...
Jul 19 21:13:53 priyasi systemd-nspawn[12059]: [ OK ] Started Update UTMP about System Runlevel Changes.
Jul 19 21:13:53 priyasi systemd-nspawn[12059]: [ OK ] Started Rotate log files.
Jul 19 21:13:57 priyasi systemd-nspawn[12059]: [ OK ] Started Daily apt download activities.
Jul 19 21:13:57 priyasi systemd-nspawn[12059]: Starting Daily apt upgrade and clean activities...
Jul 19 21:13:59 priyasi systemd-nspawn[12059]: [2B blob data]
Jul 19 21:13:59 priyasi systemd-nspawn[12059]: Debian GNU/Linux 10 SidArm64 console
Jul 19 21:13:59 priyasi systemd-nspawn[12059]: [1B blob data]
21:15 ♒♒♒ ☺ 😄

And the login

rrs@priyasi:~$ machinectl login DebSidArm64
Connected to machine DebSidArm64. Press ^] three times within 1s to exit session.
Debian GNU/Linux 10 SidArm64 pts/0
SidArm64 login: root
Password:
Last login: Thu Jun 20 10:57:30 IST 2019 on pts/0
Linux SidArm64 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) aarch64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@SidArm64:~#
root@SidArm64:~#
root@SidArm64:~# uname -a
Linux SidArm64 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) aarch64 GNU/Linux
root@SidArm64:~#

The problematic networking part. Notice the qemu emulation error messages. Basically, at this stage networking is dead.

root@SidArm64:~# ip a
Unsupported setsockopt level=270 optname=11
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: host0@if10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
request send failed: Operation not supported
link/ether f6:9d:cc:d6:ad:32 brd ff:ff:ff:ff:ff:ffroot@SidArm64:~#
root@SidArm64:~#
root@SidArm64:~# ping www.google.com
ping: www.google.com: Temporary failure in name resolution
root@SidArm64:~#
root@SidArm64:~# cat /etc/resolv.conf
nameserver 172.16.20.1

Because I prefer systemd for containers, I chose to make use of systemd’s network management tools too. Maybe that is causing the problem but in my opinion, is highly unlikely to be the cause.

Anyways….simply invoking dhclient at this stage does assign the container an ip address. But more than that the core networking stack gets back in action, even though the reporting tools still report errors.

root@SidArm64:~# dhclient host0
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unsupported setsockopt level=270 optname=11
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unsupported setsockopt level=263 optname=8
Unsupported setsockopt level=270 optname=11
Unknown target IFA type: 4
Unknown target IFA type: 3
Unknown target IFA type: 6
root@SidArm64:~#
root@SidArm64:~# ip a
Unsupported setsockopt level=270 optname=11
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
Unknown host QEMU_IFLA type: 50
Unknown host QEMU_IFLA type: 51
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: host0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
request send failed: Operation not supported
link/ether f6:9d:cc:d6:ad:32 brd ff:ff:ff:ff:ff:ffroot@SidArm64:~#
root@SidArm64:~#
root@SidArm64:~# ping 172.16.20.1
PING 172.16.20.1 (172.16.20.1) 56(84) bytes of data.
64 bytes from 172.16.20.1: icmp_seq=1 ttl=64 time=0.091 ms
64 bytes from 172.16.20.1: icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from 172.16.20.1: icmp_seq=3 ttl=64 time=0.138 ms
64 bytes from 172.16.20.1: icmp_seq=4 ttl=64 time=0.136 ms
64 bytes from 172.16.20.1: icmp_seq=5 ttl=64 time=0.152 ms
64 bytes from 172.16.20.1: icmp_seq=6 ttl=64 time=0.143 ms
64 bytes from 172.16.20.1: icmp_seq=7 ttl=64 time=0.137 ms
64 bytes from 172.16.20.1: icmp_seq=8 ttl=64 time=0.144 ms
^C
--- 172.16.20.1 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 158ms
rtt min/avg/max/mdev = 0.061/0.125/0.152/0.030 ms
root@SidArm64:~# ping www.google.com
PING www.google.com (216.58.197.68) 56(84) bytes of data.
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=1 ttl=50 time=45.2 ms
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=2 ttl=50 time=63.7 ms
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=3 ttl=50 time=63.2 ms
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=4 ttl=50 time=80.7 ms
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=5 ttl=50 time=65.4 ms
64 bytes from maa03s21-in-f4.1e100.net (216.58.197.68): icmp_seq=6 ttl=50 time=70.4 ms
^C
--- www.google.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 12ms
rtt min/avg/max/mdev = 45.190/64.758/80.717/10.597 ms

Conclusion

With some minor annoyances but otherwise this has been my fastest way to get work done, especially cross architecture stuff. When I compare it back to how I’d have thought such a use case back in the early days, I can’t imagine the speed and simplicity I have at hand today. Free and Opensource Software has been a great choice, while it started just as a curiosity of young boy.

Linux Desktop Usage 2019

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 15 Mar 2019 20:05:06 +0530

If I look back now, it must be more than 20 years since I got fascinated with GNU/Linux ecosystem and started using it.

Back then, it was more curiosity of a young teenager and the excitement to learn something. There’s one thing that I have always admired/respected about Free Software’s values, is: Access for everyone to learn. This is something I never forget and still try to do my bit.

It was perfect timing and I was lucky to be part of it. Free Software was (and still is) a great platform to learn upon, if you have the willingness and desire for it.

Over the years, a lot lot lot has changed, evolved and improved. From the days of writing down the XF86Config configuration file to get the X server running, to a new world where now everything is almost dynamic, is a great milestone that we have achieved.

All through these years, I always used GNU/Linux platform as my primary computing platform. The CLI, Shell and Tools, have all been a great source of learning. Most of the stuff was (and to an extent, still is) standardized and focus was usually on a single project.

There was less competition on that front, rather there was more collaboration. For example, standard tools like: sed, awk, grep etc were single tools. Like you didn’t have 2 variants of it. So, enhancements to these tools was timely and consistent and learning these tools was an incremental task.

On the other hand, on the Desktop side of things, it started and stood for a very long time, to do things their own ways. But eventually, quite a lot of those things have standardized, thankfully.

For the larger part of my desktop usage, I have mostly been a KDE user. I have used other environments like IceWM, Enlightenment briefly but always felt the need to fallback to KDE, as it provided a full and uniform solution. For quite some time, I was more of a user preferring to only use the K* tools, as in if it wasn’t written with kdelibs, I’d try to avoid it. But, In the last 5 years, I took at detour and tried to unlearn and re-learn the other major desktop environment, GNOME.

GNOME is an equally beautiful and elegant desktop environment with a minimalistic user interface (but which at many times ends up plaguing its application’s feature set too, making it “minimalistic feature set applications”). I realized that quite a lot of time and money is invested into the GNOME project, especially by the leading Linux Distribution Vendors.

But the fact is that GNU/Linux is still not a major player on the Desktop market. Some believe that the Desktop Market itself has faded and been replaced by the Mobile market. I think Desktop Computing still has a critical place in the near foreseeable future and the Mobile Platform is more of an extension shell to it. For example, for quickies, the Mobile platform is perfect. But for a substantial amount of work to be done, we still fallback to using our workstations. Mobile platform is good for a quick chat or email, but if you need to write a review report or a blog post or prepare a presentation or update an excel sheet, you’d still prefer to use your workstation.

So…. After using GNOME platform for a couple of years, I realized that there’s a lot of work and thought put into this platform too, just like the KDE platform. BUT To really be able to dream about the “Year of the dominance of the GNU/Linux desktop platform”, all these projects need to work together and synergise their efforts.

Pain points:

Multiple tools, multiple efforts wasted. Could be synergised.
Data accessiblity
Integration and uniformity

Multiple tools

Kmail used to be an awesome email client. Evoltuion today is an awesome email client. Thunderbird was an awesome email client, which from what I last remember, Mozilla had lack of funds to continue maintaining it. And then there’s the never ending stream of new/old projects that come and go. Thankfully, email is pretty standardized in its data format. Otherwise, it would be a nightmare to switch between these client. But still, GNU/Linux platforms have the potential to provide a strong and viable offering if they could synergise their work together. Today, a lot of resource is just wasted and nobody wins. Definitely not the GNU/Linux platform. Who wins are: GMail, Hotmail etc.

If you even look at the browser side of things, Google realized the potential of the Web platform for its business. So they do have a Web client for GNU/Linux. But you’ll never see an equivalent for Email/PIM. Not because it is obsolete. But more because it would hurt their business instead.

Data accessibility

My biggest gripe is data accessiblity. Thankfully, for most of the stuff that we rely upon (email, documents etc), things are standardized. But there still are annoyances. For example, when KDE 4.x debacle occured, kwallet could not export its password database to the newer one. When I moved to GNOME, I had another very very very hard time extracting passwords from kwallet and feeding them to SeaHorse. Then, when recently, I switched back to KDE, I had to similarly struggle exporting back my data from SeaHorse (no, not back to KWallet). Over the years, I realized that critical data should be kept in its simplest format. And let the front-ends do all the bling they want to. I realized this more with Emails. Maildir is a good clean format to store my email in, irrespective of how I access my email. Whether it is dovecot, Evolution, Akonadi, Kmail etc, I still have my bare data intact.

I had burnt myself on the password front quite a bit, so on this migration back to KDE, I wanted an email like solution. So there’s pass, a password store, which fits the bill just like the Email use case. It would make a lot more sense for all Desktop Password Managers to instead just be a frontend interface to pass and let it keep the crucial data in bare minimal format, and accessbile at all times, irrespective of the overhauling that the Desktop projects tend to do every couple of years or so.

Data is critical. Without retaining its compatibility (both backwards and forward), no battle can you win.

I honestly feel the Linux Desktop Architects from the different projects should sit together and agree on a set of interfaces/tools (yes yes there is fd.o) and stick to it. Too much time and energy is wasted otherwise.

Integration and Uniformity

This is something I have always desired and I was quite impressed (and delighted) to see some progress on the KDE desktop in the UI department. On GNOME, I developed a liking for the Evolution email client. Infact, it is my client currently, for Email, NNTP and other PIM. And I still get to use it nicely in a KDE environment. Thank you.

Laptop Mode Tools 1.72

rrs@researchut.com (Ritesh Raj Sarraf) — Thu, 01 Feb 2018 12:45:06 -0500

What a way to make a gift!

I’m pleased to announce the 1.72 release of Laptop Mode Tools. Major changes include the port of the GUI configuration utility to Python 3 and PyQt5. Some tweaks, fixes and enhancements in current modules. Extending {black,white}list of devices to types other than USB. Listing of devices by their devtype attribute.

A filtered list of changes is mentioned below. For the full log, please refer to the git repository.

Source tarball, Feodra/SUSE RPM Packages available at:
https://github.com/rickysarraf/laptop-mode-tools/releases

Debian packages will be available soon in Unstable.

Homepage: https://github.com/rickysarraf/laptop-mode-tools/wiki
Mailing List: https://groups.google.com/d/forum/laptop-mode-tools

1.72 - Thu Feb 1 21:59:24 IST 2018
* Switch to PyQt5 and Python3
* Add btrfs to list of filesystems for which we can set commit interval
* Add pkexec invocation script
* Add desktop file to upstream repo and invoke script
* Update installer to includes gui wrappers
* Install new SVG pixmap
* Control all available cards in radeon-dpm
* Prefer to use the new runtime pm autosuspend_delay_ms interface
* tolerate broken device interfaces quietly
* runtime-pm: Make {black,white}lists work with non-USB devices
* send echo errors to verbose log
* Extend blacklist by device types of devtype

What is Laptop Mode Tools

Description: Tools for Power Savings based on battery/AC status
Laptop mode is a Linux kernel feature that allows your laptop to save
considerable power, by allowing the hard drive to spin down for longer
periods of time. This package contains the userland scripts that are
needed to enable laptop mode.
.
It includes support for automatically enabling laptop mode when the
computer is working on batteries. It also supports various other power
management features, such as starting and stopping daemons depending on
power mode, automatically hibernating if battery levels are too low, and
adjusting terminal blanking and X11 screen blanking
.
laptop-mode-tools uses the Linux kernel's Laptop Mode feature and thus
is also used on Desktops and Servers to conserve power

PS: This release took around 13 months. A lot of things changed, for me, personally. Some life lessons learnt. Some idiots uncovered. But the best of 2017, I got married. I am hopeful to keep work-life balanced, including time for FOSS.

Freezing of tasks failed

rrs@researchut.com (Ritesh Raj Sarraf) — Thu, 28 Dec 2017 01:33:59 -0500

It is interesting how a user-space task could lead to hinder a Linux kernel software suspend operation.

[11735.155443] PM: suspend entry (deep)
[11735.155445] PM: Syncing filesystems ... done.
[11735.215091] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11735.215172] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11735.558676] rfkill: input handler enabled
[11735.608859] (NULL device *): firmware: direct-loading firmware rtlwifi/rtl8723befw_36.bin
[11735.609910] (NULL device *): firmware: direct-loading firmware rtl_bt/rtl8723b_fw.bin
[11735.611871] Freezing user space processes ...
[11755.615603] Freezing of tasks failed after 20.003 seconds (1 tasks refusing to freeze, wq_busy=0):
[11755.615854] digikam D 0 13262 13245 0x00000004
[11755.615859] Call Trace:
[11755.615873] __schedule+0x28e/0x880
[11755.615878] schedule+0x2c/0x80
[11755.615889] request_wait_answer+0xa3/0x220 [fuse]
[11755.615895] ? finish_wait+0x80/0x80
[11755.615902] __fuse_request_send+0x86/0x90 [fuse]
[11755.615907] fuse_request_send+0x27/0x30 [fuse]
[11755.615914] fuse_send_readpages.isra.30+0xd1/0x120 [fuse]
[11755.615920] fuse_readpages+0xfd/0x110 [fuse]
[11755.615928] __do_page_cache_readahead+0x200/0x2d0
[11755.615936] filemap_fault+0x37b/0x640
[11755.615940] ? filemap_fault+0x37b/0x640
[11755.615944] ? filemap_map_pages+0x179/0x320
[11755.615950] __do_fault+0x1e/0xb0
[11755.615953] __handle_mm_fault+0xc8a/0x1160
[11755.615958] handle_mm_fault+0xb1/0x200
[11755.615964] __do_page_fault+0x257/0x4d0
[11755.615968] do_page_fault+0x2e/0xd0
[11755.615973] page_fault+0x22/0x30
[11755.615976] RIP: 0033:0x7f32d3c7ff90
[11755.615978] RSP: 002b:00007ffd887c9d18 EFLAGS: 00010246
[11755.615981] RAX: 00007f32d3fc9c50 RBX: 000000000275e440 RCX: 0000000000000003
[11755.615982] RDX: 0000000000000002 RSI: 00007ffd887c9f10 RDI: 000000000275e440
[11755.615984] RBP: 00007ffd887c9f10 R08: 000000000275e820 R09: 00000000018d2f40
[11755.615986] R10: 0000000000000002 R11: 0000000000000000 R12: 000000000189cbc0
[11755.615987] R13: 0000000001839dc0 R14: 000000000275e440 R15: 0000000000000000
[11755.616014] OOM killer enabled.
[11755.616015] Restarting tasks ... done.
[11755.817640] PM: suspend exit
[11755.817698] PM: suspend entry (s2idle)
[11755.817700] PM: Syncing filesystems ... done.
[11755.983156] rfkill: input handler disabled
[11756.030209] rfkill: input handler enabled
[11756.073529] Freezing user space processes ...
[11776.084309] Freezing of tasks failed after 20.010 seconds (2 tasks refusing to freeze, wq_busy=0):
[11776.084630] digikam D 0 13262 13245 0x00000004
[11776.084636] Call Trace:
[11776.084653] __schedule+0x28e/0x880
[11776.084659] schedule+0x2c/0x80
[11776.084672] request_wait_answer+0xa3/0x220 [fuse]
[11776.084680] ? finish_wait+0x80/0x80
[11776.084688] __fuse_request_send+0x86/0x90 [fuse]
[11776.084695] fuse_request_send+0x27/0x30 [fuse]
[11776.084703] fuse_send_readpages.isra.30+0xd1/0x120 [fuse]
[11776.084711] fuse_readpages+0xfd/0x110 [fuse]
[11776.084721] __do_page_cache_readahead+0x200/0x2d0
[11776.084730] filemap_fault+0x37b/0x640
[11776.084735] ? filemap_fault+0x37b/0x640
[11776.084743] ? __update_load_avg_blocked_se.isra.33+0xa1/0xf0
[11776.084749] ? filemap_map_pages+0x179/0x320
[11776.084755] __do_fault+0x1e/0xb0
[11776.084759] __handle_mm_fault+0xc8a/0x1160
[11776.084765] handle_mm_fault+0xb1/0x200
[11776.084772] __do_page_fault+0x257/0x4d0
[11776.084777] do_page_fault+0x2e/0xd0
[11776.084783] page_fault+0x22/0x30
[11776.084787] RIP: 0033:0x7f31ddf315e0
[11776.084789] RSP: 002b:00007ffd887ca068 EFLAGS: 00010202
[11776.084793] RAX: 00007f31de13c350 RBX: 00000000040be3f0 RCX: 000000000283da60
[11776.084795] RDX: 0000000000000001 RSI: 00000000040be3f0 RDI: 00000000040be3f0
[11776.084797] RBP: 00007f32d3fca1e0 R08: 0000000005679250 R09: 0000000000000020
[11776.084799] R10: 00000000058fc1b0 R11: 0000000004b9ac50 R12: 0000000000000000
[11776.084801] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[11776.084806] QXcbEventReader D 0 13268 13245 0x00000004
[11776.084810] Call Trace:
[11776.084817] __schedule+0x28e/0x880
[11776.084823] schedule+0x2c/0x80
[11776.084827] rwsem_down_write_failed_killable+0x25a/0x490
[11776.084832] call_rwsem_down_write_failed_killable+0x17/0x30
[11776.084836] ? call_rwsem_down_write_failed_killable+0x17/0x30
[11776.084842] down_write_killable+0x2d/0x50
[11776.084848] do_mprotect_pkey+0xa9/0x2f0
[11776.084854] SyS_mprotect+0x13/0x20
[11776.084859] system_call_fast_compare_end+0xc/0x97
[11776.084861] RIP: 0033:0x7f32d1f7c057
[11776.084863] RSP: 002b:00007f32cbb8c8d8 EFLAGS: 00000206 ORIG_RAX: 000000000000000a
[11776.084867] RAX: ffffffffffffffda RBX: 00007f32c4000020 RCX: 00007f32d1f7c057
[11776.084869] RDX: 0000000000000003 RSI: 0000000000001000 RDI: 00007f32c4024000
[11776.084871] RBP: 00000000000000c5 R08: 00007f32c4000000 R09: 0000000000024000
[11776.084872] R10: 00007f32c4024000 R11: 0000000000000206 R12: 00000000000000a0
[11776.084874] R13: 00007f32c4022f60 R14: 0000000000001000 R15: 00000000000000e0
[11776.084906] OOM killer enabled.
[11776.084907] Restarting tasks ... done.
[11776.289655] PM: suspend exit
[11776.459624] IPv6: ADDRCONF(NETDEV_UP): wlp1s0: link is not ready
[11776.469521] rfkill: input handler disabled
[11776.978733] IPv6: ADDRCONF(NETDEV_UP): wlp1s0: link is not ready
[11777.038879] IPv6: ADDRCONF(NETDEV_UP): wlp1s0: link is not ready
[11778.022062] wlp1s0: authenticate with 50:8f:4c:82:4d:dd
[11778.033155] wlp1s0: send auth to 50:8f:4c:82:4d:dd (try 1/3)
[11778.038522] wlp1s0: authenticated
[11778.041511] wlp1s0: associate with 50:8f:4c:82:4d:dd (try 1/3)
[11778.059860] wlp1s0: RX AssocResp from 50:8f:4c:82:4d:dd (capab=0x431 status=0 aid=5)
[11778.060253] wlp1s0: associated
[11778.060308] IPv6: ADDRCONF(NETDEV_CHANGE): wlp1s0: link becomes ready
[11778.987669] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11779.117608] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11779.160930] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11779.784045] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11779.913668] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
[11779.961517] [drm:wait_panel_status [i915]] *ERROR* PPS state mismatch
11:58 ♒♒♒ ☺

Fixing Hardware Bugs

rrs@researchut.com (Ritesh Raj Sarraf) — Tue, 04 Apr 2017 07:02:35 -0400

Bugs can be annoying. Especially the ones that crash or hang and do not have a root cause. A good example of such annoyance can be kernel bugs, where a faulty hardware/device driver hinders the kernel’s suspend/resume process. Because, as a user, while in the middle of your work, you suspend your machine hoping to resume your work, back when at your destination. But, during suspend, or during resume, randomly the bug triggers leaving you with no choice but a hardware reset. Ultimately, resulting in you losing the entire work state you were in.

Such is a situation I encountered with my 2 year old, Lenovo Yoga 2 13. For 2 years, I had been living with this bug with all the side-effects mentioned.

Mar 01 18:43:28 learner kernel: usb 2-4: new high-speed USB device number 38 using xhci_hcd
Mar 01 18:43:54 learner kernel: usb 2-4: new high-speed USB device number 123 using xhci_hcd
Mar 01 18:44:00 learner kernel: usb 2-4: new high-speed USB device number 125 using xhci_hcd
Mar 01 18:44:11 learner kernel: usb 2-4: new high-speed USB device number 25 using xhci_hcd
Mar 01 18:44:16 learner kernel: usb 2-4: new high-speed USB device number 26 using xhci_hcd
Mar 01 18:44:22 learner kernel: usb 2-4: new high-speed USB device number 27 using xhci_hcd
Mar 01 18:44:22 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:22 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:22 learner kernel: usb 2-4: new high-speed USB device number 28 using xhci_hcd
Mar 01 18:44:23 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:23 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:23 learner kernel: usb 2-4: new high-speed USB device number 29 using xhci_hcd
Mar 01 18:44:23 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:23 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:23 learner kernel: usb 2-4: device not accepting address 29, error -71
Mar 01 18:44:24 learner kernel: usb 2-4: new high-speed USB device number 30 using xhci_hcd
Mar 01 18:44:24 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:24 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:24 learner kernel: usb 2-4: device not accepting address 30, error -71
Mar 01 18:44:24 learner kernel: usb usb2-port4: unable to enumerate USB device
Mar 01 18:44:24 learner kernel: usb 2-4: new high-speed USB device number 31 using xhci_hcd
Mar 01 18:44:24 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:25 learner kernel: usb 2-4: new high-speed USB device number 32 using xhci_hcd
Mar 01 18:44:30 learner kernel: usb 2-4: new high-speed USB device number 33 using xhci_hcd
Mar 01 18:44:30 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:31 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:31 learner kernel: usb 2-4: new high-speed USB device number 34 using xhci_hcd
Mar 01 18:44:36 learner kernel: usb 2-4: new high-speed USB device number 35 using xhci_hcd
Mar 01 18:44:36 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:36 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:37 learner kernel: usb 2-4: new high-speed USB device number 36 using xhci_hcd
Mar 01 18:44:37 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:37 learner kernel: usb 2-4: device descriptor read/64, error -71
Mar 01 18:44:37 learner kernel: usb 2-4: new high-speed USB device number 37 using xhci_hcd
Mar 01 18:44:37 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:37 learner kernel: usb 2-4: Device not responding to setup address.
Mar 01 18:44:38 learner kernel: usb 2-4: device not accepting address 37, error -71
Mar 01 18:44:38 learner kernel: usb 2-4: new high-speed USB device number 38 using xhci_hcd
Mar 01 18:44:38 learner kernel: usb 2-4: Device not responding to setup address.
Mar 02 13:34:05 learner kernel: usb 2-4: new high-speed USB device number 45 using xhci_hcd
Mar 02 13:34:05 learner kernel: usb 2-4: new high-speed USB device number 46 using xhci_hcd
Mar 02 13:34:05 learner kernel: usb 2-4: New USB device found, idVendor=0bda, idProduct=0129
Mar 02 13:34:05 learner kernel: usb 2-4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Mar 02 13:34:05 learner kernel: usb 2-4: Product: USB2.0-CRW
Mar 02 13:34:05 learner kernel: usb 2-4: Manufacturer: Generic
Mar 02 13:34:05 learner kernel: usb 2-4: SerialNumber: 20100201396000000
Mar 02 13:34:06 learner kernel: usb 2-4: USB disconnect, device number 46
Mar 02 13:34:16 learner kernel: usb 2-4: new high-speed USB device number 47 using xhci_hcd
Mar 02 13:34:21 learner kernel: usb 2-4: new high-speed USB device number 48 using xhci_hcd
Mar 02 13:34:26 learner kernel: usb 2-4: new high-speed USB device number 49 using xhci_hcd
Mar 02 13:34:32 learner kernel: usb 2-4: new high-speed USB device number 51 using xhci_hcd
Mar 02 13:34:37 learner kernel: usb 2-4: new high-speed USB device number 52 using xhci_hcd
Mar 02 13:34:43 learner kernel: usb 2-4: new high-speed USB device number 54 using xhci_hcd
Mar 02 13:34:43 learner kernel: usb 2-4: new high-speed USB device number 55 using xhci_hcd
Mar 02 13:34:49 learner kernel: usb 2-4: new high-speed USB device number 57 using xhci_hcd
Mar 02 13:34:55 learner kernel: usb 2-4: new high-speed USB device number 58 using xhci_hcd
Mar 02 13:35:00 learner kernel: usb 2-4: new high-speed USB device number 60 using xhci_hcd
Mar 02 13:35:06 learner kernel: usb 2-4: new high-speed USB device number 61 using xhci_hcd
Mar 02 13:35:11 learner kernel: usb 2-4: new high-speed USB device number 63 using xhci_hcd
Mar 02 13:35:17 learner kernel: usb 2-4: new high-speed USB device number 64 using xhci_hcd
Mar 02 13:35:22 learner kernel: usb 2-4: new high-speed USB device number 65 using xhci_hcd
Mar 02 13:35:28 learner kernel: usb 2-4: new high-speed USB device number 66 using xhci_hcd
Mar 02 13:35:33 learner kernel: usb 2-4: new high-speed USB device number 68 using xhci_hcd
Mar 02 13:35:39 learner kernel: usb 2-4: new high-speed USB device number 69 using xhci_hcd
Mar 02 13:35:44 learner kernel: usb 2-4: new high-speed USB device number 70 using xhci_hcd
Mar 02 13:35:50 learner kernel: usb 2-4: new high-speed USB device number 71 using xhci_hcd
Mar 02 13:35:50 learner kernel: usb 2-4: Device not responding to setup address.
Mar 02 13:35:50 learner kernel: usb 2-4: Device not responding to setup address.
Mar 02 13:35:50 learner kernel: usb 2-4: device not accepting address 71, error -71
Mar 02 13:35:50 learner kernel: usb 2-4: new high-speed USB device number 73 using xhci_hcd
Mar 02 13:35:51 learner kernel: usb 2-4: new high-speed USB device number 74 using xhci_hcd
Mar 02 13:35:56 learner kernel: usb 2-4: new high-speed USB device number 75 using xhci_hcd
Mar 02 13:35:57 learner kernel: usb 2-4: new high-speed USB device number 77 using xhci_hcd
Mar 02 13:36:03 learner kernel: usb 2-4: new high-speed USB device number 78 using xhci_hcd
Mar 02 13:36:08 learner kernel: usb 2-4: new high-speed USB device number 79 using xhci_hcd
Mar 02 13:36:14 learner kernel: usb 2-4: new high-speed USB device number 80 using xhci_hcd
Mar 02 13:36:20 learner kernel: usb 2-4: new high-speed USB device number 83 using xhci_hcd
Mar 02 13:36:26 learner kernel: usb 2-4: new high-speed USB device number 86 using xhci_hcd

Thanks to the Linux USB maintainers, we tried investigating the issue, which resulted in uncovering other bugs. Unfortunately, this bug was concluded as a possible hardware bug. The only odd bit is that this machine has a Windows 8.1 copy still lying on the spare partition, where the issue was not seen at all. It could very well be that it was not a hardware bug at all, or a hardware bug which had a workaround in the Windows driver.

But, the results of the exercise weren’t much useful to me because I use the machine under the Linux kernel most of the time.

So, this March 2017, with 2 years completion on me purchasing the device, I was annoyed enough by the bugs. That led me trying out finding other ways to taming this issue.

Lenovo has some variations of this device. I know that it comes with multiple options for the storgae and the wifi component. I’m not sure if there are more differences.

The majority of the devices are connected over the xHCI bus on this machine. If a single device is faulty, or has faulty hardware; it could screw up the entire user experience for that machine. Such is my case. Hardware manufacturers could do a better job if they could provide a means to disable hardware, for example in the BIOS. HP shipped machines have such an option in the BIOS where you can disable devices that do not have an important use case for the user. Good example of such devices are Fingerprint Readers, SD Card Readers, LOMs and mabye Bluetooth too. At least the latter should apply for Linux users, as majority of us have an unpleasant time getting Bluetooth to work out of the box.

But on my Lenovo Yoga, it came with a ridiculous BIOS/UEFI, with very very limited options for change. Thankfully, they did have an option to set the booting mode for the device, giving the choices of Legacy Boot and UEFI.

Back to the topic, with 2 years of living with the bug, and no clarity on if and whether it was a hardware bug or a driver bug, it left me with no choice but to open up the machine.

Next to the mSATA HDD sits the additional board, which houses the Power, USB, Audio In , and the SD Card reader.

Opening that up, I got the small board. I barely use the SD Card reader, and given the annoyances I had to suffer because of it, there was no more mercy in killing that device.

So, next was to unsolder the SD Card reader completely.

Once done, and fitted back into the machine, everything has been working awesomely great in the last 2 weeks. This entire fix costed me र् 0. So sometimes, fixing a bug is all that matters.

In the Hindi language, a nice phrase for such a scenario remnids me of the great Chanakya, “साम, दाम, दंड और भेद”.

Linux IO + Memory + CPU Contention

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 26 Feb 2016 12:23:33 -0500

I very recently met someone, and we had a good productive discussion on the features and (long standing) bugs of the Linux kernel. No doubt, Linux is the most featureful kernel in the market. Is also a lot appealing given its breadth of platform support.

Of that discussion we had, it led about Linux’s behavior in tighter stressed scenarios where there is a lot of contention among the core subsystems. From the conversation, I got the feedback that perhaps the issue is no more valid. My conclusion was I must have missed out on the fix because I haven’t really spent any Engineering Lab time in recent past.

But a picture says a thousand words. And no, the issue is NOT fixed. And it is still simple to reproduce the bug. And it is just that we have faster advancements in underlying hardware, that we may eventually find it much harder to reproduce these bugs.

The kernel spitting the screen is Linux 4.4 Stock Debian.

Linux Desktop in 2014

rrs@researchut.com (Ritesh Raj Sarraf) — Fri, 26 Dec 2014 11:32:01 -0500

We are almost at the end of 2014. While 2014 has been a year with many mixed experiences, I think it does warrant one blog entry ;-)

Recently, I’ve again started spending more time on Linux products / solutions, than spending time focused on a specfic subsystem. This change has been good. It has allowed me to re-cap all the advancements that have happened in the Linux world, umm… in the last 5 years.

Once upon a time, the Linux kernel sucked on the Desktop. It led to many desktop improvement related initiatives. Many accepted in kernel, while others stood as it is (out-of-tree) still as of today. Over the years, there are many people that advocate for such out-of-tree features, for example the -ck patchset, claiming it has better performance. Most of the times, these are patches not carried by your distribution vendor, which leads you to alternate sources, if you want to try. Having some spare time, I tried the Alternative Kernel project. It is nothing but a bunch of patchsets, on top of the stock kernel.

After trying it out, I must say that these patchsets are out-of-tree, for good. I hardly could make out any performance gain. But I did notice a considerable increase in the power consumption. On my stock Debian kernel, the power consumption lies around 15-18 W. That increased to 20+ W on the alternate kernels. I guess most advocates for the out-of-tree patchsets, only measure the 1-2% performance gain, where as completely neglect the fact that that kernel sleeps less often.

But back to the generic Linux kernel performance problem……

Recently, in the last 2 years, the performance suckiness of the Linux kernel is hardly noticed. So what changed ?

The last couple of years have seen a rise in high capacity RAM, at affordable consumer price. 8 - 16 GiB of RAM is common on laptops these days.

If you go and look at the sucky bug report linked above, it is marked as closed, justified Working as Designed. The core problem with the bug reported, has to do with slow media. The Linux scheduler is (in?)efficient. It works hard to give you the best throughput and performance (for server workloads). I/O threads are a high priority task in the Linux kernel. Now map this scene to the typical Linux desktop. If you end up with doing too much buffered I/O, thus exhausting all your available cache, and trigger paging, you are in for some sweet experience.

Given that the kernel highly priotizes I/O tasks, and if your underneath persistent storage device is slow (which is common if you have an external USB disk, or even an internal rotating magnetic disk), you end up blocking all your CPU cycles against the slow media. Which further leads to no available CPU cycles for your other desktop tasks. Hence, when you do I/O at such level, you find your desktop go terribly sluggish.

It is not that your CPU is slow or in-capable. It is just that all your CPU slices are blocked. Blocked waiting for your write() to report a completion.

So what exactly changed that we don ’t notice that problem any more ????

_ RAM_ - Increase in RAM has led to more I/O be accommodated in cache. The best way to see this in action is to do a copy of a large file, something almost equivalent to the amount of RAM you have. But make sure it is less than the overall amount. For example, if you have 4 GiB of RAM, try copying a file of size 3.5 GiB in your graphical file manager. And at the same time, on the terminal, keep triggering the **sync** command. Check how long does it take for the sync to complete. By being able to cache large amount of data, the Linux kernel has been better at improving the overall performance in the eyes of the user.
_ File System_ - But RAM is not alone. The file system has played a very important role too. Earlier, with ext3 file system, we had a commit interval of (5?) 30 seconds. That led to the above mentioned **sync** equivalent to get triggered every 30 secs. It was a safety measure to ensure, that at worst, you lose 30 secs worth of data. But it did hinder performance. With ext4, came delayed allocation. Delayed Allocation allowed the write() to return immediate while the data was in cache, and deferred the task of actual write() to the file system. This allowed for the allocator to find the best contiguous slot for the data to be written. Thus it improved the file system. It also brough corruption for some of the apps. :-)
_ Solid State Drives_ - The file system and RAM alone aren ’t the sole factors that led to the drastic improvement in the overall experience of the Linux desktop. If you read through the bug report linked in this article, you’ll find the core root cause to be slow persistent storage devices. Could the allocator have been improved (like Windows) to not be so pressing of the Linux desktop ? Maybe, yes. But that was a decision for the kernel devs and they believed (and believe) to keep those numbers to minimum. Thus for I/O, as for today, you have 3 schedulers and for CPU, just 1. What dramatically improved the overall Linux Desktop performance was the general availability of solid state devices. These device are real fast, which in effect made the write() calls return immediate, and did not block the CPU.

So, it was the advancement in both hardware and software that led to better overall desktop performance. Does the above mentioned bug still exist ? Yes. Its just that it is much harder to trigger it now. You’ll have to ensure that you max out your cache and trigger paging. And then try to do ask for some CPU cycles.

But it wasn’t that back then we didn’t use Linux on the desktop / laptop. It sure did suck more than, say, Windows. But hey, sometimes we have to eat our own dog food. Even then, there sure were some efforts to overcome the then limitations. The first and obvious one is the out-of-tree patchset. But ther were also some other efforts to improve the situation.

The first such effort, that I can recollect, was ulatency. With Linux adding support for Control Groups, there were multiple avenues open on how to tackle and tame the resource starvation problem. The crux of the problem was that Linux gave way too much priority to the I/O tasks. I still wish Linux has a profile mechanism, where in on the kernel command line, we could specify what profile should Linux boot into. Anyways, with ulatency, we saw improvements in the Linux Desktop experience. ulatency had in-built policies to whitelist / blacklist a set of profiles. For example, KDE was a profile. Thus, ulatency would club all KDE processes into a group and give that group a higher precedence to ensure that it had its fair share of CPU cycles.

Today, at almost the end of 2014, there are many more consumer of Linux’s control groups. Prominent names would be: LXC and systemd.

ulatency has hardly seen much development in the last year. Probably it is time for systemd to take over.

systemd is expected to bring lots of features to the Linux world, thus bridging the (many) gap Linux has had on the desktop. It makes extensive use of Control Groups for a variety of (good) reasons, which has led it to be a linux-only product. I think it should have never marketed itself as the init daemon. It rather fits better when called as the System Management Daemon.

The path to Linux Desktop looks much brighter in 2015 and beyond thanks to all the advancements that have happened so far. The other important players, who should be thanked are Mobile and Low Latency products (Android, ChromeBook), whose engagement to productize Linux has led to better features overall.

Power consumption on Linux 3.10

rrs@researchut.com (Ritesh Raj Sarraf) — Tue, 23 Jul 2013 16:08:19 -0400

The power consumption on the Linux kernel 3.10 is pretty bad.

On kernel 3.10, with the follwing config, the PowerTop results are:

# Timers subsystem

CONFIG_TICK_ONESHOT=y

CONFIG_NO_HZ_COMMON=y

# CONFIG_HZ_PERIODIC is not set

CONFIG_NO_HZ_IDLE=y

# CONFIG_NO_HZ_FULL is not set

CONFIG_NO_HZ=y

CONFIG_HIGH_RES_TIMERS=y

PowerTOP v2.0 Overview Idle stats Frequency stats Device stats Tunables

The battery reports a discharge rate of 28.0 W

The estimated remaining time is 23 minutes

Summary: 1785.5 wakeups/second, 0.0 GPU ops/second, 0.0 VFS ops/sec and 22.1% CPU use

Power est. Usage Events/s Category Description

16.3 W 2915 rpm Device Laptop fan

5.11 W 100.0% Device USB device: WALTON Primo-X1 Primo-X1

1.70 W 33.3% Device Display backlight

849 mW 33.3% Device Display backlight

425 mW 86.0 ms/s 330.7 Process /usr/bin/konsole

316 mW 63.9 ms/s 66.1 Process /usr/bin/plasma-desktop

142 mW 28.6 ms/s 396.8 Process /usr/bin/X :0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7

64.1 mW 13.0 ms/s 198.4 Process kwin -session 101261418fe3000136103713100000053880000_13746081

53.6 mW 10.8 ms/s 0.00 Process powertop

35.9 mW 7.3 ms/s 66.1 Process /usr/lib/chromium/chromium –type=plugin –plugin-path=/usr/li

24.3 mW 4.9 ms/s 396.8 Interrupt PS/2 Touchpad / Keyboard / Mouse

6.92 mW 1.4 ms/s 0.00 Interrupt [48] i915

5.94 mW 1.2 ms/s 66.1 Interrupt [9] RCU(softirq)

3.98 mW 0.8 ms/s 0.00 kWork flush_to_ldisc

3.78 mW 0.8 ms/s 66.1 Process [ksoftirqd/2]

3.33 mW 673.3 us/s 66.1 Process [rcu_sched]

1.80 mW 363.1 us/s 66.1 Interrupt [1] timer(softirq)

1.79 mW 363.0 us/s 0.00 Process [ksoftirqd/4]

Where as on the 3.9 kernel:

The battery reports a discharge rate of 13.2 W

The estimated remaining time is 43 minutes

Summary: 611.5 wakeups/second, 0.0 GPU ops/second, 0.0 VFS ops/sec and 14.2% CPU use

Power est. Usage Events/s Category Description

14.0 W 2722 rpm Device Laptop fan

1.72 W 33.3% Device Display backlight

862 mW 33.3% Device Display backlight

255 mW 65.7 ms/s 58.0 Process /usr/bin/plasma-desktop

91.9 mW 23.7 ms/s 27.5 Process /usr/lib/chromium/chromium –type=renderer –lang=en-US –forc

60.1 mW 15.5 ms/s 96.1 Process /usr/lib/chromium/chromium –type=plugin –plugin-path=/usr/li

25.0 mW 6.4 ms/s 25.1 Process kwin -session 101261418fe3000136103713100000053880000_13746094

21.5 mW 5.6 ms/s 34.2 Process /usr/bin/X :0 -auth /var/run/lightdm/root/:0 -nolisten tcp vt7

13.1 mW 3.4 ms/s 5.6 Process /usr/bin/konsole

9.82 mW 2.5 ms/s 53.7 Process [irq/48-iwlwifi]

9.11 mW 2.4 ms/s 2.2 Process /usr/bin/knemo

8.62 mW 2.2 ms/s 12.3 Process /usr/lib/chromium/chromium –password-store=detect

8.32 mW 2.1 ms/s 45.5 Interrupt [48] iwlwifi

6.96 mW 1.8 ms/s 35.3 Interrupt [7] sched(softirq)

5.13 mW 1.3 ms/s 57.1 Interrupt [47] i915

4.24 mW 1.1 ms/s 0.4 Process powertop

3.38 mW 0.9 ms/s 1.5 Timer tcp_keepalive_timer

2.85 mW 0.7 ms/s 11.9 Process /usr/sbin/mysqld –basedir=/usr –datadir=/var/lib/mysql –plu

Linux on RESEARCHUT

Immutable Debian

Immutable Atomic Linux Distirbutions

Chrome OS

Immutable Linux Distributions

Immutable Linux

Ritesh’s Immutable Debian

🏗️ System Architecture: Immutable Debian (Btrfs + MergerFS)

1. Storage Strategy: Subvolume Isolation

1.1 /etc/fstab

2. Tiered Memory Model (/tmp)

3. Hibernation & Swap Logic

3.1 systemd sleep/Hibernation

4. Immutability & Safety Mechanisms

A. The Control Script (immutectl)

B. The Smart Seal (immutability-seal.service)

5. Monitoring & Maintenance

5.1 Nagging

Immutable Debian Nag

5.2 Verification

Wrap

KDE PowerDevil Systemd Inhibit

apt-offline 1.8.6

apt-offline 1.8.6

Changelog

Resources

Systemd Service Hang

Update: 2024-08-15

Btrfs Subvol Fix

apt-offline 1.8.4

apt-offline 1.8.4

Changelog

Resources

ThinkPad AMD Debian

ThinkPad T14 Gen2 AMD

Lenovo

AMD

Power/Thermal

Linux

Migration

btrfs send/receive

snapper

Conclusion

Setting Up a Secure Webapp

Moments

Privacy

Google Photos

PhotoPrism

nginx auth prompt

PhotoPrism Photo List

PhotoPrism Menu

Updates

Wayland KDE X11

KDE Impressions

Wayland

Plasma 5.21

OBS - Open Build Service

OBS - Some detail

LUKS Headless Laptop

Laptop Mode Tools 1.74

Laptop Mode Tools 1.74

1.74 - Sat Jul 18 19:10:40 IST 2020

Resources

What is Laptop Mode Tools

Kodi PS3 BD Remote

Setting up a Sony PS3 Blu-Ray Disc Remote Controller with Kodi

Raspberry Pi

Raspberry Pi Issues

Laptop with Kodi

HDMI CEC

WiFi Remotes

Sony PlayStation 3 BD Remote

Rusted inside

Rusted inside and cover

Rusted spring

Pairing the BD Remote to laptop

Mapping the keys in Kodi

Update: Wed Jun 17 11:38:20 2020

Others

Next

A. The Control Script (`immutectl`)

B. The Smart Seal (`immutability-seal.service`)