What's wrong with normal SSGs?

When I created this blog (2023) I used Jekyll Static Site Generator (SSG) to produce HTML from Markdown files. You write the post in plain Markdown and the engine compiles it into a beautiful page with header and footer, pretty simple. I used GitHub Action to compile the site on every pushed commit.

Over time I've noticed several things that annoyed me so bad that I started to seek an alternative:

• the site source was unintelligible to me - I like to know how stuff works and I love to learn by example, but the SSG output is very obtuse and optimized for SEO and whatnot, which makes for an awful learning experience
• I got interested in Small Web/IndieWeb/smolweb and my blog did not live up to that standard (more on that later)
• GitHub changed the tools often enough that publishing every new post meant tweaking the Action configuration and upgrading Jekyll; things often broke and I had to do a few tries to get my site up
• My site was not flexible enough for me; it might be due to my lack of knowledge about the tool I was using, but I didn't want to spend time learning a tool that I would use a couple times a year if my knowledge would become obsolete after a few uses when new version changes everything

Other than that, I had a few other goals I wanted to accomplish:

• learn Bash - although I daily drive GNU/Linux systems for more than 6 years I've never really had much use for classic tools like sed, awk or even grep; I wanted a fun challenge and rewriting my blog in Bash was an excellent choice
• I wanted a system that will last - *nix shell environment is so stable that my generator will work in 2, 5 or even 10 years (probably with minor tweaks)
• the source will be human-readable - manually crafted HTML and CSS is always nice to look at
• I wanted to reactivate my blog and be more active - hopefully I will have more time to do cool stuff and write about it
• I also have something big to write about, but I won't spoil it now
• I will be migrating from GitHub and will probably buy a proper domain; I've been using Codeberg for my private repositories for a year now and I'm ready to jump the ship

Blog page structure

Header and footer

The first stage of rewriting the static site generator is the creation of header and footer. The content, written in Markdown and converted to (X)HTML, sits between those. Building the page comes down to essentially one command:

cat header.html content.html footer.html

I wrote a new header and footer to match my old layout, wrote a CSS stylesheet to make my site pretty and then wrote the parsing part. When I was writing the HTML I tried to follow the rules laid out in the smolweb guidelines. I don't think I followed them very faithfully, but I hope I managed to produce something accessible that degrades gracefully. Browsing the page in Lynx works well enough.

Body

Processing the content was fun. I decided to use md2html utility available in Debian repos as a starting point. I could have written the parser myself in Bash, but I wanted to allocate finite time to the project and parsing Markdown was not a priority. Still, the output of that tool had to be sanitized. My posts were originally written for Jekyll, so they contain the Yaml front-matter with page data at the beginning of the file. I had to parse it separately from the proper body and extract the title, type and creation date. Figuring out how to make the converted output work in the context of the whole page was fun. This is the pipeline that I made:

# footnotes parsing
  sed -E '
 s/^\[\^([^][]+)\]:[[:space:]]*/<br \/><a id="fn\1" href="#ref\1"><sup>\1<\/sup><\/a> /
 s/\[\^([^][]+)\][[:space:]]*/<a href="#fn\1" id="ref\1"><sup>\1<\/sup><\/a> /g' | md2html --github -x |
# lower the headings
  sed \
    -e 's@<h5>@<h6>@g; s@</h5>@</h6>@g' \
    -e 's@<h4>@<h5>@g; s@</h4>@</h5>@g' \
    -e 's@<h3>@<h4>@g; s@</h3>@</h4>@g' \
    -e 's@<h2>@<h3>@g; s@</h2>@</h3>@g' |
# lower h1 to h2 and add id for toc linking
  perl -pe 's{<h1>(<a href="#fn<" id="ref<"><sup><</sup></a> +)</h1>}{
   $t = $1;
   $id = lc($t);
   $id =~ s/<a href="#fna-z0-9" id="refa-z0-9"><sup>a-z0-9</sup></a> +/-/g;
   $id =~ s/^-+|-+$//g;
   "<h2 id=\"$id\">$t</h2>"}ge' |
# remove image CSS parameters from legacy posts
  sed -E 's/(<img<a href="#fn>" id="ref>"><sup>></sup></a> *>)\s*\{:<a href="#fn}" id="ref}"><sup>}</sup></a> *\}/\1/g'

I generate the HTML ToC, the title and publishing and modification dates and combine all that to create the content body.

If the layout of the page in the front-matter says "home", then it is the index page that lists all the blog posts. To list all posts with their titles, dates and excerpts I do the following:

local files=($(grep -l '^date: ' * | while read -r file; do
  local date=$(grep '^date: ' "$file" | cut -d' ' -f2)
  printf "%s\n" "$file"
done | sort -n | cut -f2-))
for ((i = ${#files[@]} - 1; i >= 0; i--)); do
  local post_file="${files[i]}"
  local date_md=$(grep '^date: ' "$post_file" | cut -d' ' -f2-)
  local date_utc=$(date -u -d "$date_md" +"%Y-%m-%dT%H:%M:%SZ")
  local title=$(grep '^title: ' "$post_file" | cut -d' ' -f2-)
  local excerpt=$(grep '^excerpt: ' "$post_file" | cut -d' ' -f2-)
  local url=$(
    printf "%s" "$post_file" |
      sed 's/^\(.\{4\}\)./\1\//;
   s/^\(.\{7\}\)./\1\//;
   s/^\(.\{10\}\)./\1\//;
   s/\.markdown/\.html/'
  )
  echo "<a href="$url"><h3>$title</h3></a>"
  echo "<small>$date_utc</small><br />"
  echo "<p>$excerpt</p>"

This goes through all the posts in creation chronological order and writes the titles, UTC dates and excerpts.

Compilation

The script I wrote compiles a single site. My blog is of course composed of many files and not all of them need to be compiled every time I change something. The obvious choice of a tool to conditionally compile changed files was make. This was the most tedious and time-consuming part of my project because I cannot write Makefiles for the life of me. I managed to make something like this:

PAGES_SRC := pages-md
POSTS_SRC := _posts
HTML_OUT := html

PAGES_MD := $(wildcard $(PAGES_SRC)/*.markdown)
POSTS_MD := $(wildcard $(POSTS_SRC)/*.markdown)

PAGES_HTML := $(patsubst $(PAGES_SRC)/%.markdown,$(HTML_OUT)/%.html,$(PAGES_MD))
POSTS_HTML := $(foreach f,$(POSTS_MD),$(HTML_OUT)/$(shell echo $(notdir $(f)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.markdown$$/\1\/\2\/\3\/\4.html/'))
POSTS_HTML_DIRS := $(foreach f,$(POSTS_MD),$(HTML_OUT)/$(shell echo $(notdir $(f)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.markdown$$/\1\/\2\/\3/'))

define post_rule
$(HTML_OUT)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.markdown$$/\1\/\2\/\3\/\4.html/'): $(1) | $(HTML_OUT)/$(shell echo $(notdir $(1)) | sed -E 's/^([0-9]{4})-([0-9]{2})-([0-9]{2})-(.*)\.markdown$$/\1\/\2\/\3/')
 /bin/bash ./blampub.sh $$< > $$@
endef

all: $(PAGES_HTML) $(POSTS_HTML) $(HTML_OUT)/feed.xml

$(HTML_OUT)/%.html: $(PAGES_SRC)/%.markdown
 /bin/bash ./blampub.sh $< > $@

$(foreach f,$(POSTS_MD),$(eval $(call post_rule,$(f))))

$(POSTS_MD): | $(POSTS_HTML_DIRS)

$(POSTS_HTML_DIRS):
 mkdir -p $@

$(HTML_OUT)/feed.xml:
 /bin/bash ./feedgen.sh > $@

clean:
 find $(HTML_OUT)/ -mindepth 1 -not -path '*/assets*' -delete

The main complication was the handling of post files names. Each markdown source file has a name like yyyy-mm-dd-arbitrary-post-title.markdown. The output page is stored in yyyy/mm/dd/arbitrary-post-title.html. I needed to extract the date from the filename, create the proper directory structure and preserve the rest of the filename. The fact that the name could contain hyphens complicated the matter, because I could not simply turn every - into / and have the path for free. This is why I need the lengthy post_rule that generates appropriate rule for every source file that exists in the foreach loop.

Feed

The Atom feed was trivial to implement. It is mostly echoing the XML boilerplate and the parsed Markdown file. This time I do not lower the heading levels. I reused the code for parsing and listing posts in chronological order.

Outcome

The result I got is a nice, understandable static site generator that I hope will serve me well. I learned a bit of Bash in the process which is what I wanted. In the process of migrating to the new generator I had to upgrade some posts to remove MathJax script CDN linking. Now I use a minimal installation of Temml that sits in the website assets and is loaded in the browser to turn every $$text$$ into LaTeX math.

Prelude

On my way to true digital independence I decided to set up my own cloud service. I tried to find the most hassle-free way to do it, hence I went with a Nextcloud instance on a Raspberry Pi 3 Model B I had lying around. It was not a totally trivial task since everything had to be set up in a quite restrictive confines of Eduroam network and without an external display. I managed to get a working service, but frequent connection issues I am having due to reverse proxy and network setup prevent me from fully enjoying the functionality of my server. Still, the project was worth the time it took.

System setup

For my system I have decided to go with plain Raspberry Pi OS Lite 64 bit. I used the Raspberry Pi Imager app from Flathub. After flashing and booting up my Pi, the machine connected to my hotspot. Te first thing I did was to install Raspberry Pi Connect to never lose the access to my Raspberry Pi in case I have problems connecting through SSH in my dorm's WI-Fi. I issued

$ sudo apt install rpi-connect-lite
$ loginctl enable-linger
$ rpi-connect on
$ rpi-connect signin

and signed in. I rebooted the Pi to see if I set up everything correctly. Establishing connection through RPi Connect takes a while, but works.
Next I set up the network connection. Eduroam which is a bit different from your usual Wi-Fi connection. I ran

$ sudo nmtui

and added a connection with such parameters:

           Profile name eduroam_________________________________
                 Device wlan0 (B8:27:EB:AA:7E:5A)_______________
                                                                
+ WI-FI                                                         
|                  SSID eduroam_________________________________
|                  Mode <Client>                                
|                                                               
|              Security <WPA & WPA2 Enterprise>                 
|        Authentication <PEAP>                                  
|    Anonymous identity PROVIDED BY UNI_________________________
|                Domain ________________________________________
|               CA cert ________________________________________
|      CA cert password ________________________________________
|                       [ ] Show password                       
|          PEAP version <Automatic>                             
|  Inner authentication <MSCHAPv2>                              
|              Username PROVIDED BY UNI_________________________
|              Password PROVIDED BY UNI_________________________
|                       [ ] Show password                       
|                       <Store password for all users>          
|                                                               
|                 BSSID ________________________________________
|    Cloned MAC address ________________________________________
|                   MTU __________ (default)                    
\                                                               
                                                                
- IPv4 CONFIGURATION    <Automatic>                             
- IPv6 CONFIGURATION    <Automatic>                             
                                                                
[X] Automatically connect                                       
[X] Available to all users                                      

I do not need a certificate to connect but I know some schools require it, so refer to your network staff's manual. For me this did the trick, although I had to wait a couple of minutes for my RPi to show up in Connect.
I wanted to store my data on external USB SSD drive, which is already formatted to ext4. I set up automatic mount:

$ sudo mkdir /mnt/ssd-data-mount
$ sudo mount -t ext4 /dev/sda1 /mnt/ssd-data-mount/
$ sudo systemctl daemon-reload
$ sudo blkid

I took note of the UUID= value, made a backup of /etc/fstab and edited it by appending

UUID=d49402b9-7e8c-4cfc-aba5-253585a8911c /mnt/ssd-data-mount ext4 defaults,auto,users,rw,nofail 0 0

where the value of UUID is the one I copied from previous command. In the end I created a directory for my data:

$ mkdir /mnt/ssd-data-mount/nextcloud-data
$ sudo chmod 750 -R /mnt/ssd-data-mount/nextcloud-data
$ sudo chown www-data:www-data /mnt/ssd-data-mount/nextcloud-data -R

Webserver

Now I could set up the proper functionality of my cloud server.
First I installed and configured UFW

$ sudo apt install ufw
$ sudo ufw allow 80
$ sudo ufw allow 443
$ sudo ufw allow 3478
$ sudo ufw enable

Since I cannot port forward in the Eduroam network I settled for Loophole to tunnel the traffic to my Pi. I downloaded the CLI Linux arm64 executable from their website. Next I created a ~/server-config directory where I store all the behind-the-scenes configuration scripts. I unzipped the package into my new directory, cded into it and ran

$ ./loophole account login

and logged in in the browser. Now to run the forwarding just run

$ ./loophole http 443 --hostname HOSTNAME

where HOSTNAME is some name you will remember.
I created startup-script.sh populated with

#!/bin/bash

cd ~/server-config
./loophole http 443 --hostname HOSTNAME &
./loophole http 80 --hostname HOSTNAME &

and made it executable and immediately added te service to systemd by creating a file /etc/systemd/system/server-startup.service:

[Unit]
Description=Startup script activation
After=multi-user.target
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=pi
ExecStart=/home/pi/server-config/startup-script.sh

[Install]
WantedBy=multi-user.target

and finally

$ sudo systemctl daemon-reload
$ sudo systemctl enable server-startup.service

I also set /etc/hostname to my domain from Loophole, e.g. HOSTNAME.loophole.site.
I went with bare metal Nextcloud install instead of Docker one. I installed the prerequisites

$ sudo apt install apache2 mariadb-server libapache2-mod-php 
$ sudo apt install php-gd php-json php-mysql php-curl php-mbstring php-intl php-imagick php-xml php-zip php-fpm
$ sudo service apache2 restart

and downloaded Nextcloud

$ cd /var/www/html
$ sudo wget https://download.nextcloud.com/server/releases/latest.zip
$ sudo unzip latest.zip
$ sudo chmod 750 nextcloud -R
$ sudo chown www-data:www-data nextcloud -R

I setup of MySQL:

$ sudo mysql

and in the interactive console I issued:

CREATE USER 'nextcloud' IDENTIFIED BY 'password';
CREATE DATABASE nextcloud;
GRANT ALL PRIVILEGES ON nextcloud.* TO 'nextcloud'@localhost IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
quit

Next I set the contents of /etc/apache2/sites-available/nextcloud.conf to

Alias /nextcloud "/var/www/html/nextcloud/"

<Directory /var/www/html/nextcloud/>
  Require all granted
  AllowOverride All
  Options FollowSymLinks MultiViews

  <IfModule mod_dav.c>
    Dav off
  </IfModule>

</Directory>

then

$ sudo a2enmod headers
$ sudo systemctl reload apache2

I also edited /var/www/html/nextcloud/config/config.php so that trusted_domains contains my domain and not the IP addresses and also changed and added the following fields a few lines down:

'overwrite.cli.url' => 'example.com',
'overwritehost' => 'example.com',

where example.com is your domain obtained from Loophole (without http:// part). This is important because otherwise the site will redirect to 127.0.0.1 everytime you want to access it through your domain. Leaving the IP of Raspberry Pi in my local network produced error 502 when accessed from the outside, even it I could access it through my domain inside of the network.
Now I could access my Nextcloud page and start tweaking and uploading my data.

Result

The Nextcloud server on Raspberry Pi 3 works, but is far from perfect. The web interface is not snappy and file uploads are slow, but for a device eight years of age and with 1 GB of RAM it fares pretty well. Unfortunately my Eduroam setup frequently breaks for seemingly no reason, website either throws error 502 or is unreachable. This is remedied either by reboot (sometimes a few times) or by leaving it be in hope it works again (sometimes it does!).
Before writing this post I tested similar setup (Loophole forwarding included) in my private network at home and to my surprise it was quite stable. Unfortunately, when I moved it over to the dorm my SSD died, probably because of a faulty power outlet. Before that I managed to run it for about two weeks during which I set up auto sync of photos and GPS tracking data from my phone and backup of files from my laptop. Overall file syncing worked like a charm. The same cannot be said about browsing said data in web UI. Loading photos took ages and browsing treks in GpxPod froze the server to the point of needing reboot. Preview Generator helped, but was not enough to get smooth experience.

Conclusion

Setting up a Nextcloud instance on Raspberry Pi is a good introduction to server management and selfhosting. I believe Raspberry Pi 4 or 5 would be more appropriate choice for this task than 3, but for those ready to forgo most of the bells and whistles of Nextcloud it can be a very usable option as well.

Motivation

The pinnacle of microcontroller lab I attended this semester is my own project based on any microcontroller I could get my hands on. I thought hard about things that are cool but also useful, things which will not end up at the bottom of my drawer. Recently gathering data has been my jam, and this has led me to the idea of a smartwatch. I figured a gizmo measuring my heartbeat and saturation is a good starter and the rest will come later.

Construction

Hardware

I set off in search of parts and quickly found what I needed. Waveshare provided me with more than what I wanted: Waveshare RP2040 Touch LCD 1.28". This is a touch-enabled round display integrated with RP2040 microcontroller, accelerometer+gyroscope module, RTC and battery header. Almost fully-featured smartwatch in itself, and to be honest I feel quite dirty using it in my project, but due to time, money and volume constraints this is the best decision.

Arguably, one might complain that the board has no connectivity, thus why the first draft included ESP32 which would talk to RP2040 through exposed GPIO ports. However, because of coming deadline and various difficulties with that soution, the idea was dropped (for now).

From the beginning the most important part besides the chip and screen was pulse oximeter. I reviewed all available options which did not cost a small fortune and settled for MAX30102 board from local eshop. This model does not contain green LED (like in MAX30105). Green light makes for better wrist measurements, but I could not find any such modules and just hoped for luck.

At this point my sole concern has been battery dimensions. I found Li-Pol Akyga 85 mAh 1S 3,7 V which might just barely fit in what I had in mind.

Code

I started by writing some basic program to show date and time. As time has been critical factor I decided to use MicroPython for the first time. Sites around the web adviced me to use Thonny IDE, so I complied. I plugged my RP2040 touch screen board with BOOT button pressed and in a matter of seconds I could flash MicroPython right away. Pretty simple.

The first thing I did was copy the example code from Waveshare and strip it off the demo functionality. This way I obtained a neat driver for my board which I placed in separate file. My next step was date and time - it turns out the whole deal is done in a few lines of MicroPython:

from machine import RTC
import time

...

rtc = RTC()

def draw_ui(LCD):
    t = time.localtime()
    time_string = str(t[3])+':'+'{:02d}'.format(t[4])+':'+'{:02d}'.format(t[5])
    LCD.fill(0)
    LCD.write_text(time_string, 25, 105, 3, 65535)
    date_string = str(t[0])+'-'+'{:02d}'.format(t[1])+'-'+'{:02d}'.format(t[2])
    LCD.write_text(date_string, 40, 140, 2, 65535)
    LCD.show()

while True:
    draw_ui(LCD)

LCD.fill(0)
LCD.show()

To me that short time necessary to get something to work was astonishing. Here I was holding a working watch done in five minutes. Happy with the result I eagerly started to give my code some structure. In the following steps I added panels which you can swipe up and down along with arrows showing whether there are more panels to swipe to. I tried to incorporate LVGL Python bindings to make nicer visuals, but my efforts went in vain, so I prioritized quick progress over doing things elegantly.

The biggest challenge was to utilize the pulse oximeter. There is a library called MAX30102-MicroPython-driver, but it has no ready example of $SpO_2$ measurement. I searched for some other solutions like MAX30105 drivers and even thought about running C pulse oximeter code on one core and MicroPython interface on the other, but in the end I settled for MAX30102-MicroPython-driver with modified example code. The modified parts were pin definitions, red LED measurement logging and some sampling rate values. I also added saturation measurement as follows:

R = sum(red_peaks[:][0]) / sum(ir_peaks[:][0]) * len(ir_peaks[:][0]) / len(red_peaks[:][0])
spo2 = 100 - 5 * R

Where red_peaks and ir_peaks are lists generated by peak finding function.

The last functionality I added to my firmware was level bubble. It is just a circle placed at scaled X and Y output of onboard accelerometer.

The whole code is available in my repository.

Pinout

There is not much to say about pinout. I connected the MAX30102's SDA to pin 26 and SCL to pin 27 of Waveshare board, although any GPIO pin pair would be ok after slight code change.

Case

I settled for 3D printed model of my making. It is made of PLA with 0,4 mm nozzle (available at my faculty). Fits very tightly.

At the time of writing the case is too short to fit all cables; this will be repaired as last-minute effort before deadline, the student way.

Closing remarks

The case needs some polishing when it comes to housing all the stuff, but overall effect is quite satisfactory for a project done in about one week. Sometimes the MicroPython code crashes, probably something related to pulse oximeter thread, but I had no time to repair it yet. For me this is, first and foremost, a platform to test new ideas and as such it suits its purpose well.

Things to do later: install straps, print more precisely, fix buggy code, add more software features, add a way to turn this thing off (shhh!).

After two years of flawless usage of my ThinkPad E15 gen-2 I ran out of space on its default 256 GB SSD drive. For christmas I bought myself a new shiny 1 TB M.2 SSD and appended it to my LVM setup. The setup was:

$ lsblk
NAME                     MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme1n1                  259:0    0 953,9G  0 disk
nvme0n1                  259:1    0 238,5G  0 disk
├─nvme0n1p1              259:2    0   476M  0 part  /boot/efi
├─nvme0n1p2              259:3    0   954M  0 part  /boot
└─nvme0n1p3              259:4    0 237,1G  0 part
  └─nvme0n1p3_crypt      254:0    0 237,1G  0 crypt
    ├─vg--1ThinkPad-swap 254:1    0  27,9G  0 lvm   [SWAP]
    ├─vg--1ThinkPad-root 254:2    0  37,3G  0 lvm   /
    ├─vg--1ThinkPad-var  254:3    0    14G  0 lvm   /var
    ├─vg--1ThinkPad-tmp  254:4    0   1,9G  0 lvm   /tmp
    └─vg--1ThinkPad-home 254:5    0   156G  0 lvm   /home

I followed this answer on Stack Exchange tailoring it to my needs. After booting up with SSD in place I opened GNOME Disks utility and checked the name and path to my new asset, in my case, /dev/nvme1n1 (visible in window titlebar upon selecting the drive). I also ran benchmark to find that the average read rate is 1,8 GB/s and write rate 1,0 GB/s. I also wanted to be able to unlock both drives at once, which I managed to do by using this tip.

I proceeded to execute the following commands based on Stack Exchange:

$ sudo cryptsetup luksFormat /dev/nvme1
$ sudo cryptsetup luksOpen /dev/nvme1
$ sudo -e /etc/crypttab

in it, I had one line about my old drive. I copied it and changed the name and UUID to match the new drive data like so:

nvme0n1p3_crypt UUID=85c7f21e-b237-4ba6-84c3-11e737968a67 none luks,initramfs,keyscript=decrypt_keyctl
nvme1n1_crypt UUID=81e37db2-a64e-47d6-ace0-b037416712c6 none luks,initramfs,keyscript=decrypt_keyctl

I took the UUID from the output of $ sudo blkid. The initramfs flag is very important; without it my new partition did not mount and the OS dropped to initramfs on every boot. I had to manually cryptsetup luksOpen it and exit to load into GUI. Solution was inspired by this problem. Then I did:

$ sudo update-initramfs -c -k all
$ sudo cryptsetup luksOpen /dev/nvme1n1 nvme1n1_crypt

and entered the new passphrase. Moving on:

$ sudo pvcreate /dev/mapper/nvme1n1_crypt
$ sudo vgs

This showed me that my volume group is vg-1ThinkPad. Next I executed:

$ sudo vgextend vg-1ThinkPad /dev/mapper/nvme1n1_crypt

I wanted to create aditional partition for data with very original name data (I will leave moving /home fot another time), so I created and formatted it to ext4:

$ sudo lvcreate -l 100%FREE -n data vg-1ThinkPad /dev/mapper/nvme1n1_crypt
$ sudo mkfs.ext4 /dev/vg-1ThinkPad/data

Finally I chose a mount point:

$ sudo mkdir /mnt/data
$ sudo mount /dev/vg-1ThinkPad/data /mnt/data/

and added appropriate line to /etc/fstab

/dev/mapper/vg--1ThinkPad-data /mnt/data            ext4    defaults   0    2

I installed keyutils from apt and updated initramfs again just in case and also run update-grub. And voilà. You have a fresh partition ready to be clogged with data.

Not your usual screen

I have to point out that epaper display does not work like normal LCD or OLED display you normally connect to Raspberry Pi. Instead of video driver, it comes with a library full of drawing routines you execute manually, like clear, draw, update and the like.¹ This is not the end of differences, though. You have to be more mindful of refresh rate as those heavily impact longevity of a display. Typically you do not want to refresh more often than something like twice a minute, and certainly not every second (Waveshare recommends 180 s). Another important thing to keep in mind is the full refresh cycle which consists of repeated flashes of black and white - it is required to fully clear the image and prevent ghosting, and is tied to how epaper displays work.

Trip to the source

There is no more fun than to do a knee-deep dive into the inner workings of a program. I want to know how vendor library does what it does in the included demo, and try to replicate the effect. Here is a little breakdown of my excursion.
For starters I cloned the repository and explored the directory structure. I am interested in RaspberryPi_JetsonNano/c/, and especially RaspberryPi_JetsonNano/c/examples/EPD_7in5_V2_test.c. The file includes EPD_Test.h and EPD_7in5_V2.h. The latter is located in RaspberryPi_JetsonNano/c/lib/e-Paper/; it consists of width and height #defines and declarations of what seems to be the core functionality. It also includes DEV_Config.h, which declares GPIO pins used by the driver.
I need to know how to communicate with the screen. A quick glance at DEV_Config.c sufficed to find a very interesting part:

#ifdef RPI
    EPD_RST_PIN     = 17;
    EPD_DC_PIN      = 25;
    EPD_CS_PIN      = 8;
    EPD_PWR_PIN     = 18;
    EPD_BUSY_PIN    = 24;

Now I know where to tap into the hardware to send my commands.

Inspired engineering

In terms of commands in question I settled for epd-waveshare Rust library. I am sure rewriting the whole Waveshare library would be lots of fun, but I have to focus on making the project in finite time. I imported linux-embedded-hal and embedded-graphics and began writing code.
The first thing I had to do to use the display is to define all pins. I sifted through all example code I could find, and, coupled with the documentation and pinout, I came up with something like this:

let mut spi = Spidev::open("/dev/spidev0.0")?;
let options = SpidevOptions::new()
    .bits_per_word(8)
    .max_speed_hz(10_000_000)
    .mode(spidev::SpiModeFlags::SPI_MODE_0)
    .build();
spi.configure(&options).expect("spi configuration");

let cs_pin = Pin::new(26);
cs_pin.export().expect("cs_pin export");
while !cs_pin.is_exported() {}
cs_pin
    .set_direction(Direction::Out)
    .expect("cs_pin Direction");
cs_pin.set_value(1).expect("cs_pin Value set to 1");

let busy = Pin::new(24);
busy.export().expect("busy export");
while !busy.is_exported() {}
busy.set_direction(Direction::In).expect("busy Direction");

let dc = Pin::new(25);
dc.export().expect("dc export");
while !dc.is_exported() {}
dc.set_direction(Direction::Out).expect("dc Direction");
dc.set_value(1).expect("dc Value set to 1");

let rst = Pin::new(17);
rst.export().expect("rst export");
while !rst.is_exported() {}
rst.set_direction(Direction::Out).expect("rst Direction");
rst.set_value(1).expect("rst Value set to 1");

let mut delay = Delay {};

let mut epd = Epd7in5::new(&mut spi, cs_pin, busy, dc, rst, &mut delay)?;
let mut display = Display7in5::default();

Which is not basically the code copied from the examples. Yeah, totally. The wiki thankfully lists all the pins - I made sure to read the BCM2835 column and confirmed that indeed my HAT and HATs in the examples use the same pins.
To make something happen I invoke:

epd.wake_up(&mut spi, &mut delay)?;

then I render blank screen:

epd.clear_frame(&mut spi, &mut delay)?;
epd.display_frame(&mut spi, &mut delay)?;

Cool. Now I want to see something on the screen. I do the most basic thing I could possibly imagine - draw a line:

let _ = Line::new(Point::new(0, 120), Point::new(0, 295))
    .into_styled(PrimitiveStyle::with_stroke(Black, 1))
    .draw(&mut display);

and update:

epd.update_frame(&mut spi, &display.buffer(), &mut delay)?;
epd.display_frame(&mut spi, &mut delay)?;

and, after exhausting day, I sleep:

epd.sleep(&mut spi, &mut delay)?;

I tried to replicate the overall structure of examples from Waveshare, especially the explicit sleep-wake up commands. This is where I did a test. I connected the HAT to Raspberry Pi, display to the HAT, and scp'd my hot fresh binary.

Troubles

The process stopped at wake up call. After a while, the display went through flickering refresh twice, then stopped at update_frame. Another moment passed before the display flickered, painted white line on black background and the program exited. When I commented out wake_up, the proces got stuck on clear_frame and everything else was as before.
It looks like the communication between Pi and screen is horribly slow. Besides that, I thought the colors would be inverted (black line on white background). I had to make more digging.
The issue with horrendous delay appears to be a known thing, the same with inverted colors. Well, at least I know where the problem lies. After some more time spent on GitHub I found this, which intrigued me very much. This fix sends data blocks instead of sending data bit by bit. Maybe the answer was right before my eyes. I instructed Cargo to download crate from commit 97425d5 and made necessary changes in my code to reflect changes to the interface. I compiled and ran the code only to see almost the same thing. The screen does not show the line (although the background is white, so that is a partial win), but has no delay on update_frame. The delay on clear_frame persists and lasts about 30 seconds.
Then I found the culprit when it comes to the line not showing up. I deduced it might have something to do with the color problem not being really solved and boy, was I right. The colors are still messed up in a sense that the background works ok, but I was drawing white line on white background. After changing Black in .into_styled(PrimitiveStyle::with_stroke(Color::Black, 1)) to White, a black line shows up on a white background.
The next day I figured what was taking so long - it was clear_frame command. The update_frame and display_frame work fine. Also, the display_frame after clear_frame is redundant. Leaving the problem for the next day works wonders! Although I found the culprit, it did not really solve my problem. Clearing frame is essential, but is exceptionally slow, unacceptably so.

Words

Rendering graphics seems to be working fine, so this was time to test the essential part of ebook reader - text display. Text generated by the embedded-graphics, however, is fitted in just one and can go beyond the display. From now on I will be using embedded-text crate instead of embedded-graphics text utilities because it offers textboxes with wrappable text out of the box.
I had to use version 0.5.0 because newer version conflicts with version 0.7.0 of embedded-graphics which I have to use because of many errors I get when using it with epd-waveshare. The compiler gives me a warning about some code that will be deprecated in future releases of Rust, but I do not care about it at the moment, I want something that works just now. By the documentation, I create a text variable holding some text (Lorem ipsum in my case), and then paste the magical lines:

let text = "Lorem ipsum...";
let character_style = MonoTextStyle::new(&FONT_10X20, Color::White);
let textbox_style = TextBoxStyleBuilder::new()
    .height_mode(HeightMode::FitToText)
    .alignment(HorizontalAlignment::Justified)
    .paragraph_spacing(6)
    .build();
let bounds = Rectangle::new(Point::zero(), Size::new(480, 0));
let text_box = TextBox::with_textbox_style(text, bounds, character_style, textbox_style);
let _ = text_box.draw(&mut display).unwrap();

The above code feels pretty self-descriptive, but I will write an overview anyway: character_style declares the font size and color of pixels of text; textbox_style sets preferences - how to fit the text in a textbox, how to align it and the paragraph spacing; bounds is a bounding box of the textbox - height 0 works because I² set a FitToText mode, meaning the textbox will vary its height to the contents; text_box is the thing I will actually pass to the display in the next line. This works like a charm. To make the text more book-like, I changed Justified to Left. Unfortunaely, embedded-graphics does not provide any variable-width fonts. Fortunately, I discovered u8g2-fonts crate, which supports non-monospace fonts in embedded environments. I spent too long figuring out how to use these two together, so without bothering you with a relation of my frantic journey, I will just show you how I did the thing.
I imported u8g2_fonts with u8g2-fonts = { version = "0.2.0", features = ["embedded_graphics_textstyle"] } in Cargo.toml - Version 0.2.0 works for me, whereas newer version does not; the feature I am enabling is needed to allow the use of U8g2TextStyle embedded-graphics compatibility layer. The modification of previous code is as follows:

let font = u8g2_fonts::fonts::u8g2_font_lubR14_tf;
let character_style = u8g2_fonts::U8g2TextStyle::new(font, Color::White);
let text_box = TextBox::with_textbox_style(text, bounds, character_style, textbox_style);
let _ = text_box.draw(&mut display).unwrap();

In a nutshell, I swapped character_style to one from 8ug2-fonts. I used the u8g2_font_lubR14_tf font because it looks very non-monospace'y for the sake of presentation. Text display - check!

What next

I know how to control the display, which is a huge step forward in my journey towards DIY ebook reader. For the next part I am planning to master the button input and maybe do some battery-level-dependent behavior. I am leaving the screen clearing problem for another day.

Footnotes

¹ Yes I am aware that at some level every video driver offers these things; what I wanted to say is that by default you cannot use it like normal screen and, say, install a DE and use it like normal *nix desktop.
² Or the documentation actually

Body and mind

Since I have now acquired all the hardware I wanted, I can learn to use it and make something great. Before I delve into designing interface, I must first install required software and check the physical layer for any issues.

Prepping Raspberry Pi

Raspberry Pi Zero W is an SBC which supports full operating systems, in contrast to other popular hobbyist platform Arduino. The usual way of doing things is to install Raspberry Pi OS (formerly Raspbian; I liked that name better, shame they changed that), an official dedicated Debian GNU/Linux based distro with a wealth of programs available in its repository. This is an excellent choice for people who do not wish to spend hours fiddling with stuff and who do not care about conserving resources to the extreme while sacrificing their sanity. Obviously this is not me, so I started looking for other things to put in my Pi.
The first thing to come to my mind was BSD. This family of OSes is not exactly known to be the first choice for Raspberry Pi, and I have never actually encountered a project using one. I have never used BSD before, so I was thrilled by the prospect of learning alien system in, I believe, quite niche setup. But then I began to worry about driver support - I have custom hardware to connect, and vendor libraries are mostly written with Linux in mind. The last straw was a table in wiki which claims that the WiFi in Zero W is unsupported.
Going back to the Linux world, I still did not give up on using something different than Raspbian Raspberry Pi OS. Looking for other options I stumbled across Arch Linux ARM. I have never really used Arch before, so the feeling was similar as before. However, doubts started to arise - am I ready to take up all those hardships of learning different distro, hardware control and electronics all at once? Debian-based distros are familiar to me, so at least if matters go haywire I am not stuck fighting with a system, especially package manager.
Defeated, I started seeking Raspbian Raspberry Pi OS derivatives that are not Raspberry Pi OS. From my earlier days of Raspberry Pi experiments I remembered DietPi, a lightweight spin on Rasp Raspberry Pi OS with an optimized set of programs. I decided to give it a go, because it scratches that minimalist itch of mine.
The whole process of configuration is unremarkable. I flashed the SD card and edited the diepi.txt and dietpi-wifi.txt files to connect the Pi to the Internet on first boot, just as instruction told me to. I then put the card in and proceeded to sweep the network in search of the device, so that I can connect through SSH and perform headless install. This process took longer than anticipated, because In my impatience I aborted the process several times thinking I missed something in the config. When I finally found the address, I connected and changed the passwords and such.

Environment

Now that I had a working platform, I could start thinking about the software stuff.
In terms of my language of choice, I picked Rust - my new favorite toy in programming. Rust is well suited for embedded programming, and work with its package manager Cargo is a breeze.
In terms of gathering drivers and libraries, I began with epaper display. According to the wiki I had to enable SPI interface, which is hidden in Advanced Options of dietpi-config. Then I installed bcm2835 library used by Waveshare's demo code, which I might later cannibalize for screen control.
Button SHIM comes with a Python library. I refuse to use Python outside scientific work, and especially in low-level applications, so I started looking for Rust alternatives. I found something just for this purpose and thus looked no further.
To use the UPS HAT I just had to enable I²C interface.

Testing hardware

Display

The first component I wanted to test was the epaper display. Thera are two ways of connecting it to the main board: the first and arguably the easiest is to put a HAT on Raspberry Pi, the more challenging way is to use the 8PIN cable that comes with the screen and manually plug in the cables onto header. For the purpose of checking if things are working all right I went the HAT way, because I do not want to risk frying the screen just yet. I connected the flat cables between boards and display and was ready to go.
The demo code I downloaded from Waveshare's site is a bit messy, but as they stated somewhere in the files, that is because the library is written for all models they sell and writing redundant code would be pointless. Instead, provided Makefile composes the right demo for chosen display model during compilation. Per included readme, I executed:

# make EPD=epd7in5V2

Make created epd executable in the working directory. When I ran it (with root), the display started flashing black and white and spat out different test pictures. I was quite impressed and satisfied with what I saw. Epaper display - check!

UPS

Raising difficulty bar by one level, I proceeded to test the power supply. I want to power my Pi with Akyga LiPol I purchased, which comes bundled with JST-BEC female connector with dangling cables for self-assembly. The board has no JST-BEC socket, but has two soldering pads for custom wiring. I proceeded to solder connector to the board. Whew! I get anxious when I solder so close to the delicate electronics, let alone power supply. Solders passed the test of strength, which made me glad. Look at these huge pins! Anyway, I put the HAT on Raspberry Pi and connected the battery. I clicked the button, the battery indicator on HAT flashed bright white and... nothing. I held the button a little longer, the battery indicator flashed bright white and... there it is! The Raspberry Pi green diode switched on. I successfully logged into Pi through SSH. As I skimmed through the Net I read somewhere that the HAT allows for gentle shutdown when the battery is almost depleted, which is really nice.
UPS HAT - check! Moving on.

Buttons

The trickiest part to test was the button SHIM, because the only way to connect it is to solder it. As I stated in the previous post, I want it connected to the underside of Raspberry Pi. I do not know whether it is going to bite me in the bottom later, but now it seems to be a great idea. pinout.xyz said I needed to solder pins 3, 4, 5, 6, 17. After I soldered everything I was mortified - I had it upside down! Thankfully I did not have to unsolder, just make more solders and this time in correct places. I know, they are not pretty, but I moved forward. As you can see, I tried to unsolder the wrong ones, but at the end of the day it does not matter. To test the buttons I had to set up Rust on my main machine to cross-compile to Raspberry Pi.

Rust for Raspberry Pi

Setting up Rust and compiling on Pi would take too long to be worth it. To cross-compile from x86_64 machine to Raspberry Pi Zero W, first I had to add a new target:

$ rustup add target arm-unknown-linux-gnueabihf

I had no luck with a toolchain provided by my distro's repository, so I had to grab one from here and put it in my project directory under rpi_tools. Then I created file .cargo/config with the following contents:

[build]
[target.arm-unknown-linux-gnueabihf]
linker = "rpi_tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc"

In this file I specified the custom linker I just downloaded. Rust is now ready to compile for Pi.

To get a quick glance over the operation of SHIM I used the rainbow.rs example included with mzyy94's library. I compiled the file issuing:

$ cargo build --release --target=arm-unknown-linux-gnueabihf

and scp'd it to my Pi. The LED of Button SHIM changes color with button presses as intended. Buttons - check!

Conclusion

I managed to successfully test all hardware without problems. With Rust set up to producing ARM binaries I am ready to write software for this device.
In the next step I will be trying to make the display show whatever I order it to. I will also decide how I want to interact with the finished contraption, i.e. whether I should write a whole custom program a'la Kodi and encapsulate all functionality within it, or to make it more like a Linux box with access to the underlying OS.

Let's build

When I digged through ebook readers available on the market, it occured to me that every option is unappealing to me due to lack of customizability. Huge part of it lies in closed-source nature of those devices, which to me as an open source enthusiast is a major turn off. This, coupled with a hunk of free time during summer break, led me to the idea of building my own ebook reader. Happy with having a new project to hone my soldering skills, I enthusiastically set about making preparations.

Goals

To instill some sense of direction into the project, I imagined it would be best to try and reach incremental goals. My first goal would be to connect every element one by one and check if every part works as intended. Next, a stage which will either make or break the whole endeavor, is to put everything together and make sure there are no conflicts between parts. And, for the last part, find some ebook software to actually use the thing, or, in the worst case, write one.

The innards

Core

I want my contraption to be both functional and similar in form to commercial counterparts. It has to be lightweight and handy, while allowing a degree of flexibility in customization. Since I have no infrastracture, means and skills to manufacture custom boards, I am bound to SBCs, which is not at all bad considering saved money and time. Ebook, like tablet or smartphone, is rather slim, which means very limited range of connection ports. There is no space and energy to spare for active cooling, so the chip has to run cold. Frankly, it does not have to be as powerful as aforementioned devices, because its primary purpose is to print text. Taking all that into consideration my vote went to Raspberry Pi Zero W. Arguments for this pick were my familiarity with Raspberry Pis, wealth of documentation and code, a range of ready-made extensions and ease of use. Integrated Wi-Fi allows for smooth hassle-free software deployment while also being a nice feature of the finished project. Model Zero is also much thinner and energy efficient than its full-sized siblings, making it a perfect fit for my application. Another chief asset of Zero is that it is powerful just enough to run a full Linux distribution, so no need to play with low level embedded. I would gladly take version 2, but sadly, they were out of order wherever I looked, and summer break is too short to wait. For convenience, I settled for Raspberry Pi Zero WH, with soldered header.

Power

Much to nobody's surprise, there are myriad ways to power Raspberry Pi. Obviously, battery power is the only sensible choice, but even that can be done in many ways.
The most barebones solution is to construct appropriate circuit and plug in the source. The upside of it is being able to custom tailor every aspect of power supply. The downside is required knowledge and ease of screwing things up. I never feel confident enough in my tinkering abilities when it comes to powering things, so I immediately discarded this option.
Another popular solution is to cannibalise power bank. This is handy, as powerbanks come with charging module, so you just plug in the cable and you are set. However, powerbanks are often a little bulky and poorly regulated, which would cause a lot of random restarts upon voltage drops. After all, they are meant to store energy, not act as a power supply, so manufacturers could not care less about stability. Another missing thing is feedback - powerbanks do not return their status to devices they power, so going below certain threshold would always cause abrupt shutdown and, over time, murder the SBC.
This leads me to the third option, which is dedicated Raspberry Pi power supply board. These little circuits provide stable voltage, have integrated charger, output battery level and play nicely with Pi, due to the whole HAT design which enables stacking of extension boards.
After quite some time spent on researching the topic and looking through offers I chose DFRobot DFR0528 UPS HAT. It fulfills all my requirements, is quite cheap and available. The official wiki seems to nicely explain how to enable battery level reading.

When it comes to power source, the obvious choice would be either Li-Ion or Li-Pol battery. I wanted something thin, light and capacious, so I chose Li-Pol Akyga 3500 mAh 1S 3,7 V which I found in local eshop. The battery comes with connector cable ready to be soldered to UPS HAT.

Screen

From the beginning it was clear that I want e-paper dispay. E-paper is easy on eyes, legible in sunlight, extremely power efficient and looks cool. In this day and age there is an array of different sizes of e-papers made for Raspberry Pi. Since I want to build a reader, I need quite a lot of surface area, but the bigger the surface, the slower the refresh rate. This is further exacerbated when different colors come into play, where refresh procedure can take up to thirty seconds. This is unacceptable to me, so I decided I am content with fast but b/w display. Finally, I found a nice compromise, Waveshare 800x480 7.5inch E-Ink Display which comes with a HAT. The screen communicates through SPI, has a refresh rate of 5 s, resolution of 800 by 480 pixels and dimensions of 163,2 by 97,92 mm.

Buttons

I almost forgot that in order to actually do stuff with my ebook, I need to control it somehow. For a while I thought I would build something from parts from my local (and physical) shop, until I found Pimoroni Button SHIM, which seems perfect for my use case.

The design

This project takes on highly experimental approach, and I am not much of a planner when it comes to those things. I very much like to get my hands dirty first and construct something that works, then worry about the details (like how to actually put the parts in place). However, perhaps, if I want to write a few blog posts about my adventure, then the usual way of doing things is not very good for that. I should pave a roadmap and pull together some kind of schematic. Without further ado I fired up Blender and made models of my parts.
The first thing that is striking is that UPS HAT's pins are huge, sprouting 1 cm from both sides. This is non-ideal in my case, so I will have to unsolder that thing or find another way to slim down the board.
While the idea of HATs is amazing in its own right, stacking just one extension board on top of Pi makes the whole setup really thick. Luckily, every board uses just a few pins from the whole header, so the idea to connect appropriate pins with cables and position all boards horizontally is not completely bonkers. To further slim down the setup and to get better access to Pi pins, I chose to solder button SHIM to the underside. I had to keep all boards with buttons and ports at the edge and also remember not to strain the FFC cable too much. Taking all that into consideration I attempted to make a draft in Blender and am pretty happy with the looks of it.

The layout feels a bit off, because Raspberry Pi with its button shim are positioned to align the middle button with the center of the screen; likewise, e-paper connector is aligned with screen tape at the top (not rendered).

Closing remarks

After fruitful team participation in robot competition I felt like taking up a solo project this time. I wanted to actually learn the process of doing project from start to finish, but now on my own terms. Ebook reader involves just a few parts, does not require much soldering, and poses a great opportunity to get a hands-on experience in electronics.
In the next step I will install and configure the OS, set up development environment, install necessary libraries and test all acquired boards.