Alpine Linux Hate Letter
Of all the things I do, running based.pt its probably the thing I have most fun doing, and also the most issues with. From scripts not working to forgeting to open the ports on the router, it seems like everything can stop working at any moment, leaving me completly lost on how to fix it.
I've gained lots of experience and knowledge simply fixing issues on my server, and despite this being my hobby, I am a software engineer student, it is my job to learn and it is easier to learn from experience. I belive the knowledge I get with this project will eventually be usefull on a job. But thats not why I have based.pt, I have it because its fun.
I've been dailing driving Linux distributions since highschool, I am 20 years old now, so you could say I am experienced with it, right? I've used arch, gentoo, debian, and alpine, both on my PC and my server. I've fixed countless issues, caused by bugs or missuse of tools on my end. Over the years I've been getting less issues and fixing them faster too, which is normal considering I have more knowledge.
From all the Linux distributions I have used, I NEVER HAD SO MANY FUCKING ISSUES, FUCK YOU ALPINE LINUX!
I Hate Alpine Linux
Alpine lured me into using it for it being small, simple and secure (as it says on their website). Because of these claims I assumed it was also a stable distribution, I mean, it uses the lts kernel and has stable releases (currently using 3.22).
One of the things that kept me from using debian was the lack of openrc, and the packages being really old, but I guess now I wouldnt mind older packages, because it seems that everytime something updates on alpine, it breaks some package.
Just now, after I got my UPS which would made my server extremly stable and have great uptime. I decide to update the server since im going to shutdown and start it again. Well, when I do that the server doesn't fucking start. I dont have a monitor on my computer so if the ssh doesnt start, im locked out, which is extremly frustrating.
Not only that, but after that the server wouldnt even mount the root partition, podman would mount /var/lib/containers/storage on it so i had to mount it manually on every boot. I would try to guess at what stage the boot was by looking at the SSD's LED, and then typing the comand to mount the /sysroot, afeter that the LED would start blinking fast I would assume it worked. Well after this update the issue with the boot was gone (magically), but of course it still doesnt boot
Fixing the issue
I had to install a GPU on the server and a monitor to see what was preventing the system from booting. It had something to do with Podman, containers wouldnt start and the boot wouldnt continue (weird issue), I also dont understand why i couldnt ssh since im pretty sure the networking service is on boot while the script that start the containers is on default. But anyways I took out the SSD on alpine and pluged it into my computer, CHROOTed into it and disabled the service.
After this the server started but obviously not my containers, that would be an issue for later.
I know, this seems like a quick process but the issue happened right before I to catch the bus so it took me 6 hours to get to it, 1 hour researching similar issues on the internet to understand what caused it, then another hour fixing. It took me a while because I was actually trying to fix the issue with the root partition first which led me to belive i had fucked up something, thus not working. But no, it was another issue that was affecting podman and preventing the containers from starting.
More issues (Alpine is cursed)
Remember when I said i had to plug the server SSD into my computer to fix the issue? Well guess what happened after that. My PC wouldnt boot either!!!!
I was stunned after I boot my PC and it said something about having to choose a valid OS or something (the message when you dont have a bootable drive). Well, another 30 minutes wasted getting it to work, I formated my boot partition, reinstalled grub and reinstalled the kernel (to regenerate the initramfs and the grub config)
Fixing the containers
So after i ssh into my server i tried to start up a container. It had some kind of error with netavark, a rust based network stack for OCI containers (mainly podman). So this error would say something like "netavark error: error netlink: error bridge" (something like that, I dont remember specifics) This imedialy seemed like some kind of driver issue, i reinstalled both bridge-utils and netavark (at this poing i have no idea what im doing). I was so desperate i deleted podman, installed docker and had the same issue but with some better info, I remembered theres a kernel module called "bridge" which is used by podman. I deleted docker, installed podman and adde "bridge" to /etc/modules
Contaiers work again afetr a reboot
It's my failt
Well this was one big time waste, obviously im the one to blame for this. I choose Alpine Linux whithout looking much into it, i had basically 2 checkboxes: it runs podman, its openrc. This alone explains alot. I had assumed Alpine Linux was super stable and i would fix issues really fast because i have backups and shit. Well backups only fix stuff if ur stuff is deleted or corrupted, which was not the case.
I had never prepared for updates that would break my current stuff.
This is not the first time, last time, it was my email, postfix had changed the email syntax and it wouldnt change automaticly, I had to migrate from a specific version that alpine did not have (ofc i only found about that later), after that i would check the alpine issues after updating. But i could never predict the kernel would come with a missing loaded module.
I also still dont know why podman was mounting its contaienrs on the root partition before the root system... And i actually only found about that 3 days ago despite having the problem for months.
Maybe if i did more research before updating and kept a log of things that happen on the server, i could fix things even faster.
Will I Change My OS?
Well that could be an option...
Debian is an easier distro to work with. It updates really slowly and seems to have a lot more testing done.
Another alternative could be Gentoo. It has great documentation and I have alot more experience with it. But updataing could slow down the server.
I would probably choose Gentoo as the new OS, but we will see, ill keep using Alpine for a while.