MadMike77 reshared this.
I got to follow rocket's journey to async and stable #rust with the PrivateBin directory service. Coming from Python flask apps, it is really easy to pick up and get going with your webservice, offering static & templated content, easy to create web forms and JSON APIs.
Thanks to rust's strict type system I could focus on the logic and didn't have to waste time double checking and casting data received by clients. If my API accepts an integer in a certain parameter, Rocket will ensure I only receive valid requests in my logic.
We are still looking for additional long term maintainers. What can we offer you in return for your help?
- We can offer you our mentorship, if this is your first time participating as a maintainer of an open source software project. We can guide you through submitting your first pull requests and work with you to ensure your change fulfills the communities quality standards, gets merged and makes it into a release.
- Your work gets publicly credited. This can help you build up a resume, showing off your growing skill set, in programming as well as your soft skills.
- PrivateBin is a smaller project. If you'd like to learn how to participate and contribute in an open source git project, this should be less overwhelming then larger projects.
- We do have a decent unit test code coverage, so it is an environment forgiving of mistakes. You may still introduce logical flaws or issues in new features, not yet covered in the tests, but you can rely on the tests preventing any regressions in other areas.
- It can be an opportunity to learn about continuos integration tools to automate tasks like tests, security scans, etc.
If you are interested in helping with any of these points, we have prepared a development guide including design goals, code structure and tools to get you started. For any questions, you can chat with the maintainers in the discussion area or reach us via email or here in the Fediverse.
Unit is a lightweight and versatile application runtime [and] was created by nginx team members from scratch [...].
There is a package for it and it's PHP 8.1 module in the stable Alpine Linux repositories (and soon also for PHP 8.2, when Alpine 3.18 releases). It's installed size is about half of the regular nginx package and because it is a single service one no longer needs a service manager in the container, to keep the two services alive.
For now I've documented these limitations in the forked repository and have started testing it first on a personal PrivateBin instance and since that worked well, now switched the main demo instance over to it. The HTTP port and all volumes still attach at the same places, the same UID & GID are used and both logs and included maintenance scripts work the same as before, so it is as close to a drop-in replacement as possible.
So far I've observed slightly higher RAM usage (223 MiB vs 190 MiB used, for a VM that also runs the nginx webserver, for TLS termination, serves the static project webpage and runs the PrivateBin directory application server, all in docker containers) and no observable difference in load (still idles at less than 1%). The images are all about 1 MiB less in compressed size (as displayed on the docker hub in the tags).
I'll probably start migrating my other PHP-fpm containers over to unit, but I'll probably wait for that Alpine 3.18 release, so I can switch to PHP 8.2 at the same time. It's a much simpler stack to maintain.
PrivateBin 1.5.0 released - Adding S3 Storage backend, storage migration script & 4 new translations
- a page with list of entries in a database, sorted by fixed criteria, for now
- a single JSON API endpoint to retrieve that same list, with optional filters
- a form to add an entry to the list
- a nearly static about page
- two tasks started via cron jobs, one running every few minutes and the other once a day
The rocket framework appealed to me because it offered a similar semantic to flask for python. It comes with several options for templating and state (aka database) integrations. I had picked tera templates, as they are near identical to jinja2 templates that are used all over python application (i.e. in pelican, ansible, flask, ...) and diesel for accessing an sqlite database. The rocket framework was already at version 0.4 when I started out with the project in 2020. It used a synchronous version of hyper, a low-level HTTP library to provide a multi-threaded HTTP server. It lets you compile the entire application into a single statically linked binary, excellent to run in a small "FROM scratch" container image.
In June 2021, the rocket 0.5 release candidate got published, and over the quieter days end of 2021 / beginning of 2022, I dug into upgrading the app. Let's start with the best bits about this process and the end result:
- I didn't need to do any changes to the templates, the CSS, the database structures or queries. The rocket routing mechanism worked the same way as before.
- rocket 0.5 is now compiling on a stable rust compiler, where 0.4 required the use of the nightly compiler (meaning it used some unstable language features). It got rewritten to use asynchronous functions, which allows it to use the latest hyper library which leverages that mechanism on top of a multi-threaded runtime provided by tokio, an implementation of the green threads concept in Rust.
For this particular application, the average latency was reduced from 67ms to 60ms (as monitored by a zabbix proxy running an a host attached to the same subnet) and the daily cronjob duration got reduced from 52s, using multi-threading, to 22s, using async and the tokio runtime (the earliest, fully linear, implementation of that task took 20 minutes).
While I am happy with these clear improvements, they did come at some cost:
- In the end I spent about 7 workdays on this upgrade, spread over 2 weekends and several evenings - git tells me that this consisted of 6188 additions and 4231 deletions, including all the license changes.
- The switch to async/await in both rocket and hyper required that many formerly synchronous functions had to be made async as well, as they now needed to call async functions themselves. Rust would offer you the block_on construct to call async functions from sync ones, but that conflicts with the tokio runtime and causes it to fail. The same is true for spawning threads, which I had done extensively in parts of the code that run multiple HTTP(S) requests in parallel. Using tokio to let the async calls be handled automatically did pay off and is indeed faster then my manual optimizations, but the rewrite into async required pretty big code changes.
- The newer hyper version lost some features I had been relying on, like shorter connection timeouts (I replaced these with a timeout wrapper on the returned future and an ugly workaround to return an error) and IDN domain names (I now use the URL parser from servo, developed for Firefox).
- The unit tests also needed to be made async aware and tokio provides a dedicated macro for wrapping tests. Unfortunately I had to give up on one unit test that I just couldn't get to work under a tokio runtime, while the error couldn't be reproduced on the running service, either manually or scripted. It seems the tokio runtime acts differently when in a unit test, where it keeps telling me it can't launch a runtime from within a runtime. The stack traces seem to go remain outside any of my code, so doesn't even seem to enter any part of the unit test itself.
- Due to all these new and upgraded libraries the statically linked binary ended up growing (after stripping and upx-ing) from 2.16 MiB to 2.83 MiB. Sure, compared to a full PHP- or Python-stack it is still at least an order of magnitude smaller, but it's also an increase of 1/3.
I would still recommend rocket to anyone familiar with flask. Creating dynamic web pages or APIs is very straight forward to do. The issues I encountered with the switch to async would not apply to new projects anyway. For existing rocket v0.4 applications, the upgrade is worthwhile if you can use the extra performance or aren't doing weird multi-threaded things in the background, like me. None of the core rocket features broke, only the database abstraction required modest changes. My struggle was mostly with the use of hyper as a client.
For the database side, diesel isn't quite as elegant as ORMs in other languages. It makes it simple to retrieve (lists of) elements from a table and do simple joins. For more complex things you can fall back to writing manual SQL queries and mapping the result onto your structs for convenience (you can mark properties that aren't tied to specific table columns, so you can populate them from calculations on joined tables or such). The database migration mechanism also requires the use of plain SQL.
Shell parsing is hard.
Yes, shell parsing is non-obvious - it does help enormously to understand that the shell takes what you type on the command line after you hit enter, parses it, replacing variables, expanding globs (wildcards) and other language constructs in the process and only then issues a system call, passing the resulting argv structure to the kernel for execution.
Exhibit A (source of the above quote): How the local shell ssh and the remote shell interact, in unexpected ways
Exhibit B: skarnet's introduction to the execline language design and grammar goes into further details of the argv structure
Exhibit C: How to use execlineb for nginx to wait for up to 10s on the startup of php-fpm, avoiding involvment of a shell process
Ich gebe Geld dafür aus, meine eigenen Server zu housen und für die Registrierung meiner eigenen Internet-Domänen. Bisher konnte ich so frei entscheiden, ob ich da Werbung und Tracking darauf einbauen will oder nicht. Ich hatte mich stets dagegen entschieden. Das war einfach, ich musste somit auch nichts in meinen Content einbauen und pflegen. Und die Seiten sind erst noch schlanker und ich kann sicher sein, dass diese auch genau so ausschauen wie ich es will und nicht von Werbung verunstaltet werden.
FLoC kehrt dieses Prinzip um: Besucher meiner Webseiten werden ge-trackt, wenn Sie Chrome benutzen und zur Gruppe gehören bei denen das "Experiment" zufällig aktiviert wurde, und ich muss nun aktiv einen neuen HTTP header in alle meine Webseiten einbauen, damit die Besucher nicht getrackt werden! Und hoffen, dass Chrome sich auch daran hält.
Nein danke, ich will die Besucher meiner Webseiten keiner unnötigen Überwachung aussetzen. Schade, dass ich nun Aufwand treiben muss um dies zu erreichen, anstatt einfach keine Tracker oder Werbung einzubauen.
Started looking into gemini space. Love how it feels - it's like the web ca. mid-90s. UI is back under your control and you can focus on reading the content instead of getting the site to work (because either you have issues with noscript turned on and sites requiring JS to display text or you have it disabled and have to click through modal windows informing you of cookie settings, sign up for that newsletter, etc. to get to the content).
I'm using Castor and wanted to merry it to my Gnome desktop, so clicking links in Firefox/Chromium opens them in Castor. Oh, and I got a gopher client for free with it as well. Was bummed when Firefox dropped gopher support. Here's how to register the gemini protocol in Gnome (and build castor):
While grep and sed are commoly used, awk fills a valuable niche when processing structured text, avoiding multiple pipes or more complicated regex extractions. Here is a handy flowchart to pick the ideal tool for your text processing task:
A plaintext chart and a simple example making use of several awk features can be found here:
Maybe a concept we could evaluate for use in our fediverse software as well:
This forms a relative reputation system. As uncomfortable as it may be, one man’s terrorist is another man’s freedom fighter, and different jurisdictions have different laws - and it’s not up to the Matrix.org Foundation to play God and adjudicate. Each user/moderator/admin should be free to make up their own mind and decide which reputation feeds to align themselves with.
Die alte Anlage bestand aus zwei Pizza-Schachteln mit je Quadcore-Xeon, 24 GiB RAM and 4 x 1 TB SATA HDDs im Software RAID-5 und DRBD. Die neue Anlage besteht aus drei Einschüben, welche sich ein 2HE Chassis teilen mit je 2 x Hexacore Xeon, 32 GiB RAM und 3 x 500 GB SATA SSDs im Software RAID-5 und Ceph. Beide Anlagen konnten durch die Replikation des Storages live Migration und Hochverfügbarkeit für die VMs nutzen.
Die neue Anlage ist neben Gigabit Ethernet (einmal DMZ für die VMs und einmal LAN für die Clusterkommunikation) auch noch mit zwei Infiniband Adaptern ausgestattet (einer onboard und der andere as PCIexpress Karte). Damit habe ich noch ein drittes Mesh-Netzwerk gemacht, welches 40 Gigabit pro Link fÃ¼r die Storage-Replikation nutzen kann. Diese Bandbreite erreiche ich zwar mit iperf nicht (10 Gb/s auf dem den Infiniband-IP-Stack) aber die SSDs wÃ¼rden unter idealen Bedingungen sowieso hÃ¶chstens 3 x 500 MB/s, also 1,5 GB/s oder 12 Gb/s liefern. In der Praxis messe ich bisher 200 - 300 MB/s wenn ich z.B. VMs kopiere. Hat also noch Luft nach oben.
Leider ist einer der beiden alten Server vor einigen Monaten ausgefallen und mehrere Reparaturversuche ergaben, dass da was mit dem Mainboard nicht mehr stimmte. Nach so vielen Jahren Dauereinsatz verÃ¼ble ich das dem Server nicht und es war die Gelegenheit endlich mal Ã¼ber eine Neuanschaffung nachzudenken.
Einige der VMs (z.B. mein Minecraft-Server) lagen auf den lokalen, nicht-DRBD Partitionen des ausgefallenen Servers und waren daher auch nicht HochverfÃ¼gbar ausgelegt. Nachdem ich letzte Woche die neue Anlage im RZ eingebaut und alle VMs migriert habe, hatte ich endlich wieder Zugriff auf diese. Da diese monatelang ausgeschaltet waren, habe ich die erst mal grÃ¼ndlich durchgepatcht. Etliche VMs mussten noch von Debian Jessie auf Stretch upgegradet werden.
Was mich dabei geÃ¤rgert hat, war der hÃ¶here Arbeitsspeicher-Verbrauch. Etliche VMs liefen noch mit 256 MiB RAM und konnten auch problemlos geupgradet werden. Nach dem Reboot liefen Sie anfangs gut, aber je nach Anwendung schlug dann bald mal der OOM (out of memory) Killer zu. Aus Performance-GrÃ¼nden betreibe ich die VMs alle ohne Swap. Soweit ich das sehen konnte ergibt sich der hÃ¶here Verbrauch einerseits aus einem etwas grÃ¶sseren Kernel und dem Wechsel vieler Systemtools auf Python 3, welches scheinbar deutlich verschwenderischer mit dem Speicher umgeht als Version 2.
Was waren das noch fÃ¼r Zeiten, als eine VM mit 128 MiB RAM mehr als genug war um einen Bind oder apt-cacher zu betreiben.