Reclaiming the Net with a Private Reader
Background
Final yr I skilled the all-too-common profession burnout. I had a few unhealthy initiatives in a row, sure, however extra typically I used to be disillusioned with the software program trade. There appeared to be a disconnection between what I used to love concerning the job, what I used to be good at, and what the market needed to purchase from me.
I did the standard factor: I slowed down, give up my job, began remedy. I revised my habits: eat higher, train, meditate. I attempted to steer clear of programming and software-related studying for some time. As a result of I didn’t just like the impact it had on me, but additionally inspired by its obvious enshittification, I give up Twitter, the final social media outlet I used to be nonetheless plugged into.
Not working was one factor, however overcoming the productiveness mandate —the sensation that I needed to make the very best of my time without work, that I used to be “recharging” to make a comeback— was one other. As a part of this detox interval, I learn How one can Do Nothing, a form of creative manifesto disguised as a self-help ebook that offers with a few of these points. The writer Jenny Odell mentions Mastodon when discussing various on-line communities. I had heard about Mastodon, I had seen some colleagues transfer over there, however by no means actually checked out it. I assumed I ought to give it a strive.
I seen a number of issues after some time on Mastodon.
First, it felt refreshing to be again in charge of my feed, to obtain strictly chronological updates as a substitute of getting an algorithmic intermediary making an attempt to promote me stuff.
Second, many individuals had been going by means of the same course of as mine, one in every of discomfort with the software program trade and the Net. A few of them had been looking back on the previous occasions for inspiration: enjoying with RSS, Bulletin Board Methods, digital gardens, and webrings; some imagined a extra open and independent Net for the long run.
Third, not solely wasn’t I considering micro-blogging myself, however I didn’t take care of many of the updates from the individuals I used to be following. I spotted that I had been utilizing Twitter, and now Mastodon, as an data hub somewhat than a social community. I used to be following individuals simply to get notified once they blogged on their web sites; I used to be following bots to get content material from hyperlink aggregators. Mastodon wasn’t the appropriate instrument for that job.
Issues clicked for me after I realized concerning the IndieWeb motion, notably their notion of social readers. Attempting Mastodon had been good, however what I wanted to reconnect with the great aspect of the Net was a feed reader, one I may alter arbitrarily to my preferences. There are many nice RSS readers on the market, and I did briefly strive a number of, however I knew this was the right excuse for me to get again in contact with software program growth: I used to be going to construct my very own private reader.
Objectives
As a person, I had some concepts of what I needed from this undertaking.
Relatively than the e-mail inbox metaphor I’ve mostly seen in RSS readers, I needed one thing resembling the Twitter and Mastodon house feed. That’s: as a substitute of a backlog to clear daily, a stream of attention-grabbing content material each time I opened the app. The feed would come with articles from blogs, magazines, information websites, and hyperlink aggregators, combined with notifications from private accounts (Mastodon, Goodreads, GitHub). The parsing ought to be customizable to make sure a constant feel and appear, impartial of the form of the information every supply printed.
I wasn’t considering implementing the “social” options of a fully-fledged indie reader. I didn’t thoughts opening one other tab to remark, nor having my “content material” scattered throughout third-party websites.
That’s what I began with however, as soon as I had the fundamental performance in place, I deliberate to drive growth by what felt proper and what felt lacking as a person. My short-term purpose was to reply, as quickly as attainable, this query: may this finally change into my major —even my sole— supply of data on the Net? If not, I’d drop the undertaking straight away. In any other case, I may transfer on to no matter was lacking to comprehend that imaginative and prescient.
As a developer, I needed to check a few of the concepts I’d been ruminating on for over a yr. Though I hadn’t but formulated it in these phrases, I needed to use what I expressed in another post as: person > ops > dev. This meant that, when prioritizing duties or making design trade-offs, I might select ease of operation over growth comfort, and I might put person expertise above all the pieces else.
Since this was going to be an app for private use, and I had no intention of turning it into the rest, placing the person first simply meant dogfooding: placing my “person self” —my wants— first. Even when I finally needed different individuals to strive the app, I thought that I had a greater probability of constructing one thing helpful by designing it ergonomically for me, than by making an attempt to fulfill some preferrred person. It was essential to me that this didn’t flip right into a studying undertaking or, worse, a portfolio undertaking. This wasn’t about productiveness: it was about reconnecting with the enjoyment of software program growth; the pleasure wouldn’t come from constructing one thing however from utilizing one thing I had constructed.
Assuming myself as the one audience meant that I may postpone no matter I didn’t want straight away (e.g. person authentication), that I may sort out overly-specific options early on (e.g. ship to Kindle), that I may assume programming information for some situations (e.g. feed parser customization, Mastodon login), and so on.
Design
Provided that psychological framework, I wanted to make some preliminary technical choices.
Person Interface
Though this was going to be a private instrument, and I needed it to work on a local-first setup, I knew that if it labored properly I’d wish to entry it from my telephone, along with my laptop computer. This meant that it wanted to be a Net software:
- it was the cost-effective method to have a single interface work for each units,
- it allowed me to make use of HTML and CSS (the UI know-how I’m most acquainted with),
- it solved gadget syncing by having the state saved on the server.
I needed the Net UI to be considerably dynamic, however I didn’t intend to construct a separate front-end software, study a brand new front-end framework, or re-invent what the browser already supplied. Following the boring tech and radical simplicity recommendation, I seemed for server-side rendering libraries. I ended up utilizing a mixture of htmx and its companion hyperscript, which felt like choosing Net growth up the place I’d left off over a decade in the past.
Structure
Making the app ops-friendly meant not solely that I needed it to be simple to deploy, however simple to arrange regionally, with minimal infrastructure —not assuming Docker, Nix, and so on.
A “correct” IndieWeb reader, at the very least as described by Aaron Parecki, must be separated into parts, every implementing a unique protocol (Micropub, Microsub, Webmentions, and so on.). This setup enforces a separation of considerations between content material fetching, parsing, displaying, and publishing. I felt that, in my case, such structure would complicate growth and operations with out shopping for me a lot as a person. Since I used to be doing all the event myself, I most popular to construct a monolithic Net software. I selected SQLite for the database, which meant one much less element to put in and configure.
Along with the Net server, I wanted some method to periodically ballot the feeds for content material. The only possibility would have been a cron job, however that appeared inconvenient, at the very least for the native setup. I had used process runners like Celery prior to now, however that required including a few additional parts: a shopper course of to run alongside the app and one thing like Redis to behave as a dealer. Might I get away with working background duties in the identical course of as the applying? That largely relied on the runtime of the language.
Programming language
At the least from my superficial understanding of it, Go appeared like the very best match for this undertaking: a easy, general-purpose language, garbage-collected however quick sufficient, with a strong concurrency mannequin and, most significantly for my necessities, one which produced easy-to-deploy binaries. (I later learn a similar case for Golang from the Miniflux writer). The large drawback was that I’d by no means written a line of Go, and whereas I understood it’s a reasonably accessible language to choose up, I didn’t wish to lose focus by turning this right into a studying undertaking.
Among the many languages I used to be already fluent in, I wanted to decide on the one I anticipated to be best with, the one that permit me construct a prototype to determine whether or not this undertaking was value pursuing. So I selected Python.
The unhealthy aspect of utilizing Python was that I needed to cope with its setting and dependency quirks, notably its reliance on the host OS libraries. Moreover, it meant I’d need to get inventive if I needed to keep away from additional parts for the periodic duties. (After some analysis I ended up selecting gevent and an extension of the Huey library to run them inside the applying course of).
The great aspect was that I bought to make use of nice Python libraries for HTTP, feed parsing, and scraping.
Testing (or lack thereof)
I made a decision to not hassle writing checks, at the very least initially. In a way, this felt “soiled”, however I nonetheless suppose it was the appropriate name given what I used to be making an attempt to do:
-
Since I used to be going to experiment, including, eradicating, and rearranging options, the price of sustaining unit checks would outweigh their worth. I didn’t thoughts introducing little logic bugs; I used to be going to make use of the app myself anyway, so I anticipated that the majority important bugs would simply floor over time.
-
In my expertise, integration checks are those that present probably the most worth when it comes to confidence that the applying works as anticipated. Extra so for this undertaking, the place the majority of the work (and nearly all of the bugs) got here from interacting with exterior sources and from the UI. However, whereas I may have caught some bugs earlier and prevented some regressions if I had integration checks in place, implementing them required an effort that simply wasn’t value it upfront.
Improvement
There’s a sort of zen stream that programmers unblock once they use their software program every day. I don’t imply simply testing it however experiencing it as an finish person. There’s no higher catalyst for concepts and experimentation, no higher prioritization driver than having to face the bugs, annoyances, and limitations of an software first-hand.
After some trial and error with totally different UI layouts and options, a utilization sample emerged: open the app, scroll down the principle feed, pin to learn later, open to learn now, bookmark for future reference.
I made a decision early on that I needed the choice to learn articles with out leaving the app (amongst different issues, to keep away from paywalls and consent popups). I attempted a number of Python libraries to extract HTML content material, however none labored in addition to the readability one utilized by Firefox. Because it’s a JavaScript package deal, I needed to resign myself to introducing an elective dependency on Node.js.
With the fundamental performance in place, an issue grew to become obvious. Even after curating the checklist of feeds and punctiliously distributing them in folders, it was laborious to get attention-grabbing content material by simply scrolling objects sorted by publication date: occasional weblog posts would get buried behind Mastodon toots, journal options behind every day information articles. I wanted to make the sorting “smarter”.
Contemplating that I solely adopted sources I used to be considering, it was secure to imagine that I’d wish to see content material from the least frequent ones first. If a month-to-month publication got here out within the final couple of days, that ought to present up on the high, earlier than any micro-blogging or every day information objects. So I categorized sources into “frequency buckets” and sorted the feed to point out the least frequent buckets first. Lastly, to keep away from this “rare content material” sticking on the high each time I opened the app, I added a function that robotically marks entries as “already seen” as I scroll down the feed. This fashion I at all times get contemporary content material and by no means miss “uncommon” updates.
At first, I left the app working on a terminal tab on my laptop computer and used it whereas I labored on it. As soon as I seen that I appreciated what was displaying up within the feed, I arrange a Raspberry Pi server in my native community to have it obtainable on a regular basis. This, in flip, inspired me to enhance the cell rendering of the interface, so I may entry it from my telephone.
I finally reached some extent the place I missed utilizing the app after I was out, so I made a decision to deploy it to a VPS. This pressured me to lastly add the authentication and multi-user assist I’d been suspending and allowed me to provide entry to a couple pals for beta testing. (The VPS setup additionally inspired me to purchase a website and arrange this web site, getting me nearer to the IndieWeb preferrred that impressed me within the first place).
Conclusion
It took me about three months of (relaxed) work to place collectively my private feed reader, which I named feedi. I can say that I succeeded in reengaging with software program growth, and in constructing one thing that I like to make use of myself, daily. Removed from a completed product, the undertaking feels extra like my Emacs editor config: a perpetually half-broken instrument that may nonetheless change into second nature, laborious to justify from a productiveness standpoint however fulfilling as a result of it was constructed by myself phrases.
I’ve been utilizing feedi as my “entrance web page of the web” for a number of months now. Past comfort, through the use of a private reader I’m again in charge of the knowledge I eat, actively looking out for attention-grabbing blogs and magazines, higher positioned for discovery and even shock.