Running an IRC Network in 2019: Challenges and Opportunities

Why Bother?

Internet Relay Chat (IRC) is one of the oldest kids on the block. Its user base has been in decline. The Internet has become increasingly centralized and web-centric; “relevant” chat platforms these days are Discord and Slack. So why should we, in 2019, still bother with a chat network running a protocol created back in the late 80s, early 90s? We have found answers of our own to that:

1. The web and the Internet at large is getting increasingly commercialized. IRC networks traditionally aren't; few of even the larger networks are even incorporated. In fact, we'd have to come up with some clever ideas to effectively push advertisements to users.
2. IRC provides a different, much more terse experience. Bells and whistles like avatars don't exist; technically, avatars exist as a client-to-client feature, but it's not in any kind of widespread use. This may be off-putting at first, but may make for a more streamlined means of communication.

The IRCv3 group has been working to create specifications that bring some of the more well liked features in services such as Discord and Slack to IRC. One of their newer proposals is adding reactions to messages, such as emoji. We've been following IRCv3 developments and have been implementing specifications from there, such as SASL authentication and SNI.

Overview

The tl;dr:

• We use Ansible, Hashicorp Vault, ZeroTier, Route 53 with geo-balancing, a custom IRCd (ircd-yeti, a heavily modified snircd fork, which itself is a fork of ircu from UnderNet), custom services (a srvx derivative) and newserv.
• Attracting an audience is hard, and proper mobile support is an unsolved problem.
• Privacy is a niche we want to be in, but we can't make guarantees as strong as we'd like.

Challenges

Accessibility

Attracting an audience towards an IRC network is no easy feat; in fact, it is harder than ever. IRC was designed for a different age. Users were (and still are) assumed to have persistent, stable connections. In an age where over half of web traffic originates from mobile devices that assumption still holds true. While this can be somewhat mitigated with third party offerings like IRCCloud and web clients, the pain points still exist.

Bouncers can also help mitigate this. A bouncer acts as an IRC server to your client and as a client to the destination IRC server, i.e. as a proxy. It keeps messages that were received while there is no client connected to the bouncer, bridging the lack of message history on IRC. This comes with several downsides: Time-stamp information is mangled to some extent, being delivered in messages rather than along with the message. Furthermore, setting up a bouncer requires technical knowledge. A bouncer requires some kind of server to run it on, which isn't free of cost. While free bouncer offerings exist, using them means trusting a third party with all your IRC traffic and passwords. Chances are that the free service has been abused by people in the past, meaning that innocuous users end up being caught in bans not meant for them because they ban the bouncer services as a whole.

Unfortunately, no workable solution for mobile connections has been found yet.

IRC commands are somewhat arcane and clunky. Services and user/channel modes are all over the place across networks, with no way to communicate meanings reasonably to clients. This means that any “IRC tutorial” is doomed from the start for any non-trivial task. We're still in the process of trying to document our IRCd, ircd-yeti, there's only so much we can do.

It's hard to teach people if they just want to get something done, rather than understand an arcane system based around historical decisions going back the 80s and 90s. For example, services manage channel registrations, but services are addressed by sending them private messages. Unless you know and understand that services were historically tacked as clients onto the existing server system, it's just weird to work with.

Another change we've made to increase accessibility is changing from RFC 1459 case mapping to ASCII case mapping. Under RFC 1459 case mapping, {}| are considered the lower-case equivalents of []\, which only exists due to historical reasons. We're considering using Unicode case mapping for channel names, but because Unicode case mapping is specific to a particular locale, we're not sure how to approach this yet. Unicode in nicknames is currently not planned, considering that IRC commands require a user's nickname to complete; it would be a an issue for users on IRC clients with a command line interface to perform actions on users with Unicode nicknames.

Privacy

IRC networks thus need a niche, something to offset that low accessibility by today's standards. One such niche is in privacy. Not being corporations and distributed all over the world, many networks are too minor to be seriously targeted by law enforcement and mass surveillance. One could, however, argue that by merely using IRC, people already set a flag of suspicion. In fact, some Internet service providers have gone so far as to outright block IRC. This is done presumably because botnets, at least historically, would use IRC as a C&C point, so this block would protect many users at the cost of a few being unable to legitimately access IRC.

However, nobody can check what code is running on the servers. Being small entities, it may become interesting to data mine messages sent by users. While we do no such thing, it's also impossible to prove that we don't. IRCd vendors have traditionally been aggressively ensuring that networks do not use their software to unduly spy on their users though. This remains an unsolved and likely an unsolvable problem.

We're also still in the middle of the process of figuring out how to properly accommodate Tor users. Tor has historically been often abused for spam and bots on IRC. However, we understand that using Tor or VPNs with a bad reputation is a necessity. Freenode has a system where Tor users must first register a nickname over an IP address that isn't associated with Tor. This is deeply unsatisfying as it requires Tor users to consciously give up their anonymity as a prerequisite. We're currently considering a CAPTCHA that issues temporary client certificates signed by us that can be used to connect temporarily, but we're not yet sure how to work this into web clients. As of now, we simply allow Tor access, and require a registered account during heavy times of abuse.

Opportunities

Security

Today there exist a few benefits for IRC networks that didn't 20-30 years ago, such as Let's Encrypt, which makes it very affordable (in fact, free as in beer), for networks to get TLS certificates by an actual Certificate Authority.  Much of IRC has been running on self-signed certificates or custom certificate authorities for about a decade. Plaintext connections are still so common that going TLS-only is still not a realistic options, especially since older bots don't support TLS.

We tunnel all our server-to-server connections over a ZeroTier software-defined network, running our own controller and moons. The servers mutually authenticate each other with TLS certificates issued by a private, internal certificate authority. For that, we've first had to implement Server Name Indication (SNI) support in ircd-yeti.

Secrets and passwords are stored in Hashicorp Vault. We use Vault extensively; secrets are put into server configuration files only where necessary. A long-term goal is to have servers talk directly to Vault to get certain secrets, such as linking passwords.

DNS

IRC servers are distributed all across the globe. Users shouldn't have to pick a specific server; plus we have an interest in distributing users out across servers.

We've had a decent amount of success with Amazon's Route 53 to distribute users according to their physical proximity to the respective server and its load. An IRC bot located on a central hub server controls the Route 53 database. When a server splits away from the network or hits a configurable maximum number of users, the bot automatically removes said server from the Route 53 pool of servers that users can be directed to. After that, servers have to be added back to the pool manually; this has been done to ensure users won't be added to servers that are repeatedly unstable. There's plans to implement some sort of health checking so this can be completely automated.

Here's some example output from the bot:

[FR/EU] [41+0/4095 (max 4095)] serverA.darenet.org / ipv4address / ipv6address / Comment
[CA/NA] [3+0/4095 (max 4095)] serverB.darenet.org / ipv4address / ipv6address / Comment
[US/NA] [52+0/4095 (max 4095)] serverC.darenet.org / ipv4address / ipv6address / Comment
Server totals: [US: 52/4095 0%] [EU: 41/4095 0%] [CA: 3/4095 0%] [AS: 0/0 0%] 
Pooled totals: [US: 52/4095 0%] [EU: 41/4095 0%] [CA: 3/4095 0%] [AS: 0/0 0%] 
Pool stats: 3/3 (100.00%) servers pooled; 96/12285 clients, 0.78% recommended capacity.

However, the back-end for that is written in C (for historical reasons because we wanted to continue using our existing newserv setup); working with XML, which Amazon Route 53 requires, in C has been an unpleasant journey.

Until about a year ago, we instead used a self-hosted PowerDNS on the back-end. PowerDNS was backed by a MySQL (later MariaDB) database. MySQL/MariaDB had been chosen for its support for streaming replication. Because of how newserv works, we needed to write a database module to talk to MySQL/MariaDB. libmysqlclient from MySQL would not expose the necessary asynchronous networking primitives, so we instead had to use specific MariaDB client libraries that exposed them. This ended up being a never-ending packaging nightmare on our Ubuntu servers. We had to change our DNS setup because we were still using an old PowerDNS database format, so we re-evaluated the options for achieving geo-distributed DNS and found Route 53. It also guaranteed more reliability than our two or three PowerDNS instances could, so we went for Route 53 and never looked back. To avoid vendor lock-in, our database keeps an authoritative copy of the desired state of the records, which is then synced over to Route 53 when something changes there. The code cleanly separates out the internal state maintenance and updating the external state, so we could change to a different provider/API or even return to a self-hosted solution if needed.

Automation

We've found a reasonable amount of success using Ansible to automate our server deployment. As a result, however, we now require root access for newly linked servers. This is rather unusual for smaller IRC networks, which are typically given none or minimal access to the server's operating system.

Our Ansible setup is tightly coupled with Vault. It pulls secrets out of its database as necessary, stores third party API keys (e.g. for Amazon Route 53), utilizes it to generate internal certificates for server-to-server connections and retrieves the Let's Encrypt certificates.

Ansible is, however, not without its faults. The initial bring-up took two people and two months. IRCd software is unusual as it is normally not installed system-wide. We deal with multiple systems (Ubuntu Server, CentOS, FreeBSD), making system configuration a pain point.

Before introducing Ansible, we had a series of rather elaborate Bourne shell scripts for various tasks, totaling ~825 lines. The scripts worked reliably if and only if invoked in the correct order, with some manual tasks that needed to be done in-between. For example, installing an IRCd on a new host would require to generate the server-to-server ahead of time, which the installation script then would download. This couldn't be automated because the installation script didn't know the server's information for filling in the certificate subject information. These scripts were invoked from an IRC bot (the same one that manages DNS, actually). The scripts were brittle, hard to maintain and unusable when the bot went down – which was usually the time when you needed the scripts the most.

Portability

Because we control the core of our software stack, we can port our software to the platforms we want to support. We support Ubuntu Server as far back as supported by upstream, CentOS 7 and FreeBSD. Personally, I'd like to add OpenBSD and illumos (OmniOS in particular) to that list, but we have enough on our plate already.

Retro Computing

IRC software even from decades ago continues to work with no to very few hacks. For retro computing, IRC is probably one of the most serious contenders for chat software. Networks just don't seem to be going away and IRC clients have historically been plentiful.

DareNET runs an IRC software archive, which provides source code to many historical and current pieces of IRC software history. If you spot something that the archive is missing and happen to have a copy or at least details that would help in finding it, please tell us.

Closing

An IRC network in 2019 is a challenge to run. On as many fronts as possible, we're trying to marry 90s technology to the tooling and the mentality of almost 2020 with, shall we say, varying results? Things are getting better though, with projects such as the IRCv3 working group trying to push IRC ahead. We're happy to talk about our setup; hit us up in #darenet on irc.darenet.org! If there's apparent interest in the nitty-gritty details, we'll continue this post as a series of posts on these individual aspects.