a revised essay I wrote a couple of years ago
Recently, I've been toying with the idea that scale itself is the problem of the Internet. The "growth at all costs" mindset; the complete lack of regard for the costs. But beyond Silicon Valley culture, I felt there was something inherent about software systems at a large scale that was detrimental to people.
Partially, this idea was motivated by my current project at work -- to serve millions of users, we needed to re-architect our service in a way that would necessarily be less efficient in order to handle more traffic. The point shifted from the single user experience to the scale of the system itself: how could we build a system that handled [X] user sessions without also driving up cost by that multiple?
Scale causes context collapse on both sides of software: for the user and for the backend as well. Whereas systems that are small are targeted towards specific people and specific needs, systems that are big are targeted to everyone and all their needs, which is to no one in particular.
It seemed to me that these two concepts - context and scale - have an inverse relationship. Expand one and the other one shrinks. When we talk about context collapse and social media, I think it's often this relationship that is coming into play.
But context is ultimately useful! Even in the most technical of definitions. Companies try to get around the
limited context that comes with scaling. For example,
they often implement Traffic Prioritization for
certain
product surfaces
or certain users. I recently had a conversation with a friend whose current project at work was to add
if client == [Big Customer] { ... }
blocks everywhere, for a customer whose performance
needs
were different from everyone else. Of course, this
customer made up a large portion of her company's revenue, hence the special attention. We add tracing to be able to inspect the exact
execution of certain requests (the context!), again oftentimes those of important customers.
What does it mean, then, to think about software at an "appropriate scale"? I can't say I have a definitive answer to this. But I think to me, it has something to do with that point at which the focus of a system shifts. When it starts to be about delivering value for the company rather than for the users - often inevitable in capitalist/corporate structures.
There are large software system that need to handle millions of users -- the public internet for example. I'm just not sure if that list would include all the other digital tools that we now depend on. For example, collaborative docs, online spaces of organizing, digital invites and event planning tools.
I am interested in how we can make it easier to write applications that will never have to handle massive scale. How can I keep my Google doc just one doc of a few hundred in my personal cloud instead of one of hundreds of milliions in a datacenter who knows where?
When I say I am against scale, this is what I am talking about. How can we create applications that never have to scale, because they are structured completely differently and operate at the level of humans rather than the level of servers and clouds? At this scale, there is no need for traffic prioritization, no need to sample log lines, no need to "find" a user running into a bug. In the inverse relationship between context and scale, I want to swing all the way towards preserving context.
I hope to create software that is in line with this idea, that makes it incredibly easy to write and deploy small-scale applications for human-scale ideas.