Wikipedia as a castle in the wilderness: modernization in the dynamic world of the internet

By Moriel Schottlender, Principal System Architect, The Wikimedia Foundation

A few months ago, I made the switch from Software Development to Systems Architecture. This change is exciting and an incredible opportunity to grow and learn. A new universe of possibilities and thinking has opened up; one I did not expect.

In this, the first of a series of posts I’ll document my journey so far into the new frontier of Systems Architecture. I’ll explore my own shift in thinking from Software to Systems and what it could mean for the Wikimedia movement and future websites. Follow-up posts will delve into the more technical realms like systems design, event modeling, and the meaning and purpose of some of the exciting ongoing experiments. First, let’s start with an analogy that aims to put the journey into perspective and to provide a common frame to think about and talk about this mental shift. 

In the beginning, there was a castle in the wilderness

Twenty years ago, when the first foundations were laid for Wikipedia, the idea that the internet’s interconnectivity can enable freely available knowledge was revolutionary. Wikipedia filled a void no one even knew needed filling. 

In the wilderness of the internet, Wikipedia was a bright castle, holding within it a collaborative library of knowledge. 

Wikipedians were alone in their task, so they invented a lot of tools from scratch. The castle had to be self-sustaining in a wilderness that had very little to offer. 

We built and strengthened our foundation as more floors were added: More communities joined to create more wikis, and more features were added to accommodate them. We devised our own infrastructure and maintained it ourselves — at the time, other existing infrastructures weren’t suitable for our task.

Our language support is a good example of our ability to innovate and create our own tools. When new communities joined the Wikipedia family, we had to find ways to enable and celebrate languages that the internet wilderness didn’t know how to handle. We did. We took it further to enable translations that are collaborative. These things are still mind-blowingly unique, even in today’s internet.

Through all of that, not only did we remain strong and self-sufficient, we grew. Our castle grew to include different projects, types of media, and unique maintenance tools. We grew into the space around us and expanded the capacity inside. With fewer resources, we solved problems that other websites were only beginning to deal with.

A destination for pilgrimage: how users come to us

Successful support of Wikipedia’s communities led to a pilgrimage of readers and contributors. Curious readers came from all over the world, either casually or extensively, some becoming editors, some visiting over and over.

We developed ways to ensure they could read, edit, and curate. They began arriving using different devices, like different browsers, mobile phones, and apps. We added more doors and gateways to the castle — our web interface, mobile interface, API endpoints. We discovered that people come to us with various intents and purposes, we made sure they were accepted and supported.

We established Wikipedia as a place where truth lives. Because of the collective wisdom of the editors, who care so much about the integrity of knowledge, Wikipedia flourished. Without our communities, the castle would have been a disintegrating ghost house.

Building collaboratively: the Jenga tower castle 

We accommodated and adjusted our castle to fit the multitudes of people who came to take part: readers from different languages and devices, editors with different technological needs, and the seemingly endless types of media types and collection tools. 

We built taller and wider, adding more features. We added a Notification system and expanded the Recent Changes feed to empower contributors to monitor content. We added powerful search tools and enhanced editing accessibility with Visual Editor. We created mobile versions of the interface to enable a better reading experience on smaller screens. This is just a small sampling of the robust features we’ve added through the years. 

The castle’s simple towers branched out into a complex of structures on top of structures, and took the shape of an elaborate, interconnected, precarious Jenga tower, with impressively branched balconies and spires.

This led to another brilliant challenge to overcome. Every time we needed a new balcony or window, we had to verify the structure of the entire castle. Every new floor had to account for the walls around it, the supports underneath it, the hot-air balloons that keep it upright. Our castle’s design became more and more elaborate as we opened more and more doorways of entry and allowed more types of knowledge to exist within its library.

The design — the architecture —  also became heavily interdependent, making each decision to add a feature more and more complicated.

At some point, the castle infrastructure itself was not enough, and we began relying on tools that were created adjacent to Wikipedia, inserted into our production systems. Wikidata is a good example, as well as the iOS and Android apps, serving content and collections through different technologies, and feeding it back to Wikipedia’s collection of articles and views. The castle was no longer a single structure. It had become dependent on the tools around it, even as they lived outside its walls.

Our users built gadgets and extensions to our content, some becoming intrinsic to the workflow of the editors and curators. The castle became dependent on them. Gadgets like HotCat, NavigationPopups, and EditTop, or tools like Twinkle, became ubiquitous to the operation of many of our communities. 

We were no longer simply a castle. All the pieces are so interconnected, it is difficult to distinguish between them. Making changes to our infrastructure became complicated, and our system became monolithic. If we wanted an additional feature, another way to view or edit — we had to adjust and accommodate not only for the structure of the castle but also for the hundreds of gadgets floating around it.

The city that replaced the wilderness: modernization efforts

Twenty years later, Wikipedia is still a beacon that people look up to in awe. Many articles have been written about the “magic” that made Wikipedia possible, and in our current political and social climate our existence matters even more. 

While we were creating a bustling hive of tools, gadgets, extensions, and features around our castle, the world around us changed too.

We are a sociotechnical system. Our technology is heavily intertwined with the social workflows of our users and communities. The emergence of new technologies changed the behavior of readers, editors, and users. They, in turn, led to changes in the technologies we’ve used to support them. This is a cycle that continues in endless loops of change.

Today, the internet is no longer a wilderness, it is a full-fledged city, with skyscrapers and highways and airports. The Wikipedia castle is no longer alone in need to support itself —it is now surrounded by a city that has services and infrastructure to offer. The changing landscape of the internet also meant our incoming traffic has changed; our castle must support people who come from farther places, languages, and cultures—people who arrive via new technology or expect to discover information in a new way.

Opportunities for modernization

Now that a bustling city has replaced the wilderness around us, new opportunities for modernization have opened. We created our own infrastructures, but the modern internet has a slew of services that we can potentially benefit from. By decoupling parts of the architecture, we can support them better, connect them more directly, and focus on their purpose. Most importantly, we can focus on our purpose, and our mission, by directing our energy and expertise to the places where they make the most impact.

Are there spaces where we can switch our own in-house code for an existing open-source tool, and contribute to it upstream? Is there an external, well-maintained library that we can utilize to decouple our monolith and provide a modern frontend framework? Can we find a way to transform our monolithic architecture in a way that will empower us and the capabilities and features we want to expose? What should we keep? What can we leave behind? What do we need to adjust as we move forward?

These are not easy questions to ask, and the answer is not necessarily clear cut, but these are questions that definitely deserve to be explored and investigated. The process itself will result in exploring where and how we can move towards modernizing ourselves for the future.

Imagine putting all our energy, expertise, and effort, full force, into the building and maintaining the tools and services that we do best, without dealing with the inherent complexity of the monolith. Imagine keeping up with the incredible size and scale we’re reaching — without having to reshape the castle’s supports. Imagine freeing ourselves to focus on what we do best: delivering knowledge to a world that is dynamically changing around us. A world that relies on us for expertise and resources.

There’s another challenge that our castle is facing now that it’s surrounded by a bustling city. This one touches on the expectations of our users, the residents, and visitors of the city.

Using distribution services: enabling the TL;DR

(Or: how the way we enable access to our information matters)

People today expect information to arrive wherever they are— on their phone as push notifications, from a news aggregator, in Alexa or Google Home services, on their smartwatch, through augmented reality apps, virtual reality, and more. New technologies are coming and going at a fast pace, and that pace has created a different expectation — from the users — about where and how they digest and accept information.

We are experiencing a shift from readers and contributors coming to Wikipedia directly to search for information — to seeing external services curating our information, translating, crunching the data, and manipulating the output, to fit whatever method they use to send it to the end-user. Google’s been displaying part of articles on its sidebar. Amazon’s Alexa, Apple’s Siri, and Google Home are reading pieces of our articles — sometimes mixed in with other sources — when answering questions by users, taking over control of our content.

People don’t go to the library as often anymore; they expect the library to come to them.

If we don’t adjust to this new reality, we risk giving up the control of how our context and information is presented to users to services that make those decisions themselves.

We have an incredible opportunity to reach our users where they’re at, to deliver experiences that make use of the trusted and robust universe of knowledge that we offer—wherever they are at, whatever technology they use, whatever method they prefer.

Making the transition: from the castle to the city; from Software to Systems

Shifting into the mental models that are needed for modernization is not easy. We need to change the questions we ask ourselves. We need to move from asking how to change the entire castle so it can support several more balconies, and ask how we can transform into a village that thrives in the dynamic, bustling city the internet has become?

I believe this mentality-shift paves the road forward and opens doors. I am excited about the opportunities we have to serve our communities and the world in new ways.

And yet, when questions arise, I still have to remind myself to step outside of my comfort zone and look differently at the capabilities we want to ship. This is where I am at now. Accepting and embracing the unknown, the uncertainty. This is hard for me. I grew up as an Engineer inside this castle. Instinctively, I think and plan and build inside it, too. I’ve learned to be overly cautious. But the truth is that modernization needs me to open my mind further. It needs us to do so together. It’s not enough to think inside this twisty castle; we need to allow ourselves to think bigger. 

The castle needs to transform itself into a village of its own, where individual structures are easy to maintain, expand, improve upon, and scale. Where the roads and alleys allow for adding more structures, we aren’t even thinking about yet. Where new people — developers and editors alike — don’t need a magical series of elaborate maps to find their way from one service to another. Or, when adding a new window, don’t need to check the stability of the entire monolith castle.

We need to modernize in a sustainable way that answers not only today’s new technology challenges but allows us to adjust and pivot 20 years from now. One guarantee is that technology will continue to change and evolve. Modernization designs with that in mind, too.

The process of modernization isn’t just technological; it’s a mental shift. It’s a cultural shift. Shifting the thinking from what we’ve been relying on for 20 years to something so different, so potentially radical — is overwhelming. It’s also incredibly exciting and opens up endless possibilities. 

Our castle, and the farmlands around it, are ready for the next stage. I am too.

About this post

Kamianets-Podilskyi Castle, Ukraine, Rbrechko, CC BY-SA 4.0